An evaluation of search and selection methods used in dental systematic reviews published in English
Michael P. Major, BSc,
Paul W. Major, DDS, MSc, FRCD(C) and
Carlos Flores-Mir, DDS, Cert Orth, DSc
 |
ABSTRACT
|
|---|
Background. Increasing numbers of systematic reviews are published each year, though little has been done to evaluate their search and selection methodology.
Methods. The authors searched dental systematic reviews published between Jan. 1, 2000, and July 14, 2005, for descriptions of how researchers used multiple electronic databases and secondary searches. They evaluated search and selection methods of identified systematic reviews against the guidelines found in the 2005 Cochrane Handbook for Systematic Reviews of Interventions.
Results. The authors identified 220 unique dental systematic reviews. They found that all aspects of search and selection methodology had improved. In 2005, most systematic reviews documented database names and search dates (90 percent), electronic search terms (95 percent) and inclusion-exclusion criteria (95 percent), and most employed secondary searching (100 percent). Many still failed to search more than MEDLINE (20 percent), document the search strategy (20 percent), use multiple reviewers for selecting studies (25 percent) and include all languages (39 percent).
Conclusions and Clinical Implications. Systematic review methodology is improving, though key components frequently are absent. Reviews should be read critically and in consideration of the methodological flaws.
Key Words: Dentistry; evidence-based practice; systematic review
The need to summarize vast amounts of information in science is nothing new. For many years, information summary and packaging have been accomplished through the publication of narrative reviews written by experts. However, it became increasingly clear that such reviews were subject to potentially large amounts of bias or error,1–4 and prevailing opinions often were based on charismatic authority of the most popular lecturers and researchers rather than on the best available evidence. This realization influenced the medical communities to seek evidence-based commentaries that summarize scientific knowledge for providing best patient care and guiding research directions. As the value of evidence-based knowledge has increased, so has the use and publication of the systematic review.
A systematic review can be defined as "a review that has been prepared using a systematic approach to minimizing biases and random errors which is documented in a materials and methods section."4(p5) The most significant distinction between the narrative review and the systematic review is the transparency and reproducibility of the literature search.4–6 By this explicit methodological distinction, the systematic review aims to remove much of the bias found in the narrative review.7 As a result, the systematic review has become the cornerstone of evidence-based health care. The quality of a systematic review is tied directly to the clarity, transparency and reproducibility of its literature search for appropriate studies.
Since the concept of evidence-based care first found its way into dentistry, there has been a rapid increase in the number of published systematic reviews and meta-analyses.6,8 Despite the large increase in evidence-based knowledge in dentistry, little has been done to evaluate the quality of dental systematic reviews and their associated meta-analyses. Because the systematic review still is establishing and refining itself in dentistry, periodic critiques are important to gauge progress and direct future development. Therefore, our aim in the study we describe here was to catalog all dentistry-related systematic reviews published in English between Jan. 1, 2000, and July 14, 2005; evaluate the keystone methodology of these systematic reviews, the search and selection of appropriate studies; and track its development during those years to gain a sense of from where the science has come, where it stands and where it still needs to go.
|
METHODS
|
|---|
The gold standard search and selection methodology of the evidence-based systematic review and meta-analysis has become the reviews produced by the Cochrane Collaboration, an international nonprofit organization that produces and disseminates systematic reviews of health care interventions. Therefore, we set out to catalog the entire library of systematic reviews and their associated meta-analyses in dentistry and then evaluate the search and selection criteria against the guidelines set out in the Cochrane Handbook for Systematic Reviews of Interventions9 and other empirical, evidence-based recommendations. The Cochrane handbook is not the only guideline available regarding the conduct of systematic reviews, but it seems to be the most recognized.
To identify all systematic reviews and meta-analyses in dentistry, we used the search terms employed by Bader and Ismail8 in addition to truncated forms of the dental specialties following the pattern proposed by Stamm and Hohoff.10 We developed the following search strategy for the Ovid interface of MEDLINE with the help of a senior health sciences librarian:
[((systematic review) OR (meta-analysis)) AND (dentistry OR dental OR tooth OR teeth OR (stomatognathic disease NOT pharyngeal disease) OR endodon$ OR orthodon$ OR oral surger$ OR oral surgic$ OR periodon$ OR prosthodon$ OR pedodon$]
Searches were limited to human studies, English language and articles published since 2000, since preliminary evaluations of evidence-based reviews before 2000 already have been done elsewhere.6 We made minor adjustments to adapt the strategy to the other electronic databases searched (Table
). From Jan. 1, 2000, to July 14, 2005, we searched the Cochrane Library (including Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, Cochrane Central Register of Controlled Trials, Cochrane Database of Methodology Reviews, Cochrane Methodology Register, Health Technology Assessment Database, NHS Economic Evaluation Database), EMBASE Excerpta Medica (Ovid), MEDLINE (including in-progress and other nonindexed citations) (Ovid), PubMed and Web of Science (Thomsen Scientific) electronic databases. We followed electronic searches with secondary searches, including hand-searching of bibliographies of identified systematic reviews and other articles pertaining evidence-based dentistry, systematic reviews known to us before the study, and systematic reviews encountered while performing literature searches for other research projects.
We considered articles to be systematic reviews only if their authors had presented their search and selection criteria in a methods section. Initial screening of articles by title and abstract was performed independently by two of the authors (M.M. and C.F.) to identify potential systematic reviews. We tended to err on the side of caution, including any article that appeared to meet the requirements. We then retrieved identified potential systematic reviews in full for a second selection process to verify potential reviews as being truly systematic. Disagreements were resolved by discussion until we were satisfied.
We evaluated the search and article selection for each evidence-based review against the gold standard provided in the Cochrane Handbook for Systematic Reviews of Interventions9 and other empirical, evidence-based recommendations. The criteria for evaluation were as follows:
- – the review documented the database names and date ranges searched9(Section 5.1.1);
- – the authors had performed electronic searches in at least two databases9(Section 5.1.1),11–13;
- – the authors had followed up electronic searches with secondary searches9(Section 5.1.2–5.1.4),14,15;
- – the review presented all electronic search terms in a clear and reproducible manner6,9(Section 5.2.2.1
[EC]
);
- – the review presented the way in which the terms were combined (that is, Boolean operators) in a clear and reproducible manner9(Section 5.2.2.1
[EC]
);
- – the article provided clear and reproducible inclusion-exclusion criteria9(Section 5.2.3);
- – included articles were selected by a team of two or more reviewers9(Section 5.2.3),16;
- – the authors’ search had included reviews in all languages.9(Sections 5.2.1,5.2.2.1),11,17,18
We evaluated the listed criteria dichotomously; we marked a criterion found in a systematic review with a "1" and reviews that failed a criterion with a "0." Evaluation of every article identified was performed independently by the two authors. We recorded assessments using Microsoft Excel spreadsheets and compared results systematically. We resolved disagreements by re-examining the article in question and then discussing it until we agreed with the decision.
|
RESULTS
|
|---|
The electronic searches yielded 1,107 hits, of which 479 were duplicates. In addition, we identified 34 additional potential systematic reviews using secondary searching methods, yielding 628 unique articles. Five hundred three articles met the definition of a systematic review and pertained to the field of dentistry. We found 26 (11.8 percent) of those systematic reviews in the Cochrane Library, 26 (11.8 percent) in EMBASE, 152 (69.1 percent) in MEDLINE, 182 (82.7 percent) in PubMed, 95 (43.2 percent) in Web of Science and 22 (10.0 percent) using secondary searching methods (Table
). After we eliminated 283 duplicate articles, we had 220 unique articles that met the definition of a systematic review and pertained to the field of dentistry.
We encountered five duplicate systematic reviews; all had been published as both a Cochrane review and as a journal article. We chose the journal versions of duplicate articles for evaluation because journal articles are more generally available to clinicians.
Encouragingly, with each year, the number (Figure 1) and quality (Figure 2) of systematic review search criteria increased. The percentage of reviews documenting and using each criterion of article searching increased from 2000 to 2005. Of articles published in 2005, nearly all systematic reviews employed secondary searching (95 percent), provided electronic search terms (95 percent) and documented the database name and date range searched (100 percent). Unfortunately, in 2005, there still remained many systematic reviews that failed to search more than one database (20 percent) and failed to provide the method by which search terms were combined to allow for a search replication (20 percent).
Similarly, the percentage of systematic reviews using sound selection methodology increased during the five years on which we focused (Figure 3). There were drastic increases in the percentage of systematic reviews that had no language restrictions and used more than one reviewer for article selection. But despite the improvement, these two categories still were the most common sources of potential bias; 39 percent of articles imposed language restrictions and 25 percent failed to use more than one selection reviewer. Well-documented inclusion criteria was the strongest aspect of all systematic reviews; approximately 95 percent of all systematic reviews published each year met this criterion, except for an unexplained temporary drop in 2001.
|
DISCUSSION
|
|---|
Having views and practice guidelines based on strong empirical evidence rather than charismatic authority has become increasingly important in health care professions. The cornerstone of all branches of evidence-based health care has become the systematic review. Systematic reviews are foundational to evidence-based health care, because they aim to eliminate bias by their well-documented, transparent and systematic search and selection of trials. Our study cataloged the systematic reviews published in dentistry since 2000 and evaluated their search and selection criteria.
A steadily increasing number of systematic reviews is published each year in a near-exponential pattern, as reported by Bader and Ismail8 and Glenny and colleagues.6 We found the same trend. The evaluations of systematic review search and selection criteria revealed an encouraging fact: increasing numbers of systematic reviews are being carried out with higher-quality methodology. In a time of evidence-based health care, dentists and oral health providers can be increasingly assured that the systematic reviews available to them are becoming more empirically sound and less prone to bias.
However, some key areas of systematic reviews still require improvement. Researchers performing systematic reviews need to ensure their electronic search strategies are well-documented with all Boolean operators and combinations of terms; begin searching more databases than just MEDLINE; have more than one reviewer selecting trials for inclusion; and remove language limitations from their literature searches.
Implementing each of the above techniques is critical, because all have been empirically linked to reducing potential introduction of bias or are required for the transparent demonstration of study methodology. Documented search strategies including Boolean operators for combination of search terms are necessary for study reproduction. Failing to search multiple databases can result in missed identification of relevant trials, which could introduce bias and thus lead the author to false conclusions.11–13 Having multiple reviewers reduces potential selection bias and decreases the possibility of accidentally excluding relevant research.16,19 Imposing language limitations also can exclude potentially relevant research and possibly lead the author to false conclusions.17,18 Therefore, to diminish potential bias, all efforts have to be made to have non-English publications translated.
Our study encountered a time lag for identifying systematic reviews. Systematic reviews published in late 2004 would not necessarily be indexed by mid-2005. This pattern was not unique to our study; it also was present in two previously published articles.6,8 With this in mind, we expect the same trend to continue with our own data and anticipate a continued exponential increase of systematic reviews in dentistry.
For systematic reviews to be useful both clinically and empirically, they must be performed in a way that maximizes their validity. From 2000 to 2005, there was an increase in both the number of systematic reviews and the number of well-done systematic reviews in dentistry. These two findings strengthen the usefulness of systematic reviews for clinical and policy decision making. However, there still is room for improvement in key areas of trial search and selection methodology: the methods of searching more databases than only MEDLINE, documenting search strategies with Boolean operators, including all languages in the search and using more than one reviewer to select included articles all need to be used more frequently. Clinicians can become increasingly confident in the conclusions of systematic reviews but still should read them critically. Researchers should accept the challenge to continue improving their systematic review methods, embracing their responsibility and obligation to ensure that any published findings are as complete, comprehensive and free of bias as possible.
We have identified some limitations with the current methodological approach. Caution has to be exercised because our study evaluated only systematic reviews published in English. Extrapolation to systematic reviews published in other languages cannot be supported. There is a potential limitation in our not having included oral pathology, oral radiology and dental public health specialties in the search terms. We retrieved several systematic reviews of those specialties, but we cannot ensure that we may have missed some by this omission.
|
CONCLUSION
|
|---|
The methodology used in systematic reviews is improving, though key components frequently are absent. Readers of these reviews should examine them critically and take into consideration the reviews’ methodological flaws.
|
FOOTNOTES
|
|---|
Mr. Major is a doctoral student, Goldman School of Dental Medicine, Boston University, Boston, Mass., and a member of the Craniofacial & Oral-health Evidence-based Practice Group, University of Alberta, Edmonton, Alberta, Canada.
Dr. Major is the program director, Orthodontic Graduate Program, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Canada, and a member of the Craniofacial & Oral-health Evidence-based Practice Group, University of Alberta, Edmonton, Alberta, Canada.
Dr. Flores-Mir is a clinical associate professor, Orthodontic Graduate Program, and the director, Craniofacial & Oral-health Evidence-based Practice Group, Faculty of Medicine and Dentistry, 4051 Dentistry/Pharmacy Centre, University of Alberta, Edmonton, Canada T6G 2N8, e-mail "carlosflores{at}ualberta.ca". Address reprint requests to Dr. Flores-Mir.
The complete list of systematic reviews evaluated in the authors’ study is available with this article as posted on JADA Online ("http://jada.ada.org"). Interested readers may link to this article online, then click on the link in the "Supplemental Data" box.
|
REFERENCES
|
|---|
- Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987;106(3):485–8.[Medline]
- Oxman AD, Guyatt GH. Guidelines for reading literature reviews. CMAJ 1988;138(8):697–703.[Abstract]
- Teagarden JR. Meta-analysis: whither narrative review? Pharmacotherapy 1989;9(5):274–81.[Medline]
- Egger M, Smith GD, O’Rourke K. Rationale, potentials, and promise of systematic reviews. In: Egger M, Smith GD, Altman DG, eds. Systematic reviews in health care: Meta-analysis in context. 2nd ed. London: BMJ Publishing Group; 2003:3–23.
- Klassen TP, Jadad AR, Moher D. Guides for reading and interpreting systematic reviews, I: getting started. Arch Pediatr Adolesc Med 1998;152(7):700–4.[Free Full Text]
- Glenny AM, Esposito M, Coulthard P, Worthington HV. The assessment of systematic reviews in dentistry. Eur J Oral Sci 2003;111(2):85–92.[Medline]
- Needleman I, Moles DR, Worthington H. Evidence-based periodontology, systematic reviews and research quality. Periodontol 2000 2005;37:12–28.
- Bader J, Ismail A, ADA Council on Scientific Affairs, ADA Division of Science. Survey of systematic reviews in dentistry. JADA 2004; 135(4):464–73.
- Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions 4.2.5 [updated May 2005]. Available at: "www.cochrane.org/resources/handbook/handbook.pdf". Accessed June 18, 2006.
- Stamm T, Hohoff A. Nonlinear behavior of search strategies for identifying relevant orthodontic articles. Angle Orthod 2004;74(3): 316–8.[Medline]
- Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309(6964):1286–91.[Abstract/Free Full Text]
- Savoie I, Helmer D, Green CJ, Kazanjian A. Beyond Medline: reducing bias through extended systematic review search. Int J Technol Assess Health Care 2003;19(1):168–78.[Medline]
- Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials 2000;21(5):476–87.[Medline]
- Helmer D, Savoie I, Green C, Kazanjian A. Evidence-based practice: extending the search to find material for the systematic review. Bull Med Libr Assoc 2001;89(4):346–52.[Medline]
- Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med 2002;21(11):1625–34.[Medline]
- Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Stat Med 2002; 21(11):1635–40.[Medline]
- Moher D, Fortin P, Jadad AR, et al. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 1996;347(8998):363–6.[Medline]
- Turp JC, Schulte JM, Antes G. Nearly half of dental randomized controlled trials published in German are not included in Medline. Eur J Oral Sci 2002;110(6):405–11.[Medline]
- Sutherland SE, Matthews DC. Conducting systematic reviews and creating clinical practice guidelines in dentistry: lessons learned. JADA 2004;135(6):747–53.
TOP
ABSTRACT
METHODS
