Systematic reviews are summaries of the world literature on a specific topic that use explicit methods to systematically search, critically appraise and synthesize the evidence from clinical research.³ They are recognized as the highest levels of research evidence⁴ and their methodology is well-documented.⁵ These reviews are undertaken with the same diligence as one expects from the primary research, with each study in the review treated as a "unit of analysis," and specific eligibility criteria being used for each study’s inclusion. The strengths of systematic reviews include a clearly defined question, a comprehensive search strategy, explicit inclusion criteria, assessment of the methodological quality of the included studies, synthesis of the data and a summary of the results. Their tremendous value lies in the rigorous methods that are used to minimize bias and in their ability to combine smaller studies and thus increase statistical power to detect a treatment effect. On the other hand, systematic reviews often do not produce final, definitive answers for health care; instead, they often show where high-quality primary studies are lacking and help to define future agendas.⁶ CPGs are defined as "systematically developed statements to assist practitioners and patients in arriving at decisions on appropriate health care for specific clinical circumstances."⁷ To produce well-developed guidelines, one should use the methods of the evidence-based process to assemble, organize and synthesize the best available evidence from clinical research and integrate this with clinical expertise in the formation of clinical recommendations.^8,9

The Canadian Collaboration on Clinical Practice Guidelines in Dentistry, or CCCD, was formed in 1999.¹⁰ This partnership of the Canadian Dental Association, the provincial dental associations and dental regulatory authorities, dental specialty groups and the Association of Canadian Faculties of Dentistry is the national, autonomous body for the development of evidence-based guidelines for the dental profession in Canada. The goal of the CCCD is to assist dentists in providing the best possible dental care for their patients. Guidelines are created using rigorous evidence-based methods: the findings of a systematic review are combined with a consensus approach to incorporate practitioner feedback and clinical expertise in the final guideline. To date, the CCCD has completed two systematic reviews^11,12 and two CPGs,^13,14 and several more guidelines, using existing systematic reviews, are in the planning stages.

Guidelines are created using rigorous evidence-based methods: the findings of a systematic review are combined with a consensus approach to incorporate practitioner feedback and clinical expertise in the final guideline.

Research overviews and evidence-based guidelines are relatively new to dentistry but are increasing in number rapidly. For example, the Oral Health Group of the Cochrane Collaboration has done 39 systematic reviews—more than double the number completed as of early 2003—and many more are in progress. Despite significant advances in the methodological aspects of performing systematic reviews and developing practice guidelines, there are many challenges and practical problems encountered along the way in doing this type of research. Yet, this is true of all good research. Solutions are needed so that dental professionals can benefit from the systematic reviews that provide the foundation for sensible recommendations for dental practice.

In this article, we share the experiences, insights and lessons learned by the Methodology Resource Group of the CCCD, of which we were co-chairs, during the various stages of the research process, in carrying out the first systematic review and developing the CPG on acute apical periodontitis, or AAP, for the CCCD. In addition, we offer practical advice to others who may undertake similar projects.

	THE SYSTEMATIC REVIEW

TOP ABSTRACT THE SYSTEMATIC REVIEW MANAGING THE PROCESS SUMMARY REFERENCES

 
Selecting the topic. The CCCD, in consultation with stakeholder groups, selects and prioritizes topics using the following criteria in the decision-making process:¹⁵

– prevalence of the clinical condition;

– burden of illness;

– amount of variation in practice patterns in managing the condition;

– relevance to local practice patterns;

– likelihood of influencing change in clinical practice;

– availability of high-quality evidence to support practice;

– cost of managing the condition.

The CCCD chose AAP as the topic for the first evidence-based guideline primarily because of its high prevalence in everyday practice and anecdotal evidence of practice variation, indicating a degree of clinical uncertainty among practitioners.

Formulating the question. The question driving the systematic review must be clear and focused, for it affects every step of the process.¹⁶ A "well-built" question¹⁷ will include four key elements: the population or condition of interest; an intervention or an exposure (for example, to a test or a risk factor); a comparison or control group; and a specific outcome. Each of these aspects must be carefully defined. These definitions then guide the literature search strategy and the identification of the eligibility criteria that determine whether a study can be included in the review.

One of the greatest challenges in the systematic review of AAP was in defining the condition. At the outset, when the topic was chosen, this difficulty was highlighted at a meeting of the clinical advisory group—a group that consisted of three general practitioners and two endodontists (each of whom holds an academic appointment in addition to considerable practice experience). In fact, there was extensive discussion on this point at the meeting. Furthermore, definitions in the literature were found to be inconsistent, and subsequent consultations with other endodontic specialists revealed disagreement among these experts as well. These difficulties led to some backtracking in the development of the search strategy. The group decided to develop what seemed to be the most accepted definition and use it with consistency throughout the review. We came to understand that AAP is not an isolated condition; rather, diseases of the pulp are on a continuum, from reversible pulpitis to necrosis and AAP to abscess. Therefore, we wrote explicit clinical and radiographic descriptors to guide the reviewers and ensure uniformity of decisions.

One of the greatest challenges in the systematic review of acute apical periodontitis was in defining the condition.

The lack of a uniform definition of AAP became glaringly apparent as we read research reports. The condition was characterized in many different ways in various studies, which affected how patients were classified and selected for entry in a particular trial. In hindsight, wider consultation with key researchers and academics at the initial question stage would have been valuable. To assist future reviews and to facilitate better clinical endodontic research, it would be advantageous if endodontic specialty associations could develop consensus statements to provide consistent definitions for endodontic disease entities that then could be used as guides for the design of future clinical trials.

Designing and executing the literature search. A planned search strategy is essential. The strategy is formed around the elements of the question. Our key search terms were developed on the basis of our definition of the disease, the interventions and the outcomes of interest, through a process of brainstorming with the research team (usually via e-mail, since the team members were located throughout Canada). We modified the terms during the course of the work as our understanding of the problem evolved, and we applied the final template—consisting of terms related to the clinical condition, the outcomes of interest, study design and language limits—to each of the various interventions. We then ran the search numerous times in an iterative approach until we felt the strategy had the best balance of sensitivity (a broad search, which yields the largest number of relevant papers but also many irrelevant ones) and specificity (meaning that the irrelevant papers are weeded out, but pertinent ones may be missed). A qualified librarian on the research team is a definite asset. Our research librarian was a dentist as well—a unique and valuable combination.

The search was exhaustive, using several databases and other sources. Although studies have shown that MEDLINE searches detect, on average, only 51 percent (range: 17–82 percent) of randomized controlled trials, or RCTs, available in an area of health care,¹⁸ we found MEDLINE to be quite useful for our topic. This is consistent with the findings of a recent study of the dental literature, which found high sensitivity and specificity for MEDLINE searches in the field of endodontics, indicating that this topic is well-indexed by MEDLINE.¹⁹ Other databases, including EMBASE and the Cochrane databases, did not identify additional reports for our review. However, we do recommend the use of these and other databases. As the specialized trial register of the Cochrane Oral Health Group, in particular, continues to grow through hand searches of journals, it likely will assume increasing importance for searches of the dental literature. We identified several articles through references in articles we had retrieved, and one key study came to our attention through communication with the author of another included study.

Because of resource restrictions, we limited the papers we retrieved to those published in Canada’s two official languages, English and French. We did, however, run the search strategy for each intervention without language restrictions to do a rough evaluation of language bias. For this topic, on average, 80 percent of the papers were in English, with a range of 59.6 percent to 100 percent for various interventions. A recent study of a number of disease areas showed that language-restricted meta-analyses, compared with language-inclusive meta-analyses, do not differ with respect to the estimate of an intervention’s benefit and effectiveness.²⁰ Nevertheless, reviewers should be aware of the possible threat to the validity of the review if a large body of non–English-language articles is ignored.

We found it useful to run a continuous search throughout the project (and afterwards, to monitor new publications) using an Automated MEDLINE Query. This free service (available at "www.medfetch.com") e-mails new citations, including abstracts, on a weekly basis.

Selection of publications, quality assessment and data abstraction. For these aspects of the review, the biggest obstacle we encountered was related to poor reporting of study details and results in articles. Unclear definitions of AAP and vague patient entry criteria in some trials made it difficult to decide whether certain trials should be included. This type of problem can lead to exclusion of some studies that might be included—thus reducing the sample size and precision of the review—or to inclusion of questionable studies. Either scenario will introduce bias to the review.

Quality assessment of the included articles was difficult; when reporting was ambiguous, it was unclear whether the study was designed and executed poorly or whether the reporting of key design features was inadequate.

Quality assessment of the included articles was difficult; when reporting was ambiguous, it was unclear whether the study was designed and executed poorly or whether the reporting of key design features was inadequate. We used the validated Jadad Scale²¹ to assess and score the quality of each included study, but supplemented it with a checklist we devised to further assess areas we believe are important in RCTs. For example, we looked for power-based sample-size estimates, intention-to-treat analyses and the proportion of dropouts and withdrawals. Double blinding is an important component of the Jadad Scale, but may not be possible, for instance, in surgical trials. We therefore modified the scale, as is commonly done for such studies, giving full points to blinding of outcome assessment in those cases. Modified Jadad Scales have never been evaluated in validation studies. Missing or unreported data were a problem when we were abstracting data for synthesis. Frequently, means were reported without standard deviations, rendering the data useless for meta-analysis.

The above-mentioned problems underscore the need for two or more independent reviewers to perform the review steps while blinded to the findings of the other reviewers, and for consensus meetings for discussion and resolution of disagreements. We found that when reporting in a study was clear, disagreements usually were related to oversights on the part of one of the reviewers or, less often, to subjective misinterpretation.

Unclear reports present a major problem for reviews of this type. Although contact with primary authors yielded missing data for and clarification of some trials, more often than not, authors could not be located or the data were no longer available to them. Missing or incomplete data resulted in some studies being omitted from the review. Clearly, the solution to this inherent problem with retrospective research of this nature is for editors to demand that the reporting of clinical trials in dentistry adhere to the standards of the Consolidated Standards of Reporting Trials, or CONSORT, statement developed by an independent panel of international researchers.²²

Synthesis of results. Clinical trials that study the same or similar interventions for the same condition may be conducted quite differently. Subjective judgments must be made as to whether the trials are similar enough that their results can be pooled in a statistical analysis (meta-analysis). Knowing that our decisions in this regard were bound to be somewhat subjective, we created a table of trial characteristics, so that readers could judge whether they felt our decisions were appropriate (Table).

	MANAGING THE PROCESS

TOP ABSTRACT THE SYSTEMATIC REVIEW MANAGING THE PROCESS SUMMARY REFERENCES

 
In addition to what we learned in doing the systematic review, we would like to share our experiences and insights in terms of organizational and process issues in managing this project.

Expertise. To conduct research overviews and develop CPGs, two types of experts are needed: content experts and methodology experts. The CCCD has developed a protocol to guide the creation of a CPG. When the CCCD has chosen a topic for a CPG, a clinical advisory group is formed to provide clinical input. Each clinical advisory group is composed of a team of volunteer practicing dentists with a particular interest or expertise in the content area of the guideline topic under development. A person experienced in guideline development chairs the clinical advisory group. The clinical advisory group for each guideline generally consists of two or three specialists (no more than one of whom can be a full-time academician) and four or five general dentists (again, no more than one of whom can be a full-time academician). The responsibility of the clinical advisory group is to coordinate the development and maintenance of an individual guideline. The clinical advisory group defines the question and then directs the methodology resource group to perform the systematic review. The methodology resource group is composed of people with expertise and methodological skills in the retrieval and evaluation of scientific evidence, or training and experience in clinical epidemiology or a related discipline. A person knowledgeable about the methodology of systematic reviews chairs that group.

At the conclusion of the systematic review, the clinical advisory group makes recommendations derived from the evidence. These recommendations may not be adequate or sufficient to guide practice because, for many topics, there has been limited or no research about aspects that nevertheless are extremely important in the management of a clinical problem. In addition, the research that has been done may have limited application to the types of patients seen or in the interventions available for use in the general population of patients. To address these limitations, the chair of the clinical advisory group seeks further input from clinicians at this stage—first from the members of the clinical advisory group and then from a broad sample of practicing dentists who have volunteered to review our in-progress guideline, as well as from practicing specialists and academicians. The clinical advisory group then revises the evidence-based recommendations on the basis of this feedback. For the first guideline, the chair of the clinical advisory group obtained practitioner feedback through a mailed survey of 20 volunteer dentists and 10 chairs of endodontic programs in Canada. The survey consisted of items evaluating the methods, results and interpretive summary used to form the draft recommendations and whether the draft recommendations should be approved as a practice guideline. Written comments were invited.

We found that this feedback was invaluable. In addition to reading the lengthy full version of the guideline-in-progress and answering the structured questionnaire, all but one of our volunteer dentist reviewers provided carefully considered, in-depth qualitative feedback that was vital to shaping the final guideline. Ninety percent endorsed the evidence-based report as a practice guideline and indicated that they were likely or very likely to use this guideline in their practice. Our intention is to expand our database of reviewers considerably, so that a larger random sample of Canadian dentists can be surveyed as part of the development of future CPGs.

We believe that this hybrid approach to guideline development incorporates the best features of evidence-based and consensus-based guidelines.

Working at a distance. Geography presented a challenge during the course of this project. Members of the clinical advisory group and the methodology resource group were located throughout Canada. All were engaged in clinical practice from one to five days per week. Although much work was done by e-mail, more costly face-to-face meetings occasionally were necessary. Phone conferences, given time zones and practice schedules, were difficult to arrange and were nearly impossible during the summer months. For future reviews and guidelines, we may consider regional clinical advisory and methodology groups.

Time management. Because this was the first CPG for the CCCD, considerable infrastructure needed to be established, not just for this guideline, but also to facilitate future CPGs. At the same time, our stakeholders in organized dentistry had made a significant investment in launching the CCCD and had put their faith in the methodology group to deliver the first "product." Our main advice to other groups considering similar projects is to never underestimate the time it takes to carry out thorough and methodologically sound systematic reviews and the subsequent development of guidelines. As with any research project, organization—especially in the form of structured data collection forms and well-maintained, up-to-date and secured study records—is imperative. It is important to set realistic time targets and then ensure that all involved adhere to them. We found during the course of the project that the time spent at the outset in developing protocols was well worth the time and effort. At the end, we felt that our methods were robust and had served us well during some potentially difficult times. Depending on the size of the project, a dedicated administrative assistant or a research assistant with some training in the methods of systematic reviews would be invaluable.

Costs. The development of evidence-based guidelines takes considerable time and therefore is costly. We estimate that we each spent 570 hours on this project. Although one of us (D.C.M.), who holds an academic position, was largely supported by her faculty to engage in this process, at least 200 hours of the total was voluntary personal time. The other author (S.E.S.), working in a fee-for-service setting, had no financial support. It is estimated that members of the clinical advisory group spent about 40 hours in meetings, conference calls and e-mail discussions, with additional undocumented time spent reading various drafts of the systematic review and guideline. At least 40 hours were spent by the research librarian in developing, refining and executing the search strategy.

The CCCD is currently working on strategies to continue to produce evidence-based guidelines in view of the time commitments and costs involved. Clearly, volunteer support of the magnitude described above is not sustainable. One of the barriers to evidence-based CPG development in Canada is the fact that there is a relatively small number of Canadian dentists who have training in clinical epidemiology, research methodology or the methods of systematic reviews. Identifying those people, sparking their interest in this exciting endeavor and providing appropriate rewards for their efforts is the next major challenge for the CCCD. As more high-quality systematic reviews, such as those produced by the Cochrane Collaboration and dentistry’s academic institutions, become available to provide the evidence base for guidelines, the task of developing evidence-based CPGs will become more manageable.

	SUMMARY

TOP ABSTRACT THE SYSTEMATIC REVIEW MANAGING THE PROCESS SUMMARY REFERENCES

 
We learned a great deal in the course of this project. Probably the most valuable lesson related to the importance of the definition and clarity of the question. We found that this step takes time, requires considerable input from experts and guides the entire process. We also realized that the rigorous methodology we established is robust. Our inclusive and transparent processes served us well. The CCCD is becoming recognized internationally as a leader in guideline development. The challenge is to maintain the momentum we have established and to take CPG development in oral health to new heights. We will provide an update on our progress in the dental literature.