Types of Studies Reviewed. The author reviewed the psychometric literature, focusing on high-stakes professional decisions. Editorials in the dental literature and position papers of involved organizations often use words from this literature without incorporating its fundamental concepts.

Results. The reliability of one-shot initial licensure examinations is estimated to be approximately r = .40, which is a value well under the standard for such tests in other professions. Validity has not been investigated rigorously, but the one-shot format and proposals to remove live patients certainly would reduce validity. The use of portfolios—a small number of evaluations in several realistic task domains—is a viable means of achieving psychometric standards for initial licensure decisions.

Clinical Implications. Boards are charged with making valid and reliable licensure decisions, not with conducting examinations. At a minimum, they must define the competencies of beginning practitioners and establish the psychometric criteria for their decisions (neither of which are done currently). Gathering data then can be delegated to whoever is best qualified to meet these standards.

The following assumptions have been advocated as being appropriate to the licensure examination process:

– Students should be granted an initial license to practice dentistry only after demonstrating competency in treating live patients.

– Students should not be allowed to substitute an advanced general dentistry experience for a patient-based evaluation. The amount of teaching is not an effective proxy for the quality of learning.

– Licensure by legislation undermines educational qualification and board testing equally. California Assembly Bill 1045 and similar legislation granting temporary licenses to graduates of dental schools not accredited by the Joint Commission on Dental Accreditation and who were not examined at any point for competency by standards applied to U.S.-trained dentists create a two-tier system.

– Simulations and standardized tests are steps away from valid testing of competency. Simulations are appropriate learning and evaluation tools when students are beginning dental school clinical experiences but not when they are concluding them.

– One-shot tests of professional competence have very low validity (regardless of their reliability) because of the large variance introduced by patients and the magnitude of subject-by-trial variation. Evaluations that fail to sample the full range of competencies required of dentists have unknown validity and thus expose the public to an unnecessary level of risk.

– Graduates of accredited dental education programs should not be granted licenses automatically. Licensure authority rests with boards, not schools. Schools should have a say in the process because they know many things that boards do not know about candidates. Boards can delegate information gathering to others, but they must retain the decision-making responsibility.

– False-positive licensure decisions (granting licenses to candidates who are incompetent to practice) are more damaging to the public and the profession than are false-negative decisions (denying licenses to qualified candidates). Mechanisms such as reexamination are in place to correct false-negative decisions but not false-positive decisions.

Portfolios use more data, more diverse data and data of a higher quality than what are used now.

All of these assumptions have been expressed in discussions regarding initial licensure. Generally, however, different constituencies advance different subsets because it is assumed that these seven assumptions are internally inconsistent. In this article, I describe how portfolios can be used in the licensure examination process without displacing or affecting any of the stated assumptions. Portfolios use more data, more diverse data and data of a higher quality than what are used now. The portfolio approach has the extra advantages of increasing fairness and being less costly than the current method.

	THE ISSUE OF VALIDITY

TOP ABSTRACT THE ISSUE OF VALIDITY PORTFOLIOS AND COMPETENCY THE PORTFOLIO APPROACH PORTFOLIO EVALUATION AND... PORTFOLIO EVALUATION AND THE... INITIAL LICENSURE IS POLITICAL... CONCLUSIONS REFERENCES

 
Validity is not a stable and unitary concept, even among psychometricians.^28–34 Two definitions of the term are solid enough to support this discussion of testing for initial licensure. Predictive validity refers to those situations in which judgments about future behavior (based on observing present behavior) tend to be correct. For example, are the people whom we think will practice consistently with standards of care (based on test data or other evidence) more likely to practice that way than those about whom we have reservations? Predictive validity data regarding initial licensure evaluation have never been published. Guidelines for Valid and Reliable Dental Licensure Clinical Examinations³⁵ states that "job competency" for a practicing dentist is "too complex" to make predictive validity possible. This is an unfortunate statement to put into writing, since some licensed dentists have been disciplined because they failed to meet the job competency for practicing dentists.

Development of initial licensure testing in dentistry tends to rely on a second definition of validity. Content validity involves the sampling relationship between the content of an evaluation process and the domain of performance expected. Content validity is desirable, but it is no guarantee of predictive validity. An evaluation with strong content validity can be spoiled by an inappropriate decision (cut-score) rule or a bad distribution of candidates.³⁶ The driver’s license test is a notorious example.

Initial licensure examinations have been validated by inquiring whether tasks evaluated on the test are representative of tasks critical to the safe and effective practice of dentistry. The guidelines describe a validation process undertaken by the ADA and the American Association of Dental Examiners, or AADE, licensure committee in 1992.³⁵ The examination blueprint (what will be tested) was validated, in part, on opinions about reliability instead of measured reliability, and the examination was characterized as subjective.

Validating the examination, however, is not exactly the right challenge. What we really want to know is whether the tasks critical to the safe and effective practice of dentistry are evaluated on the test. That could be another matter entirely from whether the tasks on the test are part of dental practice. Figure 1 shows this situation diagrammatically. Oval A represents the domain of skills, understanding and supporting values (the competencies) necessary to begin independent dental practice. Oval B represents the domain of skills and understanding sampled in a typical initial licensure examination. The content validity of the test (the proportion of the test that covers tasks used in practice) is high (the percentage of B that overlaps with A). However, the true content validity (the sampling of practice competencies tested) is not very defensible (percentage of A that overlaps with B).

View larger version (18K): [in this window] [in a new window]   Figure 1. Schematic representation of the overlap in content between competencies required of a dentist beginning practice (A) and the skills and knowledge tested on initial licensure examinations (B).

Validity—preferably predictive validity or content validity as the proportion of the target domain sampled in the test—should be the focus of any discussion about protecting the public. Improving the administration of the test or moving it to before graduation do not address these concerns. Improving examination reliability will not serve as an effective substitute. Figure 2 shows the relationship between predictive validity and reliability. The often quoted rule that reliability determines validity^24,37 is graphed here. The area under the curve depicts theoretically possible combinations of reliability and validity. When reliability is low, predictive validity also is low. When reliability is high, validity might be high or low. In cases in which tests are chosen to be objective or defensible by making them arbitrary or by using simulations, the drive for reliability is most likely to result in combinations of reliability and validity in the lower right-hand portion of the graph.

View larger version (25K): [in this window] [in a new window]   Figure 2. Relationship between reliability and predictive validity. All combinations of measured reliability and validity below the curve are theoretically possible.

In a simulation of state board examinations⁴⁰ using three live patients with two examiners each for crown preparations, reliability of any single test was estimated to be between r = .10 and r = .15. Increasing interrater consistency from the actual r = .68 to r = .99 would have increased decision consistency by about 1 percent; using two test cases instead of one would increase the consistency by more than 75 percent; three test cases would more than double the consistency.

	PORTFOLIOS AND COMPETENCY

TOP ABSTRACT THE ISSUE OF VALIDITY PORTFOLIOS AND COMPETENCY THE PORTFOLIO APPROACH PORTFOLIO EVALUATION AND... PORTFOLIO EVALUATION AND THE... INITIAL LICENSURE IS POLITICAL... CONCLUSIONS REFERENCES

 
Oftentimes, the term "portfolio" is used in the sense of a trace or work sample—a notebook with one’s work. Such items often are part of a portfolio, but the term has a larger meaning.^41–43 For example, consider your own investments. A good portfolio has diversity and balance that is proportional to risk. A portfolio is the best collection of evidence, with multiple replicates from diverse sources, chosen to reduce the risk of decision error. (The technical and operational definition is the evaluation design that maximizes the coefficient of generaliz-ability for a specific decision.) The investing analogy emphasizes multiple sources and often relies on the opinions of knowledgeable advisers, while the decision maker retains ultimate responsibility.

A portfolio is the best collection of evidence, with multiple replicates from diverse sources, chosen to reduce the risk of decision error.

In this section, I explore the connection between evaluation and level of learning. Initial licensure is a decision about competency.^44,45 Competency cannot be measured directly; it must be inferred.⁴⁶ The accuracy of that inference by state boards is the operational definition of validity.³⁵

We should avoid borrowing evaluation methods that work in one context (say, for beginning students) and assuming that these methods will work equally well in other contexts (for example, so-called "ethics tests," which actually measure knowledge of practice acts rather than propensity to practice ethically).^7–49 The table highlights the five stages that professionals go through in their 10- to 15-year growth from novice to expert (earlier versions of the table have appeared in other publications).^46,50,51 Learners’ performances at these various stages change, though under ideal circumstances, work products may show small differences. The best way to teach novices is through highly formalized lectures, but that is not the best way to teach experts. Similarly, the best way to evaluate professionals depends on the position of the candidate along the novice-expert continuum.

Competency can only be taught in realistic situations and appraised with authentic evaluation. As Messick²⁷ noted, "The evaluation of assessments should demonstrate that the level and sources of task complexity match those of the construct being measured and are attuned to the level of developing expertise of the students assessed." Authentic evaluation refers to performance that is judged by experts and represents performance expected of a candidate in realistic settings—settings that resemble the environments in which candidates are expected to be qualified to work should they past the authentic evaluation.^53,54 In dentistry, competency can be demonstrated only with live patients and with the results of managing the care of representative families of patients over an extended period. It requires an authentic portfolio. One-shots and simulations measure at the beginner level, but they do not measure competency.

Dentists in practice are expected to advance to higher levels of performance than competency. These include proficiency and expertise, or mastery. This expectation of continuous professional growth raises some issues regarding the notion of continued competency testing. Periodic reexamination seems inadequate to account for the growth that is expected of dentists in practice.⁵⁵ Specialists are poorly served when they are required to demonstrate competency with a set of skills they no longer use but not with those skills they profess and intend to use.⁷

The categories of authentic evaluation are test cases, ratings, and traces or naturally occurring outcomes of practice.⁴⁶ Initial licensure examinations are examples of test cases. They are samples of realistic performance under semistandardized circumstances in which no assistance is given to the candidate, but the patient’s health is protected. Ratings are professional judgments of characteristics of candidates’ performance observed over time by professionals qualified to make such judgments. Ratings also can be based on candidates’ defenses of case presentations. Traces are patient charts, records of cases completed, patient complaints, audits of infection control and emergency preparedness equipment, and any other evidence that results from patient treatment.

	THE PORTFOLIO APPROACH

TOP ABSTRACT THE ISSUE OF VALIDITY PORTFOLIOS AND COMPETENCY THE PORTFOLIO APPROACH PORTFOLIO EVALUATION AND... PORTFOLIO EVALUATION AND THE... INITIAL LICENSURE IS POLITICAL... CONCLUSIONS REFERENCES

 
Switching to portfolios is preferable to further attempts to increase the reliability of one-shot initial licensure. Evaluations that have known high validity can be "stepped up" to predetermined standards for reliability by using combinations of examinations. Aiming to increase reliability as the first goal normally sacrifices validity.

A portfolio is a collection of authentic evaluations, including repeated measures from several sources.^41–43 Portfolio evaluation is a four-step process. First, a responsible entity establishes the competencies expected of those who are evaluated. Next, the responsible entity establishes rules for what will be accepted as evidence. The third step is defining who is qualified to serve as a judge and how the judging will be conducted and reported. Finally, there needs to be consideration of logistics, timetables and due process. Something like a portfolio evaluation system is mandated in American dental schools by the accreditation requirements.

Establishing competencies. All dental schools have written definitions of the competencies expected of graduates. For example, the University of the Pacific School of Dentistry, San Francisco, has 59 competencies. They include such items as "restore single teeth for therapeutic reasons"; "determine differential, provisional, and definitive diagnoses"; "administer and prescribe medications commonly used in dentistry, including local anesthesia, and manage their complications"; "practice consistent with sound business principles and legal requirements and regulations"; "diagnose and treat only within one’s competence"; "assume active responsibility for one’s lifelong learning"; and "participate in organized dentistry." The ADA has developed competency and proficiency standards for the 14 categories of dental professional programs accredited by the Commission on Dental Accreditation.⁵⁶ The ADEA has approved a core set of competencies for both dentists⁵⁷ and dental hygienists.⁵⁸ State boards and testing agencies have yet to do this.

Identifying acceptable evidence. State boards generally have said that sufficient evidence for competency to begin practice includes written, standardized multiple-choice tests; live patient examinations in which the eligibility of the patient and the testing circumstances are carefully prescribed; simulations such as typodont exercises; certification of graduation; and documents showing no criminal history.

Typically, portfolio evaluation goes beyond this limited evidence. Candidates usually are given options, including "any other evidence deemed acceptable" to the examiners. In the strict testing case, the tester retains responsibility for proving whether the candidate is competent. In the portfolio approach, that responsibility is shared with the candidate. Well-known examples of portfolio evaluation include virtually all hiring, promotions in most organizations and selection for honors. Because evaluation for competency must be authentic, unnecessary standardization should be resisted.

Delegating data collection. The next step is to select qualified people to gather and evaluate the evidence and to charge them with developing a fair process for reaching their judgments. Evaluation and decision making normally are separate. State boards have deputized the ADA Joint Commission on National Dental Examinations to test didactic knowledge and law enforcement agencies to perform background checks. In only a handful of states have board members met candidates or seen their dentistry. Boards delegate one-shot testing to examining agencies under very carefully specified conditions.⁵⁹ They also could delegate multiple-method evaluation to dental schools under the same sort of carefully specified conditions.

Logistics. The final step in portfolio evaluation is to develop logistics, timetables, protocols for managing candidates and others, data and records procedures; and due process standards.

Dental schools have dental records for families of patients who have been treated in comprehensive patient care settings by students over extended periods. The schools can produce patients for examination and provide opportunities for case presentations and oral, patient-based examinations. Test cases can be conducted with varying degrees of standardization and preparation. The dental school could be thought of an as in-office visit with performance requirements.

State boards develop the protocol for managing portfolio evaluation. Once it is known what the competencies are, where the evidence is and who is qualified to judge that evidence, the details of the process can be developed. Most likely, candidates will assemble their portfolios in the final six months of their dental school educations, and visits by board members and their external delegates then will begin and continue until graduation. Some candidates would be judged by boards to be competent to practice on graduation, while others whose competencies must improve or who require one-shot, standardized performance testing or other special evaluations would be appraised shortly before graduation or at a time when they meet the qualifying standards.

	PORTFOLIO EVALUATION AND RELIABILITY

TOP ABSTRACT THE ISSUE OF VALIDITY PORTFOLIOS AND COMPETENCY THE PORTFOLIO APPROACH PORTFOLIO EVALUATION AND... PORTFOLIO EVALUATION AND THE... INITIAL LICENSURE IS POLITICAL... CONCLUSIONS REFERENCES

 
Figure 3 shows a schematic view of the competency domains addressed by initial licensure and dental education and what is required to practice dentistry. Naturally, there will be arguments about the details of this representation. However, there probably will be universal agreement that there is substantial overlap, with practice demanding more than education and education covering more than initial licensure testing. The parts of practice not covered in education or initial licensure testing may not be problematic because dentists are excellent on-the-job learners. Competent beginning dentists have become proficient and even expert practitioners in predictably large numbers.^60,61

View larger version (19K): [in this window] [in a new window]   Figure 3. Schematic representation of the overlap in content between competencies required of a dentist beginning practice (A), competencies taught and measured in dental schools (C), and skills and knowledge tested on initial licensure examinations (B).

Annually, about 3 to 4 percent of students studying in American dental schools are dismissed, with no opportunity to return.⁶² The ADA 2000 Survey of Dental Graduates⁶³ places the number of graduates who are licensed and professionally active within one year of graduation at 97 percent. Each year, the public makes complaints against 9 percent of American dentists, and five in 1,000 licensed dentists are disciplined.⁶⁴ Disciplined dentists represent double false-negative decisions. The schools should not have graduated them, and the boards should not have passed them. In some jurisdictions, practitioners who are disciplined for incompetent practice are remediated by means of having to receive specific hours of education, with dental schools having the final say over whether practice privileges should be restored.

Pass rates on various licensure examinations and the number of newly licensed dentists compared with graduates are recorded by the ADA.⁶⁵ Over the past several years, the national first-time pass rate on the one-shot examination has been 77 percent. The difference between 77 percent and the 97 percent of graduates who are licensed within a year is a result of candidates taking examinations in more than one jurisdiction or taking the tests several times. Taking tests in multiple jurisdictions also accounts for the difference between the 95 percent pass rate reported by examining agencies⁶⁶ and the overall 97 percent reported by the ADA.⁶³

It should be intuitive that a testing system that passes virtually everyone after three attempts would have weak reliability. Next, I focus on this idea, with psychometric precision based on classical test theory regarding reliability.^{24,26,67–69} Although I do not present the full details, the discussion is precise enough to demonstrate the insight and the power provided by classical measurement theory tools.

The overall reliability of all one-shot initial licensure examinations given in the United States is approximately .40, meaning that no more than 16 percent of the variance in competence to begin independent dental practice is available in a consistent fashion for those who make licensure decisions. Such a coefficient is low compared to similar high-stakes and professional licensure examinations.^23,29,30,70

This estimate is derived from the fact that reliability is the square root of the ratio of true variance to total variance. These variances are known in the case of one-shot initial licensure examinations because the variance of pass-fail tests is the product of the pass rate multiplied by failure rate. Total variance is the true variance plus the error variance, or the variance on a single examination administration. Seventy-seven percent of candidates pass each test. The true pass rate is approximated at 97 percent—the proportion who eventually earn licensure. The square root of (.97 x .03)/(.77 x .23) is .40.

One-shot initial licensure examinations typically contain four or five sections. The Spearman-Brown formula can be used to determine that each section has an average consistency in the range of r = .12 to .15.²⁴ This is remarkably similar to the estimates obtained by Chambers and Loos⁴⁰ for simulations of a single section of an initial licensure examination in crown preparations.

It is difficult to reconcile the numbers on overall initial and eventual pass rates for initial licensure examinations with the explanation that candidates learn between administrations of these tests. If candidates pass each of three administrations of the tests at the same rate (77 percent), the cumulative pass rate of this series is 97.8 percent. This is essentially the same as the 97 percent overall national pass rate. The ultimate pass rate can be explained entirely in terms of retesting, with no need to assume that learning is occurring. The actual second-time national pass rate is 75 percent.⁶⁵

Figure 4 shows the theoretical relationship between reliability and validity. The letter X appears approximately where the initial licensure process might be currently with respect to reliability. What matters is how we propose to change this by any reasonable modification of the process. Simulations, elimination of patients, moving one-shot tests into schools or any restriction of the system for the sake of objectivity will move the X along the path labeled S for simulation.^10,71,72 The Objective Standardized Clinical Evaluation is not properly a level of testing. It is a format for presenting items, which may be authentic, but usually are simulations. A better alternative is portfolios—multiple replicates of multiple sources of evidence regarding the range of skills, understanding and values required to start independent practice.

View larger version (31K): [in this window] [in a new window]   Figure 4. Schematic representation of current initial licensure examination reliability and validity and potential alternatives. X: Approximate current reliability and validity. S: Change to approaches based on simulation (beginner level). P: Changes to multiple replicate, multiple source authentic evaluation (portfolios).

	PORTFOLIO EVALUATION AND THE INITIAL LICENSURE DECISION

TOP ABSTRACT THE ISSUE OF VALIDITY PORTFOLIOS AND COMPETENCY THE PORTFOLIO APPROACH PORTFOLIO EVALUATION AND... PORTFOLIO EVALUATION AND THE... INITIAL LICENSURE IS POLITICAL... CONCLUSIONS REFERENCES

Step one. Each licensure jurisdiction (not each testing agency) should start by defining the competencies necessary to begin the independent practice of dentistry. This step normally can be accomplished in a few weekends, each separated by a few months.⁷³ Jurisdictions that generate different competency sets logically would want to use different evaluation systems. The competencies become the evaluation blueprint. Evidence would include candidates’ possessing understanding, skills and supporting values for every aspect of dentistry defined by the boards as being essential to independent dental practice.

A common mistake has been to make licensure decisions with only a general notion of what is required of practitioners. Guidelines for Valid and Reliable Dental Licensure Clinical Examinations³⁵ was based on surveys of some of the most frequently performed procedures and opinions about what would be easiest to test. That work is useful, but it is incomplete to protect the public.

Step two. State boards define what evidence they will need to make their decisions about licensure and what standards this evidence must satisfy to be accepted. Boards should be able to justify their decisions based on evidence about the quality of their decisions. (Several boards are under court-ordered monitoring precisely because they have not been able to provide this assurance.) Boards might say, for example, that candidates are required to present results of written and patient-based examinations, evidence of satisfactory completion of various tasks that dentists perform, and records of ethical conduct that collectively demonstrate competence in all defined areas with a reasonable probable error rate. Note that the responsibility of the candidate is increased, and the board is controlling the quality of its decisions more precisely. (It is a technical matter, but probable error²⁴ is a measurable function of three components: validity [addressed through competency statement blueprints], reliability [addressed through portfolio construction] and measured candidate performance.)

A common mistake is to aim for the single best testing approach possible. The best may not be good enough, as educators and examiners would be quick to tell an incompetent but motivated candidate. An evaluation plan should be designed by putting together the variety of evidence required. When requisite reliability cannot be achieved by satisfactory consistency in the existing evaluation (for example, standardization of conditions, calibration of examiners or clarity of interpretation of standards), it can be achieved by increasing the number of evaluations.⁷⁴ Test theory has shown that consistency of evaluation can be achieved in two ways: increase internal consistency or increase the number of replications.^{24,26,38–40,68,69,75,76} These approaches are psychometrically interchangeable, meaning that committing to a one-shot testing scheme in advance unnecessarily compromises the quality of licensure decisions.

Here is an example of crafting a portfolio designed to reach a predetermined, acceptable level of confidence. Assume that we are locked into an average reliability for each piece of evidence at r = .15. The portfolio consists of data gathered in dental school under the supervision of boards of dentistry. It includes two endodontic test cases and two crown placements, three comprehensive periodontal management cases, a removable prosthesis and a surgery case, three randomly selected chart reviews and defense of three completed cases with predetermined parameters. This portfolio has a projected reliability of r = .82, even though each part is reliable at the r = .15 level. It explains two-thirds of the variance and is four times as consistent as the average one-shot initial licensure testing system. If evidence of case management, ethical conduct, diagnostic skill and professional involvement were added, the reliability and validity of the licensure decision would be higher. (The psychometric foundation for these calculations is found in classical test theory,^24,26,69 the Spearman-Brown formula²⁴ and powerfully in generalizability theory.^75,76)

Step three. The concept of delegated authority rests on differentiating examination from licensure or distinguishing between the test and the decision. State boards delegate the gathering of almost all the evidence on which they base licensure decisions, and that system has proven to be workable.⁵⁹ What boards need to ensure in their delegation processes is that standards exist, that they are monitored and made available to those who need to know about them, and that they are used to manage the relationship between boards and those to whom they delegate. Organizations that perform assessments should be selected based on their performance. There is a tremendous need for boards to make their standards known and to share outcomes data. A chair of the examination committee for the Northeast Regional Testing Boards said that there are no acceptable statistical models to determine reliability of clinical examinations.⁷⁷ It is possible that such factually inaccurate statements prompted the ADA to consider studies of predictive validity at several House of Delegate meetings.

Higher-quality data now are available at almost no additional cost to the licensure decision process.

There is risk associated with the view that only those who have examined candidates can decide who should be licensed. That approach is not applied uniformly across all evidence boards used. It would be interesting to speculate what sources of evidence would be selected if we started over today and asked those with relevant data to compete based on predetermined standards.

Step four. Cost and fairness are especially important in logistics considerations. The argument seems sound that the cost of one-shot licensure examination (for example, fees, travel, patient concerns) is small relative to the magnitude of the decision being made. That same argument might be extended to justify the collection of the additional data needed to bring the reliability of the decision into the r .80 range that is considered acceptable by other professional licensing groups.

There are two ways that acceptable reliability can be achieved while actually lowering costs. First, higher-quality data now are available at almost no additional cost to the licensure decision process. These are the data normally collected by highly calibrated faculty members in dental schools. If dental schools demonstrate that they meet the evaluation standards set by boards, just as testing agencies do, they should be able to provide the needed information at a reduced cost. They also can provide information that testing agencies have eschewed.

Because of legal challenges to the one-shot testing system, boards and agencies have placed barriers between themselves and the candidates that restrict access to evidence other than the work product. Subjective determinants of practice such as character, ethics, patient care management and case management are the primary causes of action against licensed dentists, but they are not part of the one-shot format. Subjective data are valid in any decision in which such data will be used to judge performance in practice,^78,79 and they have shown good predictive validity in dental education.⁸⁰ In a series of court decisions, including one by the U.S. Supreme Court, the right of educators to make such judgment decisions has been established. Horowitz v. Curators of the University of Missouri stated that educational evaluation is inherently subjective and that courts will not substitute their decisions for the decisions of educators who are qualified to make such judgments.⁸¹

The second way that in-school assembly of portfolio evaluation information reduces cost is that there is a longer period over which the evaluation can be spaced.⁶⁵ Testing everyone on a weekend after graduation may be superfluous in the cases of some candidates, but it certainly provides too little information in the cases of candidates who are near the cut-off score. Ambiguous results should be explored until they can be resolved. The rule should be to test until a predetermined acceptable level of confidence is established that the candidate either is or is not competent. The level of confidence in protecting the public should be predetermined and not varied; instead, the amount of evidence needed to reach this standard should be varied. If some of the testing takes place in dental schools, the amount of postgraduation testing can be reduced substantially, hence lowering the overall cost of initial licensure examination.

The ultimate criterion for fairness in initial licensure decisions is a high level of predictive validity. It is patently unfair to design examinations to protect testing agencies. It is the public boards are empowered to protect. Fortunately, alternatives to one-shot practical tests, such as the portfolio system, are fairer in both the details of test administration and in the likely validity of licensure decisions.

Because state boards are strapped for resources, they have delegated the data-gathering portion of their responsibility but not their decision-making responsibility regarding initial licensure to regional testing agencies. The use of portfolios does not require expenditure of any further resources by boards because portfolios are a richer and alternative way of gathering data. The use of portfolios by regional testing agencies can be mandated by state boards or mediated and delegated in part to dental schools.

	INITIAL LICENSURE IS POLITICAL AS WELL AS RATIONAL

TOP ABSTRACT THE ISSUE OF VALIDITY PORTFOLIOS AND COMPETENCY THE PORTFOLIO APPROACH PORTFOLIO EVALUATION AND... PORTFOLIO EVALUATION AND THE... INITIAL LICENSURE IS POLITICAL... CONCLUSIONS REFERENCES

 
It is unlikely that initial licensure decisions will be made differently in the future. Alternatives are available now that outperform the current system in terms of reliability, validity, cost and fairness. Licensure is a political decision. Boards recently have been sunsetted in Texas and California, and legislatures have found ways around one-shot tests in New York, California and North Dakota. Boards in Hawaii and Florida are under court-appointed scrutiny, and the ADA and ADEA have passed policy resolutions calling on boards and testing agencies to discuss alternatives to the present system. It is, however, unlikely that any appreciable change will take place based on what is known about psychometrics or the science of making sound decisions.

It is necessary to inventory what interests might be threatened by a portfolio system that substantially increases the soundness of initial licensure decisions and does so at less cost and with greater fairness than the current system.

Marginal dental students may object to such a system. Decisions that are based on repeated trials of a test with low reliability produce substantial numbers of false-positive decisions when the results of early trials are not considered. Candidates who fail one-shot tests are given multiple opportunities to prove that the initial decision was wrong; those who pass are not re-examined to prove that the decision was correct. The current system appears to protect more adequately the rights of the candidate than the rights of the public.

Some dental educators will find fault with portfolios on the grounds that dental boards will be coming into the schools. Boards and schools, however, are in the same business, and they can be more effective working together than independently. It may be inconvenient for schools to open their records to examiners, and it may be uncomfortable for them to discuss frankly how they determine competency. However, it will be better for the profession and the public if this is done. In the end, examiners can improve the quality of dental education just as educators can provide information about candidates that strengthens the boards’ charge of protecting the public.

Regional testing agencies might be jealous of any new partner that the boards would include. If they are, they should state why. The matter of trusting our professional partners to be free of conflicts of interest is a challenge that goes both ways. It is better to define operationally what must be done to ensure a working relationship than to begin with an untested assumption that others cannot be trusted.⁷¹

The greatest objection to portfolio evaluation will come from those who are unwilling to stand behind their professional judgments. Perhaps a few will have read this entire article without believing the statement "It is not the test that matters: it is the decision." Competence to join a profession is determined by comprehensive judgment. No one but the members of state dental boards can make such judgments. They have available a much broader range of quality evidence on which to make such judgments than they have used so far. The public would welcome the additional level of protection afforded them through expanding the portfolio of initial competency evidence.

	CONCLUSIONS

TOP ABSTRACT THE ISSUE OF VALIDITY PORTFOLIOS AND COMPETENCY THE PORTFOLIO APPROACH PORTFOLIO EVALUATION AND... PORTFOLIO EVALUATION AND THE... INITIAL LICENSURE IS POLITICAL... CONCLUSIONS REFERENCES

 
A portfolio process for evaluating professional competence to begin practice increases the scope and amount of data available to examiners while reducing cost. At no time would boards relinquish their control over defining what competency is, what evidence is acceptable for demonstrating competence, who will judge competence, what standards they must meet and how the process is to function. What will be reduced is the cost of producing all the sources of evidence and the restrictions that costs and tradition have placed on the range of information considered.

The portfolio approach meets the seven criteria for an effective system I discussed at the beginning of this article. Portfolios will reduce the number of false-positive licensure decisions by providing more valid evidence than the one-shot, restricted range test does. Using a portfolio approach is preferable to abandoning live patients, substituting advanced education as a bypass or using simulations. At every stage, boards control the licensure process as they take advantage of all the partners who are able to help in this responsibility.

View larger version (127K): [in this window] [in a new window]   Dr. Chambers is associate dean for academic affairs and scholarship, School of Dentistry, University of the Pacific, 2155 Webster St., San Francisco, Calif. 94115, e-mail "dchambers{at}pacific.edu". Address reprint requests to Dr. Chambers.