At its July 4, 2023, meeting, CPATA’s Registration Committee made decisions regarding how the standards for demonstrating competency will be determined for the new qualifying examinations. Starting in 2024, standards for demonstrating competency on the exams will be determined after the tests are written, based on both subject matter expertise and candidate performance data.
Through the competency initiative and the exam redevelopment process, CPATA aims to:
- Establish what knowledge and skills entry-level practitioners require in order to be competent practitioners
- Create a fair qualifying examination process that tests that these competencies have been acquired
We do not aim to make the exams ‘easier’ but rather to ensure that they measure the right thing, in the right way.
The changes we have been making to the licensure process are likely to improve candidates’ ability to learn the competencies necessary for entry level practice, and to study and prepare for the examinations. They promote public protection and fairness by allowing for a more precise determination of who is ready to be fully licensed based on the technical competency profiles. An increased pass rate may be a consequence, but is not a goal, of the exam redevelopment process.
Specifically, the Modified Angoff method will be used to determine the standard for demonstrating competency on the knowledge examinations and the Borderline Regression method will be used to determine the standards for demonstrating competency on the skills examinations. These standard-setting processes are considered best practice for high-stakes examinations and are more precise and fairer because they reflect the difficulty of each exam administration.
What standard will candidates need to meet to be successful on the new qualifying examinations?
Based on the standard setting analyses conducted by psychometric experts, CPATA’s Registration Committee will approve the standard for demonstrating competency for each exam administration, after the exams are written. Starting in 2024, exam performance will be determined after the tests are written, based on both subject matter expertise and candidate performance data. CPATA will use both test-centred and candidate-centred methods to determine the appropriate fair standard to be successful on each particular administration of the exam. These standard-setting processes are considered best practice for high-stakes examinations and are more precise and fairer because they reflect the difficulty of each exam administration.
What are the Modified Angoff method and Borderline Regression methods?
Modified Angoff method: In this test-centred method, a panel of subject matter experts (SME panelists, in this case, practising licensees) estimates the probability that a minimally competent candidate would answer each item on the test correctly. Several rounds occur, with panelists able to adjust their estimates given different information. The aggregate of these estimates informs the overall minimum standard. The specific standard used to determine “competent/not-yet-competent” decisions would be approved following the exam administrations. This ensures that the standards for demonstrating competency are set with the benefit of the statistical performance of the items and the test overall.
Borderline Regression Method: In this candidate-centred method, SME panelists (in this case, the examination advisory group members who are practising licensees) categorize candidates into 3 or more categories, of which one is a “borderline” category (e.g., below the standard, borderline, above the standard) but incorporates a linear regression approach. The specific standards used to determine “competent/not-yet-competent” decisions would be approved following the exam administrations. This is because the standard can only be determined using the data obtained during the examination process. Because the specific content of the skills examinations changes with every administration, resulting in potential changes to the level of difficulty, Borderline Regression analysis will be conducted following each exam administration.
Why were these methods selected for the new examinations?
There is no single method to determine a defensible entry to practice standard for all tests or purposes; different methods are suited to different circumstances. As a licensing body, CPATA needed to select an approach to standard setting based on the purpose of the examinations, which is to assess entry level competencies, as well as how closely the method aligned with assessment goals.
For a high-stakes licensing examination (in other words, a test with important consequences), the selected standard-setting approach should relate to the purpose of the examination, be appropriate to the level of complexity of the competencies to be assessed by the examination, be appropriate to the specific format of examination (e.g., multiple-choice, performance-based), be supported by scholarly research supporting its use, and (for practical reasons) should reflect the availability of resources (e.g., money, time, expertise in statistical analysis, availability of experts).
The Modified Angoff method:
- Is the most well-studied and documented standard setting method;
- Is the most commonly used method for standard setting in credentialing contexts; and
- Uses a standards-referenced approach, which means that the standard reflects the expectations of the profession.
The Borderline Regression method:
- Allows the standard to be set using test performance from all test-takers, not just a subset from the “borderline” test-takers;
- Is particularly suited to small-scale assessments because it uses data from all examinees in the derivation of the minimum standard (as opposed to only borderline examinees); and
- Is an efficient use of the SME’s time because the evaluation occurs at the time the examination is being marked.
Will the change in pass mark setting lead to more people passing or failing the exam?
We do not aim to make the exams ‘easier’ but rather to ensure that they measure the right thing, in the right way. The changes we have been making to the licensure process are likely to improve candidates’ ability to learn the competencies necessary for entry level practice, and to study and prepare for the examinations. They promote public protection and fairness by allowing for a more precise determination of who is ready to be fully licensed based on the technical competency profiles. An increased pass rate may be a consequence, but is not a goal, of the exam redevelopment process.
Are there opportunities to participate in the standard setting process?
Yes! Later this year, CPATA will be recruiting practising Class 1 and 2 licensees to participate as subject matter expert panelists for the March 2024 knowledge examinations.
Questions?
Please contact registration-inscription@
