At its June 13, 2023, meeting, CPATA’s Registration Committee made decisions regarding how the pass marks will be determined for the new qualifying examinations. Starting in 2024, exam pass marks will be determined after the tests are written, based on both subject matter expertise and candidate scoring data.
Through the competency initiative and the exam redevelopment process, CPATA aims to:
- Establish what knowledge and skills entry-level practitioners require in order to be competent practitioners
- Create a fair qualifying examination process that tests that these competencies have been acquired
We do not aim to make the exams ‘easier’ but rather to ensure that they measure the right thing, in the right way.
The changes we have been making to the licensure process are likely to improve candidates’ ability to learn the competencies necessary for entry level practice, and to study and prepare for the examinations. They promote public protection and fairness by allowing for a more precise determination of who is ready to be fully licensed based on the technical competency profiles. An increased pass rate may be a consequence, but is not a goal, of the exam redevelopment process.
Specifically, the Modified Angoff method will be used to calculate the pass mark required for the knowledge examinations and the Borderline Regression method will be used to calculate the pass mark for the skills examinations. These standard-setting processes are considered best practice for high-stakes examinations and are more precise and fairer because they reflect the difficulty of each exam administration.
What pass mark will candidates need to be successful on the new qualifying examinations?
Based on the pass mark analyses conducted by psychometric experts, CPATA’s Registration Committee will approve the pass mark for each exam administration, after the exams are written. Starting in 2024, exam scores will be determined after the tests are written, based on both subject matter expertise and candidate scoring data. CPATA will use both test-centred and candidate-centred methods to determine the appropriate fair pass mark to be successful on each particular administration of the exam. These standard-setting processes are considered best practice for high-stakes examinations and are more precise and fairer because they reflect the difficulty of each exam administration.
What are the Modified Angoff method and Borderline Regression methods?
Modified Angoff method: In this test-centred method, a panel of subject matter experts (SME panelists, in this case, practising licensees) estimates the probability that a minimally competent candidate would answer each item on the test correctly. Several rounds occur, with panelists able to adjust their estimates given different information. The aggregate of these estimates informs the overall pass mark. The specific pass marks used to calculate the “pass/fail” or “competent/not-yet-competent” decisions would be approved following the exam administrations. This ensures that the pass marks are set with the benefit of the statistical performance of the items and the test overall.
Borderline Regression Method: In this candidate-centred method, SME panelists (in this case, the examination board members who are practising licensees) categorize candidates into 3 or more categories, of which one is a “borderline” category (e.g., failing, borderline, passing) but incorporates a linear regression approach. The specific pass marks used to calculate the “pass/fail” or “competent/not-yet-competent” decisions would be approved following the exam administrations. This is because the pass mark can only be calculated using the data obtained during the examination process. Because the specific content of the skills examinations changes with every administration, resulting in potential changes to the level of difficulty, Borderline Regression analysis will be conducted following each exam administration.
Why were these methods selected for the new examinations?
There is no single method to determine a defensible pass mark for all tests or purposes; different methods are suited to different circumstances. As a licensing body, CPATA needed to select an approach to standard setting based on the purpose of the examinations, which is to assess entry level competencies, as well as how closely the method aligned with assessment goals.
For a high-stakes licensing examination (in other words, a test with important consequences), the selected standard-setting approach should relate to the purpose of the examination, be appropriate to the level of complexity of the competencies to be assessed by the examination, be appropriate to the specific format of examination (e.g., multiple-choice, performance-based), be supported by scholarly research supporting its use, and (for practical reasons) should reflect the availability of resources (e.g., money, time, expertise in statistical analysis, availability of experts).
The the Modified Angoff method:
- Is the most well-studied and documented standard setting method;
- Is the most commonly used method for setting cut scores in credentialing contexts; and
- Uses a standards-referenced approach, which means that the pass mark reflects the standard and expectations of the profession.
The Borderline Regression method:
- Allows the pass mark to be set using test performance from all test-takers, not just a subset from the “borderline” test-takers;
- Is particularly suited to small-scale assessments because it uses data from all examinees in the derivation of a cut score (as opposed to only borderline examinees); and
- Is an efficient use of the SME’s time because the evaluation occurs at the time the examination is being marked.
Will the change in pass mark setting lead to more people passing or failing the exam?
We do not aim to make the exams ‘easier’ but rather to ensure that they measure the right thing, in the right way. The changes we have been making to the licensure process are likely to improve candidates’ ability to learn the competencies necessary for entry level practice, and to study and prepare for the examinations. They promote public protection and fairness by allowing for a more precise determination of who is ready to be fully licensed based on the technical competency profiles. An increased pass rate may be a consequence, but is not a goal, of the exam redevelopment process.
Are there opportunities to participate in the standard setting process?
Yes! Later this year, CPATA will be recruiting practising Class 1 and 2 licensees to participate as subject matter expert panelists for the March 2024 knowledge examinations.
Questions?
Please contact registration-inscription@
