Introduction
Latest developments in generative AI allow certification bodies for persons to employ this technology to create new schemes, examinations, assessments, and more. Generative AI not only simplifies and speeds up the organizational tasks involved but also could reduce costs. AI-generated new certification programs may help generate new business and provide access to disadvantaged groups and populations. It may also usher in a new way of evaluating the competence of people that is not possible through traditional assessments.
Certification bodies interested in using generative AI will need to consider areas of use along with concerns and safeguards necessary to implement an effective system.
Scheme Development and Job Analysis
A certification scheme based on a valid job or practice analysis identifies the knowledge, skills, and attributes required for successful job performance. This process is extremely resource and time intensive. It is not uncommon for certification bodies to spend months if not years on the development of a new scheme. Ongoing validation of the scheme to maintain relevancy is also challenging. Consequently, a vast majority of certification bodies validate their schemes every five years or so. This is not a very efficient process, as the competence requirements today are changing at a much faster rate. To expedite this process, generative AI could be used in conducting background job description/role delineation and literature review. Starting with basic prompts to customized, pretrained models using proprietary data, certification bodies can unlock new frontiers in scheme development.
Even while using generative AI, subject matter expert participation in scheme development is required to ensure all elements of clause 8 of ISO/IEC 17024 are met. Furthermore, there is a need to independently validate the work of AI generated content.
Exam Development and Assessments
Despite the limitations of traditional testing, assessments are a necessary component of certification of persons. Fair, valid, and reliable assessment ensures that people who are certified have demonstrated competence to practice in the field.
There are several types of AI assessment methods. First is a rule-based AI which uses decision making rules to evaluate competence requirements. Second is a machine learning-based AI which is more powerful and uses large multilayered data sets on candidate performance. As a matter of fact, the AI model can also input data sets of practitioners in the field and predict successful performance. This form of assessment gets better over time with more training on test taker datasets.
Generative AI can help certification bodies go beyond multiple-choice examinations to other types of assessments that includes work samples, portfolio assessments, essays, oral examination. Many certification bodies have been reluctant to use these types of assessment methods due to the difficulty in supporting these types of exams with psychometric analysis on account of the need to use examiners. Natural language processing using AI and other prediction systems that use large scale trained data could be a good alternative to overcome these limitations. Vision-based AI could also be used for assessment. This could be particularly helpful for people who are visual learners and have difficulty with traditional testing modalities. Voice based assessments can also serve as an important tool to assess soft skills and work readiness. Another example of natural language processing involves the use of automated essay scoring to grade written examination. Stealth assessment that uses evidence centered designs to create models in conjunction with data mining techniques is also gaining in popularity to measure real world competencies (Shute, Lu and Rahimi, 2022).
Many certification bodies are challenged by the current manual process of exam item development. This process involves assembling subject matter experts, training them in item writing and spending countless hours in item development. In addition, the new items must be piloted with test takers followed by psychometric analysis to establish their reliability and efficacy. Consequently, exam item banks are very expensive to develop and maintain. Furthermore, a single security breach can render the entire bank useless.
It is now possible to develop examinations using fewer subject matter experts with AI assistance. Generative AI could be used for exam generation, exam item refinement, duplicate item detection and more. In addition, AI can generate exam questions that are personalized to each candidate’s needs. However, AI cannot fully replace subject matter experts. It is a new tool with tremendous benefits. A tool to be used by subject matter experts and not the other way around.
Application Review/Eligibility Checks
Another potential area for generative AI is to review prerequisites (if applicable) for a certification program. Review of applications and objective evidence that demonstrates the meeting of prerequisites has been a challenge for large volume certification programs. Many certification bodies review only a certain percentage of applications to confirm meeting eligibility requirements. This is typically done through a very intensive time-consuming process using human reviewers. The review can be even more challenging when work samples or educational transcripts are submitted in different languages. These limitations can be overcome by generative AI that can be trained to review prerequisite fulfillment by the applicants. It is important to constantly review and validate the performance of AI with documented evidence.
Streamlined Operations and Logistics
Generative AI can nimbly manage logistics, from registration, enquiries, and complaint handling to result delivery. Furthermore, innovations like facial recognition, plagiarism detection, large item banks, and virtual proctor assisting can improve test security and reduce operational costs.
Addressing AI Concerns
Understandably, it is important to address the concerns and issues related to the use of AI in certification of persons. These concerns include the following:
- Technology Ethics and Algorithmic Bias
The data that is used by AI models contains built in data from the worldwide web. The incoming data bias produces output bias that breeds lack of trust and perpetuates past inequities and gaps in access. It is also important to watch for AI “hallucination” whereby a system generates false output resembling a true statement.
- Transparency and Explainability of Autonomy
Certification bodies need to ensure that they are transparent in disclosing the use of AI. A key component of transparency is to make publicly available how AI is being used in different certification activities. What models are being used and how do these models derive their conclusions? What data is used to build AI models? Addressing these questions is the quintessence of explainability of autonomy. This is important to generate trust in the AI results.
- Privacy and Fairness
Certification bodies need to ensure that they are meeting all statutory regulations related to collection, use, and sharing of personal information. It is also important to ensure that information collected is necessary and appropriate for the intended purpose.
- Safety, Security, and Reliability
Like any new technology, generative AI comes with known and unintended risks that require accountable design, risk mitigation, and safe application to ensure responsible innovation. As an example, an examination developed by an open-source AI would not be copyrighted by the U.S. Patents Office. Similarly, a lot of content that is generated by AI might be copyrighted and its use could violate applicable laws.
- Supervision and Oversight
While generative AI is a great tool to assist in various certification activities, it is not meant to replace humans. There is a need for human oversight and supervision. All certification decisions must be made by the certification body and not AI. Similarly, subject matter experts must be the final authority for approval of the scheme and assessments.
How Certification Bodies Can Benefits from AI
First, certification bodies need to deepen their understanding of AI. In addition, these organizations need to invest in research to develop better models in areas they want to deploy AI.
Second, certification bodies should invest in independent technology options. It is not recommended to use off-the-shelf tools without proper testing and customization.
Third, certification bodies should ensure that they have personnel who have the required competence in AI.
Conclusion
Certification bodies should leverage the immense benefits of AI to assess competency of persons. From scheme development to assessments, intelligent monitoring, stealth assessments, games, and virtual reality, there are wide variety of ways to use this technology.
One way to approach the use of generative AI in certification of persons is to treat it as “outsourced” work. A certification body must demonstrate compliance to all of the requirements under 6.3.2 of ISO/IEC 17024 and:
- take full responsibility for all AI work;
- ensure that the personnel/body conducting AI work is competent and complies with the applicable provisions of ISO/IEC 17024;
- assess and monitor the performance of the personnel/bodies conducting AI work in accordance with its documented procedures;
- have records to demonstrate that the personnel/bodies conducting AI work meet all requirements relevant to the outsourced work;
- maintain a list of the bodies conducting AI work.
Past experiences with new technologies like computers have shown that initial fear about how these technologies will replace humans is eventually replaced with awareness of how we can leverage them to improve the quality of certification. At the end of the day, generative AI is a new tool that provides exciting opportunity to reimagine a more robust, creative, and equitable way to assess competence of persons.
Finally, “the use and adoption of AI must be human-centric and grounded in human rights, inclusion, diversity, and innovation, while encouraging sustainable economic growth” (The Global Partnership on Artificial Intelligence).