Virtual Reality Assessment for Personnel Certification Bodies under ISO/IEC 17024

Virtual Reality (VR) has been applied to different types of training in many industries, from product design to operating equipment and facilities. While the use of VR in training is established, can it be used to evaluate knowledge, skills, and abilities for a personnel certification program that meets the international standard ISO/IEC 17024:2012, General requirements for bodies operating certification of persons?

What is Virtual Reality (VR)?

Virtual Reality (VR) refers to the variety of technological computer-generated experiences that produce a human-machine simulation of visions, sounds, and other perceptual concepts. Similar terms include artificial reality, virtual environment, and virtual interface. The central idea behind VR is to mimic a real experience and make it as close to reality as possible. VR training provides tremendous advantages, especially in learning new skills that require experimentation, design, or tinkering or that are too dangerous to practice in real life. For instance, VR training is especially advantageous on construction sites, where the rate of accidents is double the industrial average caused primarily because of lack of safety knowledge and skills.

Assessing Competence With VR Examinations

ISO/IEC 17024 does not require a certification body to assess competence only through a written, oral, or practical examination. The standard uses the term “examination” in a broad context and includes any means of assessment provided it is reliable and objective (9.3.1). In other words, the standard does not explicitly prohibit VR examination, provided the certification body meets all applicable requirements in the standard.

If a certification body wants to use VR assessment, this specific assessment method must be defined in the certification scheme (9.2.1) and developed as per clause 8 of ISO/IEC 17024. The certification scheme based on a valid job/task analysis is the foundation of a certification program (8.4e).

The subject matter experts (SMEs) who are involved with the development of the scheme must consider whether the use of VR is appropriate for the scope of the certification and assessment of competence and abilities. The scheme must consider and specify if VR assessment would be used for both initial certification and recertification or only for initial certification.

Furthermore, the expert panel should involve, in addition to SMEs in the scope area, other appropriate experts, particularly those with VR technology and measurement/psychometric expertise.

Challenges with VR Assessments

It is important to clearly define the structure that represents the interest of all parties significantly concerned without any interest predominating (8.4b). This can be a challenge particularly as it relates to VR, as most experts developing the certifications scheme or developing the assessment are likely to be trainers. Hence, it is important to have SMEs in the job task analysis who are practitioners in the field rather than an over-representation of trainers. A validation survey would also help to uncover any tasks or competence criteria that might have been overlooked by the expert panel. It is important to define the frequency of updating the scheme given the rapid changes happening with VR in general. 

Another requirement in the ISO/IEC 17024 standard that must be included in the development of the scheme relates to the alignment of the VR assessment with the competence criteria (8.4d). All the content domains as laid out in the examination blueprint must be assessed using the VR modality. This also requires a certification body to determine how to develop the scoring rubric to prevent candidates from guessing. This is important, as the risk or accidents that could result from guessing is zero in a VR environment as compared to an examination at a real work location.

If the certification body offers training, the 17024 standard’s requirement to separate training from testing could be a challenge when VR is used for assessment. Individuals who create simulations for training/education must not be involved in creating the VR assessment. The certification body would need to develop sufficient safeguards such that the training processes are independent to ensure confidentiality, information security, and impartiality (5.2.3).

Examination security could be a challenge if the VR platform does not contain the required encryptions and security protocols. A certification body that intends to use VR must provide evidence to support that the assessment is valid and fair to all the applicants (9.2.4). For instance, if the target population does not have access to a computer or other technology platform to take the test, this could pose significant challenges. Other factors that must be considered include accommodation of special needs within reason and any other regulation such as privacy. 

VR Assessments for ISO/IEC 17024 Accreditation

A certification body that intends to use VR for assessment should consider developing practice exams that provide familiarity to the candidates. The VR assessment should not be contingent on a candidate’s ability to use such technology; rather, it should assess their competence to perform in a real-world environment.

Provided certification bodies consider all elements of the 17024 standard, VR assessment could play a valuable role in assessing skills and abilities. ANAB recently accredited Industrial Training International under ISO/IEC 17024 for its Construction Hazard Identification certification, a program that uses VR assessment. Perhaps VR assessment will become more widely used as certification bodies continue to explore new ways to incorporate technology in assessing competence.