A topic that generates a lot of confusion, questions, and non-conformities in the Certificate Accreditation Program (CAP) accreditation process are three foundational practices that may appear the same but are quite different. The first practice addresses program execution, the second addresses program impact, and the third addresses program review. These practices are given names in the ASTM E2659 Standard:
- Internal audit
- Program evaluation
- Management review
This post will clarify things by looking at each component individually and then see how they tie together. In the end, you’ll see that meeting these requirements of the ASTM E2659 Standard is not as complicated as it appears initially.
All three of these practices are conducted by the certificate issuer. The internal audit looks at policies and procedures. The program evaluation adds the perspective of external impact, and the management review looks at the result of those two items, plus some additional data, to decide if changes are necessary.
Internal Audit
The internal audit is defined in the terminology section of the Standard as a first-party review of the certificate issuer’s policies and procedures to ensure adherence to this practice’s requirements.
ASTM E2659 itself doesn’t provide a lot of detail regarding the internal audit. It mandates two things:
- Internal audits have to be planned and conducted on a regular basis
- The results have to be documented and communicated to program management
The best clue is the word Internal. This requirement needs to be viewed in context as part of Section 5.2, which focuses on the certificate user’s management system. Since 5.2 spells out what the management system must include, the internal audit has to specifically address all its components. These are:
- Advisory group composition and responsibilities
- Certificate program instructional design plan
- Certificate issuance and use
- Invalidating a certificate
- Complaints
- Appeals
- Privacy, confidentiality, and security
- Program commercial support and disclosure
- Fees, cancellation, and refunds
- Nondiscrimination
- Personnel
- Outsourcing
- Records control
- Document control
- Internal audit
- Management review and
- Corrective and preventive action
However, it won’t be sufficient to have the internal audit focus only on whether a policy exists. Some of these policies need procedures to ensure certificate program staff enact the policies consistently. The topic of complaints is a good example of this. Complaints appears in two areas in the standard: first in 5.2.1.3(5) as a policy; and later in 5.8 as a process. Throughout the Standard, processes are also required because often just a policy alone may not be sufficient. This is true for complaints, appeals, records control, document control, corrective and preventive actions, management review, and even the internal audit.
Remember that the internal audit is defined by ASTM E2659 as a first-party review of the certificate issuer’s policies and procedures to ensure adherence to this practice’s requirements. Not meaning to be tricky, the standard has different ways of referring to procedures, using alternate terms such as process or mechanism. When these other terms are encountered in the standard, these should be understood as procedures that also need to be incorporated into the internal audit.
Ensuring adherence to policies and procedures is the purpose of the internal audit. This is the certificate issuer looking internally. “Are we following the procedures—based on our policies—that we said we would follow? The evidence of that adherence is contained in the internal audit report. The internal audit report contains the methodology and results of the audit, and this report needs to be shared with program management. It is likely that corrective and preventive actions will come directly from this report.
The certificate issuer is not required to have a process titled Internal Audit, nor must it create a document titled Internal Audit Report. As long as the certificate issuer reviews and reports on all 17 elements on a regular basis, the Standard may be met.
Certificate Program Evaluation
This element of the Standard focuses on the impact of the program on its stakeholders, which always includes learners, but may also include employers, the public, or anyone else who may be impacted by the program. So, in contrast to the internal audit, the certificate program evaluation looks outside to focus on the program’s results and impact.
Section 6.1.10 of ASTM E2659 requires that certificate program evaluation:
- Is conducted on a regular basis
- Measures program performance against stated program objectives
- Measures the quality and effectiveness of the ways the mastery of learning outcomes is assessed
- Includes learner feedback
- Includes mechanisms to identify the need for changes to the certificate program
- Is documented
Since the overall objective of any certificate program is learning, the focus of the program evaluation is making sure that participants are actually mastering the learning outcomes! So, a major focus of the program evaluation is on the assessment of learners. If the learning outcomes are acquisition of knowledge, and the assessment is a type of test based on learner responses, the evaluation determines if the test itself measures the knowledge at an appropriate level (as determined by a panel of experts.) This is known as criterion-referenced testing. It also looks at the value of each question in measuring that knowledge.
If the learning outcome is performance or behavior, the evaluation measures whether the judges or assessors of that performance are making that judgement or assessment properly. In other words, the assessment component of the program evaluation is designed to ensure that the learner’s mastery of the outcomes is measured in a valid and reliable way.
It also helps to ensure that the program is meeting the needs of the learners from the learners’ perspective. Is the program delivering what real users want and need?
Finally, are the criteria for changes to the program defined in an objective manner? What triggers a change to the content or the assessment? How does program management decide when to make changes and what they should be? This must be documented.
Management Review
The management review is the most comprehensive look that the certificate issuer takes. It takes place after the internal audit and the program evaluation and uses the internal audit and the program evaluation (among other things) as input. Its purpose is to determine if the issuer’s management system is assuring
- Compliance with ASTM E2659
- Alignment with the needs of the learners
Other inputs that are considered during the management review are:
- Results of external audits, if available
- Status of corrective or preventive actions
- Results of contractor performance monitoring
- Complaints received
- Appeals received
- Follow-up actions from previous management reviews
The results of the management review must include documentation about decisions and, if applicable, actions related to the:
- Improvement of the management system
- Improvement of the certificate program activities
- Resource needs
Internal Audit, Program Evaluation, Management Review: Putting it Together
Above, these three pillars were presented in the order they occur within the program year. First, the internal audit is conducted to evaluate how well the certificate issuer is managing the program by adhering to its own policies and procedures. Next, a program evaluation is conducted to evaluate the impact that the program has on meeting the objectives of the program. Note that the order of the internal audit and program evaluation could be switched or conducted at the same time. Although the order is inconsequential, it is helpful to have an annual calendar that ensures these happen every year and ideally in the same month as the year before. Finally, the management review is conducted. The ASTM E2659 Standard is very specific about the inputs and outputs of the review. Having the program evaluation and internal audit already done before going into the management review can help reduce stress associated with trying to get things wrapped up prior to the next year’s surveillance.
Knowledge on CAP Foundational Practices Limits Application Confusion
With this information, certificate issuers and applicants will not be stymied as to the purpose, difference, and sequencing of these three foundational practices. Having knowledge of each of these and how they differ will help to reduce the number of non-conformities that appear in these sections on the Certificate Accreditation Program (CAP) application.
Contributing Author: Dan Hiltz, PhD
Dan Hiltz is a global speaker, coach, and consultant who focuses on individual and team performance. He’s been recognized by Training magazine, eLearning, and Chief Learning Officer. His crusade for supercharging leaders has taken him to 32 states in the US and 13 countries on 4 continents.
Dan is a graduate of Thomas More University in Crestview Hills, KY, and holds a masters and doctorate from the University of Notre Dame. He is also an assessor for the ANSI National Accreditation Board.
His personal accreditations include:
- Gallup Certified Strengths Coach
- CompTIA Certified Technical Trainer (CTT+)
- Certified Healthcare Financial Professional (CHFP)
Dan can be reached at dan@leadershipsquared.com
Contributing Author: Kathy Tuzinski
Kathy Tuzinski is a Contract Assessor for ANAB’s Certificate Accreditation Program. She has a science-practioner’s background in the testing industry as a psychometrician, consultant, and industrial/organizational psychologist. She started Human Measures to provide the testing industry support in developing testing programs that are based on science – advising companies on job analysis and competency modeling, test development and test validation, legal requirements for employee selection, and relevant testing standards, including the AERA, APA, and NCME Standards for Educational and Psychological Testing and the SIOP Principles for the Validation and Use of Personnel Selection Procedures. She is a member of the American Psychological Association and the Society for Industrial/Organizational Psychology, has published several articles in peer-reviewed journals, and is co-editor of the book Simulations for Personnel Selection. She holds a Master of Arts degree in Industrial/Organizational Psychology from the University of Minnesota. You can reach her at https://www.human-measures.com/.
