In the hope of fostering improvement, this is the second of two posts reviewing some of the “bad auditor behaviors” observed by ANAB in the course of performing thousands of witnessed audits.
In our first post on bad auditor behavior, we explained the importance of the witnessed audit, provided a description of the process from IAF MD 17, and described some of the bad auditor behavior ANAB has observed while performing thousands of witnessed audits over the years. We also identified some of the common ways we’ve seen auditors waste time.
Some additional examples of time-wasting include the following:
- Interviewing techniques: This is a basic auditing issue, but we see some auditors asking leading questions. Yes, sometimes it’s difficult to get the client’s employees to open up and answer questions or get over their fear of the audit. But we object to leading the client to expected answers while gathering data. For instance, “You do know your quality policy that is posted on the wall over there, correct?” Or my all-time favorite, “I have the procedure here that describes the process for your job; do you do everything in here?” Auditors should just get the facts and not help with the answers.
- Opening and closing meetings: Again, while auditors need to be personable and be able to deal with clients and the various temperaments of those they are auditing, from the CEO to the janitor, we still expect to see a degree of formality in the opening and closing meetings. ISO/IEC 17021-1 defines the requirements clearly and we have written NCRs for lack of coverage of the requirements, but we’ve seen abbreviated opening meetings and, at worst and more common, very informal discussions that are supposed to be the closing meetings at small and mid-sized organizations where the auditor has familiarity.
- Not following audit trails: Really good auditors often connect the dots of an organization’s process and QMS by following a single activity from start to finish and all the support processes to determine the effectiveness of the system during an audit. But some auditors just go through the motions. We see evidence collected on one process where the process effectiveness remians unknown because just one piece of the process is examined. For example, the auditor looked at five purchase orders and all appeared to have been issued correctly, but did the auditor check that at least one or two of those orders were received and that they complied with the requirements of the purchase order? Sometimes the auditor goes to the receiving area and looks at any available purchase orders, not the ones checked earlier. Another example is checking on monitoring and measuring equipment and failing to determine if there are specific requirements for the monitoring and measuring (calibration) and if those requirements were followed, not just that it was tracked and calibrated. Sometimes we observe an auditor collecting objective evidence on a process and then saying, “Oh look at the time! Per the plan, we need to move on to another process.” This is not doing the client any favors or validating the certificate.
- Lack of sampling: While related to the above item, we understand that when sampling an organization’s processes and gathering objective evidence there may not be more than a sample of one item for a small organization. But that’s usually the exception rather than the rule. When an auditor falls behind or is in a rush during a witnessed audit – and often when they’re unfamiliar with an organization or its processes – they tend to skim the surface and do insufficient sampling. Why is this not an NCR? This is a very subjective area and there are few rules on how much is sufficient. The amount of sampling may be based more on the time allotted than the volume of activity the process executes. Sampling three purchase orders issued in an organization that issues 10 per week on average may seem sufficient, but how about an organization that issues 30 per day? Is looking at only three still adequate? The laws of probability would say no. The same holds true for manufacturing processes: How many product lines, and how many operations? While we use employees as a gauge for time allocation, we need to look at process volume for audit time application, and we don’t always see that happening.
In summary, while ANAB is focused on the CB and the CB’s processes, in the course of conducting witnessed audits, we see thousands of auditors and all sorts of audit behavior, both good and bad. We know CBs get an earful from clients about any very bad behaviors of their auditors, but we see a lot that’s not reported, and there’s a lot of room for improvement.
Maybe we need to have CBs and auditors pay more attention to ISO 19011 for auditor traits. Maybe we need better feedback on auditor performance. Either way, both ABs and CBs have a joint mission to the certificated organizations to give their customers confidence that the goods and services they receive meet requirements. The processes we audit are supposed to be defined, controlled, and measured so that they can be continually improved, and unwanted variation can be eliminated. The same is true of our certification processes. But the biggest and most complex variable in the process is the auditor, and we should work diligently on improvement.