# AAIA interview scorecard

Candidate: ______  Interviewer: ______  Date: ______

| # | Question | Listen for | Score 1 to 4 |
|---|---|---|---|
| 1 | A vendor reports one accuracy figure for the model we are buying. What do you ask next? | Accuracy on what data and for which version; how rare the outcome is; recall and precision where the two errors cost different amounts; results by group; and whether the figure says anything about our own deployment. | |
| 2 | How would you test whether our AI inventory is complete? | Treat it as a claim. Reconcile it to sources the AI program does not control: purchase orders, expense claims, sign-in records, cloud bills and vendor release notes. A system found outside it is a finding, and so is the control that missed it. | |
| 3 | A supplier tells us it is ISO/IEC 42001 certified. How much do you rely on that? | Check accreditation, scope and dates, then read the statement of applicability for exclusions. The certificate covers a management system on a sample, not a model's accuracy or fairness, and it is not proof of EU AI Act compliance. | |
| 4 | How would you audit change management for a model that retrains every week? | Get the version history first. Test the controls on the pipeline: bounds on what a retrain may change, tests before promotion, monitoring after each release, rollback. For hosted models, version pinning and notice in the contract. | |
| 5 | Our claims team says a person reviews every model decision. How would you test that? | Watch the work. Override rates, review time per case, seeded cases with a known right answer, and whether a reviewer can disagree without penalty. A signed procedure shows only that oversight was designed. | |
| 6 | What AI-specific threats would you expect in the risk register for a customer assistant built on a language model? | Prompt injection, data leakage, poisoning, insecure output handling and excessive agency, each tied to a control that touches the model, its data or what it can do. A single line saying cyber attack is a finding about the risk assessment. | |
| 7 | Halfway through fieldwork you find a model wrongly refusing applicants, and it is still happening. What do you do? | Tell management and the incident owner at once. Do not switch it off yourself: that is a management decision. Preserve the evidence, and report it later as a finding on detection, containment and root cause. | |
| 8 | The model you sampled was retrained halfway through the audit period. What does that do to your conclusion? | Each version is a stratum. Sample from each or limit the conclusion to the versions tested and say so. Test the change control over the transition, and roll forward from an interim conclusion. | |
| 9 | Write the condition for a fairness finding on a claims model. | Stated as fact a reader can check: model version, period, the measured gap, the approved tolerance and the population tested. Criterion, cause, effect and recommendation follow. | |
| 10 | How would you use an AI tool in your own audit work, and what would you never let it do? | An approved tool whose terms rule out retention and training on inputs. Every factual output traced to its source before it enters a workpaper. Tool, version, prompt and output recorded. A sample of its work reviewed. The auditor signs the report, not the tool. | |

Source: https://credentialpress.com/guides/aaia-interview-questions
