Course / III. Development / 3.4
Testing, explainability and security
13 min read
Testing proves the system meets the requirements set at design.
What to test
| Area | Examples |
|---|---|
| Unit and integration | Each component works on its own (unit), and works with the data pipelines and systems around it (integration) |
| Performance | Accuracy, precision, recall, error rates on held-out data that matches real use |
| Fairness | Metrics across groups chosen at design; intersectional groups |
| Robustness | Noisy, unusual or out-of-distribution inputs; stress tests |
| Security | Adversarial examples, poisoning resistance, model extraction, inversion and membership inference, prompt injection |
| Explainability | Can the right people understand outputs well enough for their role? |
| Threat modeling | Before security testing, list who might attack or misuse the system and how, using a catalog such as MITRE ATLAS or the OWASP Top 10 for LLM Applications; then test those paths |
| Safety and misuse | Red teaming: structured adversarial testing for failure modes and harms |
| Overfitting | Big gap between training and test performance |
Verification asks "did we build it right?" (meets specs). Validation asks "did we build the right thing?" (fit for its real-world purpose).
Explainability
- Global: how the model behaves overall (feature importance across the dataset).
- Local: why this particular output (the main reasons this applicant was declined).
- Techniques: SHAP and LIME (feature attributions), counterfactual explanations ("approved if income were $5,000 higher"), surrogate models.
Explanations must fit the audience: regulators, operators and affected individuals need different things.
Independence and acceptance
High-risk systems benefit from testing by people independent of the builders (second line or external auditors). Compare results to the acceptance criteria set at design, and document any accepted residual risk with a named approver.
Check your understanding
Check 1 of 2Domain III. Development
An applicant asks why their loan was declined. What type of explanation do they need?
Check 2 of 2Domain III. Development
A model scores 99% on training data and 70% on test data. What does this MOST likely indicate?
Answer the checks, then mark this lesson complete
Completion and scores are saved in this browser.