Assessing the use case and human oversight
Most organizations deploy AI they did not build. Their duties start with the context of use.
Context decides risk
The same model can be low risk drafting marketing copy and high risk screening job applicants. Assess this use: purpose, affected people, decisions influenced, scale, reversibility, and legal triggers (EU AI Act tier, GDPR Art. 22, anti-discrimination law, state laws).
Model and deployment choices
The choice of model and how it's deployed changes the risk as much as the use case does.
| Choice | Options | What it changes |
|---|---|---|
| Model type | Classic (predicts or classifies) vs generative (produces content) | Generative output varies and can be wrong in new ways, so it needs output review and labeling |
| Source | Proprietary vs open source or open weights | Who you can hold accountable, what you can inspect, license terms |
| Size | Small vs large | Cost, speed, where it can run, how broad and unpredictable its abilities are |
| Modality | Language only vs multimodal (images, audio, video) | New inputs and outputs, such as biometric data and synthetic media |
| Where it runs | Cloud vs on-premise vs edge (on the device) | Where data goes, who controls security, how you update or switch it off |
| How it's adapted | As is, fine-tuned, retrieval-augmented generation (RAG), or agentic | Fine-tuning can make you responsible for the changed model; RAG adds data access risk; agents act on their own and need tighter limits |
Check the readiness of the workforce too: who will use it, whether they're trained, and how their work changes.
Assessments for deployers
- AI impact assessment (internal)
- DPIA where personal data is processed at high risk
- FRIA for listed EU deployers of high-risk AI
- Any assessment a state or sector law requires, such as a NYC LL 144 bias audit for a hiring tool
- Security and privacy review of the vendor and integration
Human oversight models
| Model | Meaning | Fits |
|---|---|---|
| Human-in-the-loop | A human approves each decision | High-stakes individual decisions |
| Human-on-the-loop | System acts; human monitors and can intervene | High-volume operations with escalation |
| Human-out-of-the-loop | No human involvement | Low-risk, reversible uses only |
Make oversight real
Automation bias is the tendency to over-trust the machine. Oversight on paper fails if reviewers approve everything in seconds. Make it meaningful:
- Train reviewers on the system's limits
- Give them explanations and the information needed to disagree
- Give them authority and time to override
- Monitor override rates; a near-zero rate is a warning sign
- Avoid workloads that force rubber-stamping
Check your understanding
Which oversight model fits AI that recommends whether to deny a disability benefit?
What does a near-zero human override rate MOST likely suggest?