Course / III. Development / 3.6
Governing generative AI
12 min read
Generative AI and foundation models raise every risk above, plus some of their own.
Risks
| Risk | Description |
|---|---|
| Hallucination / confabulation | Fluent, confident, false output |
| Prompt injection and jailbreaks | Hidden instructions or crafted prompts that override safeguards |
| Data leakage | Confidential prompts retained or used for training; memorized training data reproduced |
| IP | Copyrighted training data; outputs resembling protected works; unclear output ownership |
| Harmful content | Toxic, dangerous or illegal output |
| Synthetic media | Deepfakes, impersonation, fraud |
| Over-reliance | Users trust output without checking |
| Agentic actions | Tools and autonomy turn wrong outputs into wrong actions |
Controls
- Grounding: retrieval-augmented generation (RAG) on approved sources, with citations
- Guardrails: input and output filters, system prompts, restricted tools and permissions
- Red teaming before release and periodically after
- Human review for high-stakes outputs; least-privilege and approval steps for agents
- Data controls: enterprise agreements with no training on your data; retention limits; DLP
- Provenance: watermarking and content credentials (C2PA); labeling to meet EU Art. 50
- Use policy and training for staff
- Evaluation: benchmarks for accuracy, safety and bias tailored to your use case
Fine-tuning vs prompting vs RAG
Fine-tuning changes the model's weights with your data (stronger customization, harder to remove data). Prompting and RAG keep the model fixed and supply context at run time (easier to update and govern). Choosing the lighter approach is often the lower-risk option.
Check your understanding
Check 1 of 2Domain III. Development
A customer-service bot invents a refund policy. Which control MOST directly reduces this?
Check 2 of 2Domain III. Development
A web page contains hidden text telling an AI assistant to email the user's files to an attacker. This is:
Answer the checks, then mark this lesson complete
Completion and scores are saved in this browser.