Data governance and bias
Models learn whatever the data teaches, including its gaps and prejudices.
Data governance checklist
- Provenance and lineage: where did the data come from, and how was it transformed?
- Rights: lawful basis, consent scope, licences, copyright, contractual limits, web-scraping terms.
- Quality: accuracy, completeness, timeliness, consistency.
- Representativeness: does it reflect the population the system will be used on?
- Labeling: clear guidelines, trained labelers, quality checks, and labeler working conditions.
- Minimization and protection: remove unneeded personal data, de-identify where possible, secure storage.
- Documentation: a datasheet for the dataset recording all the above.
Sources of bias
| Bias | What happens | Example |
|---|---|---|
| Historical | Data reflects past discrimination | Past hiring favored men, so the model does too |
| Representation / sampling | Some groups are under-represented | Few older applicants in training data |
| Measurement | Proxy or label measures the wrong thing | Using arrests as a proxy for crime |
| Aggregation | One model for groups that differ | One medical model across populations with different baselines |
| Evaluation | Test set doesn't represent real users | Benchmark mostly from one region |
| Deployment | Used outside its intended context | A tool built for one country used in another |
Proxies: removing a protected attribute does not remove bias if other features (ZIP code, school name, gaps in employment) stand in for it.
Fairness metrics conflict
- Demographic (statistical) parity: equal positive-outcome rates across groups.
- Equal opportunity / equalized odds: equal true-positive (and false-positive) rates.
- Predictive parity: equal precision across groups.
When base rates differ between groups, these generally cannot all be satisfied at once. Governance chooses the metric that fits the use case and the law, records why, and tests against it.
Mitigation options
Pre-processing (rebalance, reweight, collect more data), in-processing (fairness constraints during training) and post-processing (adjust thresholds), plus human review and narrowing the use case.
Check your understanding
A model excludes race but uses ZIP code, and outcomes still differ sharply by race. What is the likely issue?
A model trained on one hospital's patients performs poorly at another with a different population. This is BEST described as: