01 / GROUP OUTCOMES
Confusion & Rate Table
One threshold: 0.50 for both groups
| Group | n | Base | TN | FP | FN | TP | Select | TPR | FPR | TNR | PPV | NPV | Accuracy |
|---|
Fairness audit bench / UCI Adult / Logistic regression
One Classifier. Incompatible Definitions of Fairness.
Audit a real held-out score set against demographic parity, equal opportunity, equalized odds, predictive parity, and calibration. Then move along the measured accuracy-fairness frontier using group-specific decision thresholds.
- Held-out rows
- Loading…
- ROC AUC
- Loading…
- Features
- Loading…
- Sensitive inputs
- 0
Live Audit
Decision Surface
Loading held-out model scores…
- Group A Base Rate
- 0.000
- Group B Base Rate
- 0.000
- Base-Rate Gap
- 0.000
- Overall Accuracy
- 0.000
- Accuracy Cost
- 0.000
02 / CRITERIA
Fairness Criteria
Pass if |gap| ≤ 0.05; impact ratio ≥ 0.80
| Criterion | Definition | Value | Check |
|---|
03 / CONSTRAINT
Impossibility, Measured
Unequal base rates activate the incompatibility result for a non-perfect calibrated score.
- Base-rate gap
- 0.000
- FPR gap
- 0.000
- FNR gap
- 0.000
- Worst calibration residual
- 0.000
| Rule | Targets | Gives Up |
|---|
This is an empirical illustration, not a proof. The theorem’s exceptions include equal base rates and perfect prediction.
04 / DISCRIMINATION
ROC & Achievable Region
Threshold-free score behavior
A group threshold selects one point on each curve. Randomized threshold mixtures can reach convex combinations between points.
ROC Data Table
| Group | Threshold | FPR | TPR |
|---|
05 / CALIBRATION
Reliability Within Groups
10 equal-width score bins
Reliability Data Table
| Group | Bin | n | Mean Score | Observed | Residual |
|---|
06 / POST-PROCESSING
Accuracy vs Equalized-Odds Frontier
Computing threshold sweep…
Drag the plotted point, focus it and press Left or Right, or use this range control.
Selected Threshold Pair
- Group A
- 0.50
- Group B
- 0.50
- Accuracy
- 0.000
- EO Gap
- 0.000
- Accuracy Cost
- 0.000
Pareto Frontier Data Table
| Point | Group A Threshold | Group B Threshold | Accuracy | EO Gap | Dominated |
|---|
Method
Definitions, Not Proxies
All rates come from the held-out arrays in the browser. A positive prediction means predicted income above $50,000 at the active threshold. Gap signs are Group A minus Group B.
- Selection rate
- (TP + FP) / N.
- TPR / Equal opportunity
- TPR = TP / (TP + FN). Equal-opportunity difference = TPRA - TPRB.
- FPR / Equalized odds
- FPR = FP / (FP + TN). Equalized-odds difference = max(|TPR gap|, |FPR gap|).
- Demographic parity
- Selection-rate difference = selectionA - selectionB. Disparate impact = min(selection) / max(selection).
- Predictive parity
- PPV = TP / (TP + FP). Predictive-parity difference = PPVA - PPVB.
- Calibration within groups
- For each group and score bin: observed positive rate should equal mean predicted probability. The chart reports expected calibration error and the worst signed bin residual.
- Other rates
- TNR = TN / (TN + FP). NPV = TN / (TN + FN). Accuracy = (TP + TN) / N.
- Pareto frontier
- Every 0.01 threshold pair is evaluated. A point remains only when no other pair has at least its accuracy and at most its equalized-odds gap, with one strict improvement.
Data
Adult Income, With Its History Intact
The model trains on the UCI Adult training file and this page loads its official held-out test split after removing rows with missing fields. The target is whether annual income exceeded $50,000 in 1994 Census data. Race and sex are excluded from the feature matrix but retained for this audit. Their omission does not remove proxy relationships in education, occupation, hours, or other features.
Recorded categories reflect the source dataset, not a current or complete account of identity. “Non-White” is an audit aggregation chosen to keep this 2-group bench statistically legible; it collapses materially different groups and must not be used to make deployment claims.
Limitations
A Passing Metric Is Not a Fairness Certificate
- Group fairness is not individual fairness. Equal aggregate rates do not mean similar people receive similar decisions.
- The income label may encode historical exclusion and structural inequality. Accurately predicting it can reproduce those patterns.
- One-attribute analysis misses intersectionality. Sex and race are audited separately here, so subgroup harms can remain hidden.
- Small groups and sparse calibration bins have higher sampling variance. The 80% rule is a screening rule of thumb, not a legal conclusion.
- Deterministic threshold sweeps approximate post-processing but do not implement the randomized mixtures in the full Hardt, Price, and Srebro construction.
- Threshold post-processing uses sensitive group membership at decision time. Whether that is lawful or appropriate depends on context and jurisdiction.