These fairness metrics are used for classification models to ensure equitable outcomes across different groups. Future metrics will be added based on DAO Governance voting to adapt and expand our fairness assessments.
A: Protected attribute (e.g., race, gender) where \( A = 0 \) represents the unprivileged group and \( A = 1 \) represents the privileged group.
Y: Actual outcome (ground truth), where \( Y = 1 \) indicates a positive outcome and \( Y = 0 \) indicates a negative outcome.
\(\hat{Y}\): Predicted outcome by the classification model, where \( \hat{Y} = 1 \) indicates a positive prediction and \( \hat{Y} = 0 \) indicates a negative prediction.
The statistical parity difference measures the difference in the positive outcome rates between the unprivileged group and the privileged group.
\[ \text{SPD} = P(\hat{Y} = 1 \mid A = 0) - P(\hat{Y} = 1 \mid A = 1) \]
Example:
- Unfair Outcome: SPD significantly different from 0 (e.g., -0.4 or 0.4)
- Fair Outcome: SPD close to 0 (e.g., -0.1 to 0.1)
Disparate impact compares the ratio of the positive outcome rates between the unprivileged group and the privileged group.
\[ \text{DI} = \frac{P(\hat{Y} = 1 \mid A = 0)}{P(\hat{Y} = 1 \mid A = 1)} \]
Example:
- Unfair Outcome: DI significantly different from 1 (e.g., less than 0.8 or greater than 1.25)
- Fair Outcome: DI close to 1 (e.g., 0.8 to 1.25)
The average odds difference measures the difference in false positive rates and true positive rates between the unprivileged group and the privileged group.
\[ \text{AOD} = \frac{1}{2} \left[ \left( P(\hat{Y} = 1 \mid A = 0, Y = 0) - P(\hat{Y} = 1 \mid A = 1, Y = 0) \right) + \left( P(\hat{Y} = 1 \mid A = 0, Y = 1) - P(\hat{Y} = 1 \mid A = 1, Y = 1) \right) \right] \]
Example:
- Unfair Outcome: AOD significantly different from 0 (e.g., -0.2 or 0.2)
- Fair Outcome: AOD close to 0 (e.g., -0.1 to 0.1)
The equal opportunity difference measures the difference in true positive rates between the unprivileged group and the privileged group.
\[ \text{EOD} = P(\hat{Y} = 1 \mid A = 0, Y = 1) - P(\hat{Y} = 1 \mid A = 1, Y = 1) \]
Example:
- Unfair Outcome: EOD significantly different from 0 (e.g., -0.2 or 0.2)
- Fair Outcome: EOD close to 0 (e.g., -0.1 to 0.1)