Probability calibration, measured and repaired in the browser
Accuracy asks whether a classifier is right. Calibration asks whether a stated probability is trustworthy. This lab fits two post-hoc recalibrators on an independent calibration split, then measures their effect on untouched UCI Adult test predictions.
01 / Live analysis
All values below are computed locally from committed classifier outputs. No key, server, or library is used.
Loading held-out predictions…
Figure 1
Points compare mean predicted probability with the observed positive rate. The bars report the sample count behind each point. ECE and MCE change with the selected binning rule.
| Method | Bin | Mean probability | Observed rate | Count |
|---|
Table 1
Deltas are against raw predictions. Lower is better except for resolution and AUC. The Murphy rows group identical issued probabilities, independent of the display bins, so REL - RES + UNC reconstructs the Brier score exactly.
| Metric | Raw | Temperature | Delta | Isotonic | Delta |
|---|
Decision study
Predict positive when the probability exceeds CFP / (CFP + CFN). Expected cost uses the displayed probabilities. Observed cost uses the held-out labels.
| Method | Predicted positive | False positive | False negative | Expected cost / 100 | Observed cost / 100 | Observed delta |
|---|
02 / Method
Each bin compares its observed positive rate acc(Bm) with
its mean probability conf(Bm).
ECE = sum |B_m|/n * |acc(B_m) - conf(B_m)|.
MCE is the largest bin gap. Both are estimators whose value depends on bin edges,
bin count, and sample size, so this lab exposes both binning schemes and the counts.
BS = n^-1 sum (p_i - y_i)^2 = REL - RES + UNC.
Reliability penalizes disagreement between issued probabilities and event rates.
Resolution rewards forecasts that separate groups with different event rates.
Uncertainty is y_bar(1 - y_bar). Distinct issued probability values are
the groups used for the exact identity.
The browser applies sigmoid(logit(p) / T) and selects the single positive
scalar T that minimizes calibration-split negative log likelihood. A bounded
ternary search over T in [0.05, 20] replaces an optimizer dependency.
PAVA sorts calibration probabilities, pools adjacent blocks whenever their fitted event rates decrease, and returns a non-decreasing step function. It can repair any monotone distortion, but its flexibility makes it easier to overfit small calibration sets.
Log loss is the mean binary negative log likelihood. AUC is computed directly from average ranks, including ties. Temperature scaling is strictly monotone and preserves AUC. Isotonic steps can create ties, so its AUC can move slightly. The decision threshold follows the Bayes rule for user-supplied false-positive and false-negative costs.
03 / Data
Adult is a public binary-classification dataset derived from 1994 US Census records. The target is whether annual income exceeds $50K. The committed artifact contains only labels and four-decimal probabilities for the calibration and test splits. Raw demographic features are not shipped to the browser.
Four fixed-seed models provide deliberately different probability shapes: logistic regression, random forest, gradient boosting, and Gaussian naive Bayes. The source dataset is licensed CC BY 4.0. See the README for exact reproduction commands and citations.
04 / Limitations