Calibration Explorer · Assessment record

Handwritten digits · real classifier

Policy locked before test inspection in this session

1797 observations in test. Generated 2026-10-04T08:41:18.279Z.

Findings

Accuracy stayed unchanged (94.9% to 94.9%); NLL decreased from 0.1760 to 0.1575 on these observations.
ECE decreased from 0.0365 to 0.0101 with the selected binning; this is not a deployment verdict.
Temperature was fitted on calibration observations only. Changes on other splits must be measured separately.

Measurement	Original	After temperature scaling
Accuracy	94.9%	94.9%
ECE	0.0365	0.0101
NLL (nats / observation)	0.1760	0.1575

Confidence and correctness

15 equal-width bins. ECE is on a 0–1 scale; lower is not proof of calibration.

● Original◆ After scaling

Bin / series	Count	Confidence	Accuracy	ECE contribution
Original 1	0	No observations	Unavailable	0.0000
Original 2	0	No observations	Unavailable	0.0000
Original 3	0	No observations	Unavailable	0.0000
Original 4	0	No observations	Unavailable	0.0000
Original 5	1	32.2%	100.0%	0.0004
Original 6	11	36.6%	45.5%	0.0005
Original 7	27	43.8%	63.0%	0.0029
Original 8	30	49.9%	53.3%	0.0006
Original 9	44	56.2%	61.4%	0.0013
Original 10	34	63.1%	67.6%	0.0009
Original 11	55	70.3%	78.2%	0.0024
Original 12	60	76.4%	86.7%	0.0034
Original 13	110	83.4%	92.7%	0.0057
Original 14	172	90.4%	97.7%	0.0070
Original 15	1253	98.2%	99.8%	0.0116
After 1	0	No observations	Unavailable	0.0000
After 2	0	No observations	Unavailable	0.0000
After 3	0	No observations	Unavailable	0.0000
After 4	0	No observations	Unavailable	0.0000
After 5	0	No observations	Unavailable	0.0000
After 6	4	38.8%	50.0%	0.0002
After 7	9	43.0%	55.6%	0.0006
After 8	23	50.3%	56.5%	0.0008
After 9	30	56.6%	60.0%	0.0006
After 10	25	63.6%	56.0%	0.0011
After 11	42	70.2%	61.9%	0.0019
After 12	29	77.4%	79.3%	0.0003
After 13	63	83.4%	79.4%	0.0014
After 14	113	90.7%	90.3%	0.0003
After 15	1459	99.2%	99.5%	0.0028

Acceptance at 80.0%

1635 accepted · 162 abstained · 31 accepted mistakes · 1.9% observed error among accepted predictions.

Configuration and provenance

{
  "data": {
    "name": "Handwritten digits · real classifier",
    "kind": "logits",
    "sha256": "18f281567b4a749dabe3819a427d32d228b85a7226c7785c10a99d88cf0d46d7",
    "rowCount": 2945,
    "splitCounts": {
      "calibration": 574,
      "policy_validation": 574,
      "test": 1797
    },
    "classes": [
      "0",
      "1",
      "2",
      "3",
      "4",
      "5",
      "6",
      "7",
      "8",
      "9"
    ],
    "provenance": {
      "source": "Bundled UCI handwritten-digits reference",
      "dataset": "UCI Optical Recognition of Handwritten Digits",
      "datasetDoi": "10.24432/C50P49",
      "datasetUrl": "https://archive.ics.uci.edu/dataset/80/optical+recognition+of+handwritten+digits",
      "datasetVersion": "Original optdigits.tra and optdigits.tes; identified by SHA-256 below",
      "license": "CC BY 4.0",
      "attribution": "Alpaydin, E. & Kaynak, C. (1998). Optical Recognition of Handwritten Digits. UCI Machine Learning Repository.",
      "model": {
        "id": "optdigits-logistic-regression-v1",
        "revision": "sha256:fc400aabfabd726df19d674b31863db1ee588cc60850796689a48472d6e5b8bd",
        "estimator": "sklearn.linear_model.LogisticRegression",
        "logitDefinition": "decision_function scores; softmax probabilities; columns follow model.classes_"
      },
      "preprocessing": {
        "dtype": "float64",
        "transform": "64 original integer features divided by 16",
        "learnedTransforms": false
      },
      "generation": {
        "script": "scripts/prepare_digits.py",
        "scriptSha256": "df730e9c9ad62882aef79f419452db27769c8884f951c41f4762443e1440ab07",
        "command": "python scripts/prepare_digits.py",
        "requirements": "scripts/requirements-reference.txt"
      },
      "splitMethod": {
        "name": "Two deterministic stratified train_test_split calls within optdigits.tra; official optdigits.tes preserved",
        "membershipSha256": "66306e9bc90ca705114ec4f152d4f7fa7e6f99016988911691e2a26d55f79308",
        "firstHoldoutSize": 1148,
        "secondHoldoutSize": 574
      }
    }
  },
  "configuration": {
    "split": "test",
    "bins": 15,
    "strategy": "equal-width",
    "temperature": 0.7259783904070843,
    "threshold": 0.8,
    "group": null
  },
  "policy": {
    "testViewed": true,
    "changedAfterTest": false,
    "locked": {
      "temperature": 0.7259783904070843,
      "threshold": 0.8
    }
  },
  "fit": {
    "temperature": 0.7259783904070843,
    "nllBefore": 0.15739905266200538,
    "nllAfter": 0.1413242121853789,
    "improved": true,
    "atBound": null,
    "status": "converged",
    "iterations": 26
  },
  "fitAttempt": {
    "status": "applied",
    "result": {
      "temperature": 0.7259783904070843,
      "nllBefore": 0.15739905266200538,
      "nllAfter": 0.1413242121853789,
      "improved": true,
      "atBound": null,
      "status": "converged",
      "iterations": 26
    },
    "calibrationRows": 574,
    "message": "Temperature was fitted on calibration observations only. Changes on other splits must be measured separately."
  },
  "learning": null,
  "evaluationFailure": null,
  "software": {
    "app": "calibration-explorer/0.2.0",
    "numericalLibrary": "@m-sanchez/calibrated/2.0.1",
    "upstreamCommit": "6e74b92dfc5c545a1cd926f96bd5bc44ae6c887b",
    "appSourceSha256": "9ce40339840b8e81d3c709af90b81d4e9c5108b53872625f0f932cfaebc0d05f",
    "numericalDistributionSha256": "3331be184fa4cb43dd85f42d2dc67abffb03d47b1bbf574c466e80758d057dd2"
  }
}

Limits of this assessment

This measures top-prediction confidence calibration, not all class probabilities or production safety.
Split separation is checked within this file and this session. Previous external use, entity overlap and statistical independence are not established.
ECE depends on binning and finite sample size. Better ECE does not guarantee a better acceptance policy or performance under distribution shift.
The acceptance rule is confidence >= threshold. Tied confidences are accepted together; an empty accepted set has no estimated error rate.
Group results and threshold exploration are descriptive. No correction for repeated comparisons is applied.
The input fingerprint identifies exact file contents. Reproduction without observations requires the matching input file and the recorded software.
Aggregate results are not anonymisation: small bins and groups can disclose individual outcomes. Review this report before sharing.
Default risk-coverage exports use a fixed 0.01 threshold grid rather than every individual confidence value.

Raw observations and row identifiers are omitted from this report. This HTML has no scripts, remote assets or telemetry.