References

This page lists the research papers that informed the design and implementation of Calibration Toolbox.

Key Papers

General Calibration Error Framework

Kull et al. (2019): “Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration.” NeurIPS 2019. arXiv:1910.12656
Nixon et al. (2020): “Measuring Calibration in Deep Learning.” CVPR Workshops 2020. arXiv:1904.01685

Foundational Metrics

Expected Calibration Error (ECE)

Naeini et al. (2015): “Obtaining Well Calibrated Probabilities Using Bayesian Binning.” AAAI 2015. Paper
Guo et al. (2017): “On Calibration of Modern Neural Networks.” ICML 2017. arXiv:1706.04599

Root Mean Square Calibration Error (RMSCE)

Hendrycks et al. (2019): “Deep Anomaly Detection with Outlier Exposure.” ICLR 2019. arXiv:1812.04606

Class-Conditional Metrics

Static and Adaptive Calibration Error (SCE, ACE)

Nixon et al. (2020): “Measuring Calibration in Deep Learning.” CVPR Workshops 2020. arXiv:1904.01685

Top-k Calibration Error

Gupta et al. (2021): “Calibration of Neural Networks using Splines.” ICLR 2021. arXiv:2006.12800

Overconfidence Error

Thulasidasan et al. (2019): “On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks.” NeurIPS 2019. arXiv:1905.11001

Additional Resources

Calibration Methods

Platt (1999): “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods.”
Zadrozny & Elkan (2002): “Transforming Classifier Scores into Accurate Multiclass Probability Estimates.” KDD 2002.
Kull et al. (2017): “Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers.” AISTATS 2017. arXiv

Review Papers

Roelofs et al. (2022): “Mitigating Bias in Calibration Error Estimation.” AISTATS 2022. arXiv:2012.08668
Minderer et al. (2021): “Revisiting the Calibration of Modern Neural Networks.” NeurIPS 2021. arXiv:2106.07998
Kumar et al. (2019): “Verified Uncertainty Calibration.” NeurIPS 2019. arXiv:1909.10155

Citation

If you use Calibration Toolbox in your research, please cite:

@software{calibration_toolbox,
  author = {Pearce, Jonathan},
  title = {Calibration Toolbox: A Python Library for Model Calibration Evaluation},
  year = {2026},
  url = {https://github.com/Jonathan-Pearce/calibration-toolbox}
}

Contributing

We welcome contributions! Please see the repository’s CONTRIBUTING.md file for guidelines.