Calibrators

All calibrators inherit from BaseCalibrator and provide a consistent API for fitting and applying calibration transformations.

BaseCalibrator

Bases: Module, ABC

Abstract base class for all calibrators.

All calibrators are nn.Module subclasses for PyTorch compatibility. They implement a monotonic transformation of scores.

fit(scores, labels, **kwargs) abstractmethod

Fit the calibrator to validation data.

Parameters:

Name	Type	Description	Default
scores	Tensor	Predicted scores, shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
**kwargs	Any	Additional calibrator-specific parameters	{}

Returns:

Type	Description
'BaseCalibrator'	self

forward(scores) abstractmethod

Apply calibration to scores.

Parameters:

Name	Type	Description	Default
scores	Tensor	Uncalibrated scores, shape (n_samples,) or (batch, n_samples)	required

Returns:

Type	Description
Tensor	Calibrated scores, same shape as input

TemperatureScaling

Single-parameter calibration that scales logits by a learned temperature.

Bases: BaseCalibrator

Temperature scaling calibrator.

Applies a learned temperature parameter to scale logits

calibrated = sigmoid(logit(scores) / temperature)

This is the simplest parametric calibrator and serves as a strong baseline. It's differentiable and can be trained end-to-end.

__init__(init_temperature=1.0)

Initialize temperature scaling.

Parameters:

Name	Type	Description	Default
init_temperature	float	Initial temperature value. Default 1.0 (no scaling).	1.0

fit(scores, labels, lr=0.01, max_iter=100, tol=1e-06, **kwargs)

Fit temperature parameter using NLL loss.

Parameters:

Name	Type	Description	Default
scores	Tensor	Predicted scores in (0, 1), shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
lr	float	Learning rate for optimization	0.01
max_iter	int	Maximum optimization iterations	100
tol	float	Convergence tolerance	1e-06
**kwargs	Any	Unused, for API compatibility	{}

Returns:

Type	Description
TemperatureScaling	self

forward(scores)

Apply temperature scaling to scores.

Parameters:

Name	Type	Description	Default
scores	Tensor	Uncalibrated scores in (0, 1)	required

Returns:

Type	Description
Tensor	Calibrated scores in (0, 1)

IsotonicCalibrator

Non-parametric calibration using isotonic regression.

Bases: BaseCalibrator

Isotonic regression calibrator.

Fits a piecewise constant monotonic function using pool adjacent violators algorithm (PAVA). Non-parametric and guaranteed monotonic.

Note: Not differentiable, but fast and reliable baseline.

fit(scores, labels, **kwargs)

Fit isotonic regression using PAVA algorithm.

Parameters:

Name	Type	Description	Default
scores	Tensor	Predicted scores, shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
**kwargs	Any	Unused, for API compatibility	{}

Returns:

Type	Description
IsotonicCalibrator	self

forward(scores)

Apply isotonic calibration via interpolation.

Parameters:

Name	Type	Description	Default
scores	Tensor	Uncalibrated scores	required

Returns:

Type	Description
Tensor	Calibrated scores

PiecewiseLinearCalibrator

Differentiable piecewise linear calibration with monotonicity constraints.

Bases: BaseCalibrator

Monotonic piecewise linear calibrator.

Uses linear interpolation between learnable knot values with monotonicity constraints enforced via softplus on increments. This provides a flexible, differentiable calibration function that balances expressiveness with stability.

The calibrator learns values at fixed knot positions and interpolates linearly between them. Monotonicity is guaranteed by construction.

__init__(n_knots=10)

Initialize piecewise linear calibrator.

Parameters:

Name	Type	Description	Default
n_knots	int	Number of interior knots. Total knots will be n_knots + 2 (including endpoints at 0 and 1).	10

fit(scores, labels, lr=0.1, max_iter=500, tol=1e-06, **kwargs)

Fit calibrator parameters using NLL loss.

Parameters:

Name	Type	Description	Default
scores	Tensor	Predicted scores in (0, 1), shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
lr	float	Learning rate for optimization	0.1
max_iter	int	Maximum optimization iterations	500
tol	float	Convergence tolerance	1e-06
**kwargs	Any	Unused, for API compatibility	{}

Returns:

Type	Description
PiecewiseLinearCalibrator	self

forward(scores)

Apply piecewise linear calibration.

Parameters:

Name	Type	Description	Default
scores	Tensor	Uncalibrated scores in (0, 1)	required

Returns:

Type	Description
Tensor	Calibrated scores in (0, 1)

MonotonicNNCalibrator

Neural network calibrator with monotonicity constraints for maximum flexibility.

Bases: BaseCalibrator

Monotonic neural network calibrator.

Uses a neural network with constrained architecture to ensure monotonicity: - All weights are non-negative (via softplus) - All activations are monotonic (ELU)

This is the most flexible calibrator, suitable for complex calibration patterns. Fully differentiable and trainable end-to-end.

__init__(hidden_dims=(16, 16))

Initialize monotonic neural network.

Parameters:

Name	Type	Description	Default
hidden_dims	Tuple[int, ...]	Tuple of hidden layer dimensions.	(16, 16)

fit(scores, labels, lr=0.01, max_iter=1000, tol=1e-06, batch_size=None, **kwargs)

Fit neural network using NLL loss.

Parameters:

Name	Type	Description	Default
scores	Tensor	Predicted scores in (0, 1), shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
lr	float	Learning rate for optimization	0.01
max_iter	int	Maximum optimization iterations	1000
tol	float	Convergence tolerance	1e-06
batch_size	Optional[int]	Batch size for training. None for full batch.	None
**kwargs	Any	Unused, for API compatibility	{}

Returns:

Type	Description
MonotonicNNCalibrator	self

forward(scores)

Apply neural network calibration.

Parameters:

Name	Type	Description	Default
scores	Tensor	Uncalibrated scores in (0, 1)	required

Returns:

Type	Description
Tensor	Calibrated scores in (0, 1)