Decision Tools

Tools for making decisions based on calibrated scores, including threshold optimization and utility analysis.

Threshold Optimization

optimal_threshold

Find the threshold that maximizes utility.

Utility = benefit * true_positives - cost * false_positives

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores, shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
benefit	float	Benefit of a true positive	1.0
cost	float	Cost of a false positive	1.0
n_thresholds	int	Number of thresholds to evaluate	100

Returns:

Type	Description
Tuple[Tensor, Tensor]	Tuple of (optimal_threshold, maximum_utility)

threshold_for_coverage

Find the threshold that achieves a target coverage.

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores	required
target_coverage	float	Desired fraction of items to include (0, 1]	required

Returns:

Type	Description
Tensor	Threshold value

Budget-Constrained Selection

budget_constrained_selection

Select top-k items given a budget constraint.

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores, shape (n_samples,)	required
budget	int	Maximum number of items to select	required

Returns:

Type	Description
Tensor	Boolean mask indicating selected items, shape (n_samples,)

expected_utility_at_budget

Compute expected utility given a budget constraint.

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores	required
labels	Tensor	Binary relevance labels	required
budget	int	Maximum number of items to select	required
benefit	float	Benefit per true positive	1.0
cost	float	Cost per false positive	1.0

Returns:

Type	Description
Tensor	Utility value

Curves

utility_curve

Compute utility curve across thresholds.

Utility = benefit * true_positives - cost * false_positives

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores, shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
benefit	float	Benefit of a true positive	1.0
cost	float	Cost of a false positive	1.0
n_thresholds	int	Number of threshold points	100

Returns:

Type	Description
Tuple[Tensor, Tensor]	Tuple of (thresholds, utility) tensors

utility_budget_curve

Compute utility as a function of budget.

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores	required
labels	Tensor	Binary relevance labels	required
max_budget	Optional[int]	Maximum budget to evaluate (default: all samples)	None
benefit	float	Benefit per true positive	1.0
cost	float	Cost per false positive	1.0

Returns:

Type	Description
Tuple[Tensor, Tensor]	Tuple of (budgets, utilities) tensors

risk_coverage_curve

Compute risk-coverage curve.

Shows the trade-off between coverage (fraction of items above threshold) and risk (error rate on items above threshold).

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores, shape (n_samples,)	required
labels	Tensor	Binary relevance labels, shape (n_samples,)	required
n_thresholds	int	Number of threshold points	100

Returns:

Type	Description
Tuple[Tensor, Tensor]	Tuple of (coverage, risk) tensors, each shape (n_thresholds,)

Plotting

plot_utility_curve

Plot utility curve.

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores	required
labels	Tensor	Binary relevance labels	required
benefit	float	Benefit of a true positive	1.0
cost	float	Cost of a false positive	1.0
n_thresholds	int	Number of threshold points	100
title	str	Plot title	'Utility Curve'
figsize	Tuple[float, float]	Figure size	(8, 6)

Returns:

Type	Description
Figure	Matplotlib Figure object

plot_risk_coverage

Plot risk-coverage curve.

Parameters:

Name	Type	Description	Default
scores	Tensor	Calibrated scores	required
labels	Tensor	Binary relevance labels	required
n_thresholds	int	Number of threshold points	100
title	str	Plot title	'Risk-Coverage Curve'
figsize	Tuple[float, float]	Figure size	(8, 6)

Returns:

Type	Description
Figure	Matplotlib Figure object