finetuner.tuner.callback.evaluation module#

class finetuner.tuner.callback.evaluation.EvaluationCallback(query_data, index_data=None, metrics=None, exclude_self=True, limit=20, distance='cosine', num_workers=1)[source]#

Bases: finetuner.tuner.callback.base.BaseCallback

A callback that uses the Evaluator to calculate IR metrics at the end of each epoch. When used with other callbacks that rely on metrics, like checkpoints and logging, this callback should be defined first, so that it precedes in execution.

Parameters
  • query_data (DocumentArray) – Search data used by the evaluator at the end of each epoch, to evaluate the model.

  • index_data (Optional[ForwardRef]) – Index data or catalog used by the evaluator at the end of each epoch, to evaluate the model.

  • metrics (Optional[Dict[str, Tuple[Callable[…, float], Dict[str, Any]]]]) – A dictionary that specifies the metrics to calculate. It maps metric names to tuples of metric functions and their keyword arguments. If set to None, default metrics are computed.

  • exclude_self (bool) – Whether to exclude self when matching.

  • limit (int) – The number of top search results to consider when computing the evaluation metrics.

  • distance (str) – The type of distance metric to use when matching query and index docs, available options are 'cosine', 'euclidean' and 'sqeuclidean'.

  • num_workers (int) – The number of workers to use when matching query and index data.

on_fit_begin(tuner)[source]#

Called at the start of the fit method call, after all setup has been done, but before the training has started.

on_epoch_end(tuner)[source]#

Called at the end of an epoch, after both training and validation (or just training if no validaton is provided).