finetuner.tuner.evaluation module

class finetuner.tuner.evaluation.Evaluator(query_data, index_data=None, embed_model=None, metrics=None)[source]

Bases: object

The evaluator class.

Build an Evaluator object that can be used to evaluate the performance on a retrieval task.

Parameters

  • query_data (DocumentArray) – The evaluation data. Both class and session format is accepted. In the case of session data, each document should contain ground truth matches from the index data under doc.matches. In the case of class format, each document should be mapped to a class, which is specified under doc.tags[finetuner.__default_tag_key__].

  • index_data (Optional[ForwardRef]) – The data against which the query data will be matched. If not provided, query data will be matched against themselves. Both class and session format is accepted. In the case of session data, ground truth matches are included in the query_data, so no additional information is required in the index data. In the case of class data, each doc in the index data should have class labels in doc.tags[finetuner.__default_tag_key__].

  • embed_model (Optional[ForwardRef]) – The embedding model to use, in order to extract document representations. If set to None, documents are assumed to carry representations.

  • metrics (Optional[Dict[str, Tuple[Callable, Dict[str, Any]]]]) – A dictionary that specifies the metrics to calculate. It maps metric names to tuples of metric functions and their keyword arguments. If set to None, default metrics are computed.

Returns

None

static default_metrics()[source]

Get default metrics.

Return type

Dict[str, Tuple[Callable, Dict[str, Any]]]

evaluate(exclude_self=True, limit=20, distance='cosine', num_workers=1, **embed_kwargs)[source]

Compute evaluation metrics. :type exclude_self: bool :param exclude_self: Whether to exclude self when matching. :type limit: int :param limit: The number of top search results to consider when computing the

evaluation metrics.

Parameters

  • distance (str) – The type of distance metric to use when matching query and index docs, available options are 'cosine', 'euclidean' and 'sqeuclidean'.

  • num_workers (int) – The number of workers to use when matching query and index data.

  • embed_kwargs – Keyword arguments to pass to the embed call.

Return type

Dict[str, float]

Returns

dictionary with evaluation metrics