finetuner.tuner.evaluation module#
- class finetuner.tuner.evaluation.Evaluator(query_data, index_data=None, embed_model=None, metrics=None)[source]#
Bases:
object
The evaluator class.
Build an Evaluator object that can be used to evaluate the performance on a retrieval task.
- Parameters
query_data (DocumentArray) – The evaluation data. Both class and session format is accepted. In the case of session data, each document should contain ground truth matches from the index data under
doc.matches
. In the case of class format, each document should be mapped to a class, which is specified underdoc.tags[finetuner.__default_tag_key__]
.index_data (
Optional
[ForwardRef
]) – The data against which the query data will be matched. If not provided, query data will be matched against themselves. Both class and session format is accepted. In the case of session data, ground truth matches are included in the query_data, so no additional information is required in the index data. In the case of class data, each doc in the index data should have class labels indoc.tags[finetuner.__default_tag_key__]
.embed_model (
Optional
[ForwardRef
]) – The embedding model to use, in order to extract document representations. If set to None, documents are assumed to carry representations.metrics (
Optional
[Dict
[str
,Tuple
[Callable
,Dict
[str
,Any
]]]]) – A dictionary that specifies the metrics to calculate. It maps metric names to tuples of metric functions and their keyword arguments. If set to None, default metrics are computed.
- Returns
None
- static default_metrics()[source]#
Get default metrics.
- Return type
Dict
[str
,Tuple
[Callable
,Dict
[str
,Any
]]]
- evaluate(exclude_self=True, limit=20, distance='cosine', num_workers=1, **embed_kwargs)[source]#
Compute evaluation metrics. :type exclude_self:
bool
:param exclude_self: Whether to exclude self when matching. :type limit:int
:param limit: The number of top search results to consider when computing theevaluation metrics.
- Parameters
distance (
str
) – The type of distance metric to use when matching query and index docs, available options are'cosine'
,'euclidean'
and'sqeuclidean'
.num_workers (
int
) – The number of workers to use when matching query and index data.embed_kwargs – Keyword arguments to pass to the embed call.
- Return type
Dict
[str
,float
]- Returns
dictionary with evaluation metrics