zuoxingdong / recsys_metrics Goto Github PK

• Efficient (vectorized) implementations over mini-batches.
• Standard RecSys metrics: precision, recall, map, mrr, hr, ndcg, $\alpha$-ndcg
• Beyond-accuracy metrics: coverage, diversity, novelty, serendipity, etc.
• All metrics support a top-k argument.

You can install recsys_metrics from PyPI:

pip install recsys_metrics

Or you can also install the latest version from source:

pip install git+https://github.com/zuoxingdong/recsys_metrics.git@master

Note that we support Python 3.7+ only.

Let us take Hit Rate (HR) to illustrate how to use this library:

preds = torch.tensor([
    [.5, .3, .1],
    [.3, .4, .5]
])
target = torch.tensor([
    [0, 0, 1],
    [0, 1, 1]
])
hit_rate(preds, target, k=1, reduction='mean')

>> tensor(0.5000)

The first example in the batch does not have a hit (i.e. top-1 item is not a relevant item) and second example in the batch gets a hit (i.e. top-1 item is a relevant item). Thus, we have a hit-rate of 0.5.

The API of other metrics are of the same format.

Let us calculate serendipity:

preds = torch.tensor([
    [.5, .3, .1],
    [.3, .4, .5]
])
indexes = torch.tensor([
    [1, 2, 3],
    [4, 5, 6]
])
target = torch.tensor([
    [0, 1, 1],
    [0, 1, 1]
])

Suppose we are given the following tensors for the category of items and historical interactions for each example in the mini-batch:

item_category = torch.tensor([
    [2, 1, 8],
    [3, 9, 4]
])
hist_seq_category = torch.tensor([
    [5, 0, 1, 2, 3, 0, 4],
    [1, 3, 6, 0, 2, 3, 5]
])

Then, we could use a helper function provided in this library to calculate a tensor of unexpectedness (or surprise) with a window size of the most recent interactions:

unexpectedness = category_unexpectedness(
  item_category, 
  hist_seq_category=hist_seq_category[:, None, :].tile(1, 3, 1), 
  window_size=3
)

Finally, let us calculate the serendipity:

serendipity(preds, target, unexpectedness, k=3, reduction='mean')

>> tensor(0.6667)

This library also offers report functions to conveniently calculate standard IR metrics or beyond-accuracy metrics in an one-liner style.

rank_report(
    preds=preds, 
    target=target, 
    k=None, 
    to_item=True, 
    name_abbreviation=True,
    rounding=4,
)

>> {'prec': 0.5,
 'rec': 1.0,
 'map': 0.6667,
 'mrr': 0.6667,
 'hr': 1.0,
 'ndcg': 0.75}

beyond_accuracy_report(
    preds=preds,
    indexes=indexes,
    k=None,
    to_item=True,
    name_abbreviation=True,
    target=target,
    unexpectedness=unexpectedness.long(),
    rounding=4,
)

>> {'cat_cov': 1.0,
 'dist_cov': 1.0,
 'mil': 1.0,
 'serend': 0.6667}

Metrics	Single Example	Mini-Batch
Precision
Recall
MAP
MRR
HR
NDCG

This work is inspired by Torchmetrics from PyTorchLightning Team.

Please use this bibtex if you want to cite this repository in your publications:

@misc{recsys_metrics,
      author = {Zuo, Xingdong},
      title = {recsys_metrics: An efficient PyTorch implementation of the evaluation metrics in recommender systems.},
      year = {2021},
      publisher = {GitHub},
      journal = {GitHub repository},
      howpublished = {\url{https://github.com/zuoxingdong/recsys_metrics}},
    }