kocmitom / mt-thresholds Goto Github PK

Would be great to have metricx models in the benchmarks and online tool: https://github.com/google-research/metricx

I have added new metrics into thresholds, can we make them partly hidden somehow (user needs to click a button for example)? Both in the selection list and in printed list?

Thanks for the work.

Regarding the tool: https://kocmitom.github.io/MT-Thresholds/

What does it mean to have +1 score increment having certain accuracy with humans? Wouldn't it make more sense that X score has Y human-agreement score. +1 increment from 20 to 21 is surely not the same as +1 from 80 to 81.

Having a tool that can be called as:

import mtthresholds

accuracy = mtthresholds.accuracy(delta, metric_name)

Would be really helptul to disseminate the work.

When placing the difference, one may place negative sign (for degradations), which now is resolved as is. We should always use absolute value (but perhaps we should alert user?)

When running mt_thresholds.accuracy(-1.0, "bleu') it doesn't take only the absolute value, but tries to project sigmoid to negative values