Output structured diff results about benchmarkstt HOT 7 CLOSED

This is a good idea

from benchmarkstt.

There will other metrics. Each metric should have its own structure.

{
"hypothesis": "machine-transcript.txt",
"reference":"human-transcript.txt",
"metrics":[
  {"type":"word-diff", "result":{"substitution":5, "deletions":10, "insertions":1, "correct":37}},
  {"type":"wer", "result":0.76}
]
}

from benchmarkstt.

@MikeSmithEU in the future we may report metrics that are not numbers, so perhaps we should make all values strings, like:

{"type":"word-diff", "result":{"substitution":"5", "deletions":"10", "insertions":"1", "correct":"37"}},
  {"type":"wer", "result":"0.76"}

This way we leave the interpretation to the client.

from benchmarkstt.

Indeed @EyalLavi Yet when the type fits a js type I would not pass it as a string, but the type itself.
In practice this means supporting: array, object, int, float, string, null/undefined (would use null, never undefined).

from benchmarkstt.

The discussion above revolved around the summary of the diff results. I would like to add the diffs themsleves to the scope. This will allow apps to do their own processing on the results. How do we output those? Standard diff format wrap in the json? Or separately as standard diff?

from benchmarkstt.

For Release 1, output the worddiffs metric. Something like this:

{
	"worddiffs":
	[{
	"ref":"hello",
	"hyp":"hello" 	         //match
	},
	{
	"ref":"world",
	"hyp":null		//deletion
	},
	{
	"ref":null,
	"hyp":"foo"		//insertion
	},
	{
	"ref":"bar",
	"hyp":"gar"		//replacement
	}]
}

For future releases, consider using the canonical transcript format.

from benchmarkstt.

Discussion with @MikeSmithEU today:

• Output JSON format is metric-dependent
• Currently, -o json only

Example:

--worddiffs -o json

Produces:

[
    {"metric": "worddiffs",
    "result": [
        {
            "hypothesis": "this",
            "type": "equal",
            "reference": "this"
        },
        {
            "hypothesis": "is",
            "type": "equal",
            "reference": "is"
        },
        {
            "hypothesis": null,
            "type": "delete",
            "reference": "reference"
        }
     ]
}
]

from benchmarkstt.