Comments (7)
This is a good idea
from benchmarkstt.
There will other metrics. Each metric should have its own structure.
{
"hypothesis": "machine-transcript.txt",
"reference":"human-transcript.txt",
"metrics":[
{"type":"word-diff", "result":{"substitution":5, "deletions":10, "insertions":1, "correct":37}},
{"type":"wer", "result":0.76}
]
}
from benchmarkstt.
@MikeSmithEU in the future we may report metrics that are not numbers, so perhaps we should make all values strings, like:
{"type":"word-diff", "result":{"substitution":"5", "deletions":"10", "insertions":"1", "correct":"37"}},
{"type":"wer", "result":"0.76"}
This way we leave the interpretation to the client.
from benchmarkstt.
Indeed @EyalLavi Yet when the type fits a js type I would not pass it as a string, but the type itself.
In practice this means supporting: array, object, int, float, string, null/undefined (would use null, never undefined).
from benchmarkstt.
The discussion above revolved around the summary of the diff results. I would like to add the diffs themsleves to the scope. This will allow apps to do their own processing on the results. How do we output those? Standard diff format wrap in the json? Or separately as standard diff?
from benchmarkstt.
For Release 1, output the worddiffs
metric. Something like this:
{
"worddiffs":
[{
"ref":"hello",
"hyp":"hello" //match
},
{
"ref":"world",
"hyp":null //deletion
},
{
"ref":null,
"hyp":"foo" //insertion
},
{
"ref":"bar",
"hyp":"gar" //replacement
}]
}
For future releases, consider using the canonical transcript format.
from benchmarkstt.
Discussion with @MikeSmithEU today:
- Output JSON format is metric-dependent
- Currently,
-o json
only
Example:
--worddiffs -o json
Produces:
[
{"metric": "worddiffs",
"result": [
{
"hypothesis": "this",
"type": "equal",
"reference": "this"
},
{
"hypothesis": "is",
"type": "equal",
"reference": "is"
},
{
"hypothesis": null,
"type": "delete",
"reference": "reference"
}
]
}
]
from benchmarkstt.
Related Issues (20)
- Clarify status of API HOT 3
- Separate repo for connectors? HOT 1
- Prepare API for release HOT 2
- Inserts and deletions prioritised calculating worddiffs/WER HOT 14
- Metric: Named entity recognition EditNew Page HOT 3
- Restore build instructions HOT 2
- Change 'Benchmarking STT' to 'BenchmarkSTT' in docs
- Edit Distance documentation HOT 3
- Change VERSION to 1.0.0 before releasing.
- Add code coverage and test badges? HOT 1
- Fix typo in setup.pi
- Tutorial file contains duplicate content HOT 2
- Tests failing on fresh install HOT 5
- language support HOT 4
- Character Error Rate (CER) HOT 4
- Add BEER tutorial
- [Documentation] add package API docs, e.g. as PythonDoc
- Worddiff output: split into several lines?
- Worddiff output as HTML: default CSS for actually showing the differences?
- Add WER based on minimal edit distance (Levenshtein distance)? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchmarkstt.