Comments (3)
I think pointing at a good resource like https://en.wikipedia.org/wiki/Edit_distance could be enough.
from benchmarkstt.
I think pointing at a good resource like https://en.wikipedia.org/wiki/Edit_distance could be enough.
It is a two part problem: first generating the "delta" and then using that delta to calculate the WER. I've described how I see documenting WER in #31
The WER part is quite clear and non-confusing, however there is some variance in determining the deltas, and how to solve the LCS problem ( https://en.wikipedia.org/wiki/Longest_common_subsequence_problem ). This can impact the WER score.
To compare, sclite supports both unix diff and "dynamic programming string alignment", ref.
http://www1.icsi.berkeley.edu/Speech/docs/sctk-1.2/sclite.htm#dynam_prog_0
I propose using the Hunt–McIlroy algorithm ( https://en.wikipedia.org/wiki/Hunt%E2%80%93McIlroy_algorithm ) for now (this is what is used by diff
and is well established), we might add DP string alignment later or leave it to the community to implement.
Internal note (not necessarily for v1), future other easily implemented metrics:
- Weighted WER with 0.5 weights on insertions and deletions
- Word Accuracy (i.e. 1 - WER)
from benchmarkstt.
The Hunt–McIlroy algorithm is the one used internally by Python. Link to https://docs.python.org/3/library/difflib.html
from benchmarkstt.
Related Issues (20)
- Clarify status of API HOT 3
- Separate repo for connectors? HOT 1
- Prepare API for release HOT 2
- Inserts and deletions prioritised calculating worddiffs/WER HOT 14
- Metric: Named entity recognition EditNew Page HOT 3
- Restore build instructions HOT 2
- Change 'Benchmarking STT' to 'BenchmarkSTT' in docs
- Edit Distance documentation HOT 3
- Change VERSION to 1.0.0 before releasing.
- Add code coverage and test badges? HOT 1
- Fix typo in setup.pi
- Tutorial file contains duplicate content HOT 2
- Tests failing on fresh install HOT 5
- language support HOT 4
- Character Error Rate (CER) HOT 4
- Add BEER tutorial
- [Documentation] add package API docs, e.g. as PythonDoc
- Worddiff output: split into several lines?
- Worddiff output as HTML: default CSS for actually showing the differences?
- Add WER based on minimal edit distance (Levenshtein distance)? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchmarkstt.