Comments (1)
Hi, I'm glad you like our tool :)
If you mean candidate set in the sense of "all entities that can potentially be linked by a system", i.e., the (sub)set of entities from the KB used by the system:
I would say it depends on what exactly you want to evaluate.
If you want to know how well an entity linker performs on a given benchmark out of the box, then certainly its candidate set should be included in the evaluation. This means, ground truth entities that are not in the candidate set should not simply be ignored).
If you want to evaluate how well different disambiguation methods work on a given benchmark, independent of the candidate set, then a comparison can be made fair by using same candidate sets for different methods (as is often done in the literature) or potentially by filtering out ground truth entities that are not in the candidate set (although this is probably not the preferred method).
In ELEVANT, we have two error categories related to the candidate set: "Wrong Candidates" and "Multiple Candidates".
The meaning of "candidate set" here is slightly different from the one above (but closely related): The candidates are those entities which were considered for the disambiguation of a particular mention. Typically this candidate set is made up of all entities from the system's KB that have a name or alias which matches the mention.
The error category "Wrong Candidates" shows the percentage of disambiguation errors (not counting NER errors) where the true entity was not in the candidate set.
The error category "Multiple Candidates" shows the percentage of disambiguation errors (not counting NER errors) where the candidate set consisted of more than a single entity and the true entity was one of them. I.e., cases where the system chose a wrong entity from a set of candidate entities.
However, not all systems return information about these candidate sets. REL for example does not (as far as I know). For such linkers, these error rates cannot be computed.
I hope this helps!
from elevant.
Related Issues (14)
- Error while running evaluate_linking_results.py on new results file HOT 2
- Experiment working, but some parsing errors in the document display in GUI HOT 2
- Accidental experiment title cannot be expunged HOT 3
- GENRE using deprecated numpy attribute HOT 5
- Spacy model version issues? HOT 2
- Marking coref mentions in benchmark dataset HOT 4
- Article results not being displayed HOT 2
- Any plan of adding popular EL benchmarks HOT 3
- Error installing neuralcoref during docker build: Variables cannot be declared with 'cpdef'. Use 'cdef' instead. HOT 3
- Dependency conflict between radboud-el and xrenner HOT 4
- make download_all: alias_to_qids.db: truncated gzip input HOT 2
- Benchmark conversion from NIF format producing incorrect results HOT 2
- Experiment with REL failed on new benchmark dataset HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elevant.