Giter VIP home page Giter VIP logo

Comments (2)

flackbash avatar flackbash commented on August 26, 2024

Hi, there seem to be two problems:

The first one is that there is non-nested overlapping ground truth label included in your benchmark, namely this one:
{"id": 23, "span": [116, 146], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}
which overlaps with this label:
{"id": 8, "span": [107, 120], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}
Nested labels can be handled by ELEVANT (e.g. "[northeastern [Syria]]"), but not non-nested overlapping labels ("[U.S. mili{tary] said Tuesday. The milit}ary ..."). That label anyhow does seem to be a mistake in the data or in the parsing of the benchmark.

The other problem is that the benchmark labels (and label ids) are not sorted by their corresponding span start.
This should actually be automatically the case when using ELEVANT to add a benchmark in NIF format using the add_benchmark.py script (due to this line here). Is that how you added the benchmark or did you use another procedure?

I fixed these two problems and the benchmark was displayed just fine:

{"id": 1, "title": "1", "text": "A helicopter accident in northeastern Syria over the weekend left 22 American service members injured, the U.S. military said Tuesday.\n\n\nThe military statement said that the cause of the accident was under investigation and that no enemy fire involved.\u201cA helicopter mishap in northeastern Syria resulted in the injuries of various degrees of 22 U.S. service members,\u201d US Central Command said. \u201cNo enemy fire was reported.\u201d\n\n\n\"The service members are receiving treatment for their injuries and 10 have been evacuated to higher care facilities,\" Centcom added in a statement.\n\n\nA spokesman for the U.S.-backed Syrian Kurdish forces did not immediately respond to an Associated Press request for comment.\n\n\nThere are at least 900 U.S. forces in Syria on average, along with an undisclosed number of contractors. U.S. special operations forces also move in and out of the country, but are usually in small teams and are not included in the official count.\n\n\nU.S. forces have been in Syria since 2015 to assist the Kurdish-led Syrian Forces in the fight against the militant Islamic State group. Since the extremist group was defeated in Syria in March 2019, U.S. troops have been trying to prevent any comeback by IS, which swept through Iraq and Syria in 2014, taking control of large swaths of territory.\n\n\nHowever, IS sleeper cells remain a threat. There are also about 10,000 IS fighters held in detention facilities in Syria and tens of thousands of their family members living in two refugee camps in the country's northeast.\n\n\nOver the past years, U.S. troops have been subjected to attacks carried out by IS members and Iran-backed fighters there. In late March, a drone attack on a U.S. base killed a contractor and wounded five American troops and another contractor. In retaliation, U.S. fighter jets struck several locations around the eastern province of Deir el-Zour, which borders Iraq.\n\n\nU.S. Defense Secretary Lloyd Austin said at the time that the strikes were a response to the drone attack as well as a series of recent attacks against U.S.-led coalition forces in Syria by groups affiliated with Iran\u2019s Revolutionary Guard.\n\n\nIn a related development, Syrian Kurdish-led authorities announced Saturday that hundreds of IS fighters held in prisons around the region will be put on trial after their home countries refused to repatriate them.", "evaluation_span": [0, 2357], "labels": [{"id": 0, "span": [25, 43], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [1], "optional": false, "type": "Q27096213|Q43229"}, {"id": 1, "span": [30, 35], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 2, "span": [38, 59], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [3], "optional": false, "type": "OTHER"}, {"id": 3, "span": [43, 48], "entity_id": "Q858", "name": "Syria", "parent": 2, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 4, "span": [67, 92], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}, {"id": 5, "span": [107, 120], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 7, "span": [164, 168], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 8, "span": [214, 218], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 9, "span": [276, 294], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [10], "optional": false, "type": "Q27096213|Q43229"}, {"id": 10, "span": [282, 286], "entity_id": "Q30", "name": "United States of America", "parent": 9, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 11, "span": [345, 349], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 12, "span": [368, 386], "entity_id": "Q1476046", "name": "United States Central Command", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 13, "span": [544, 551], "entity_id": "Q1476046", "name": "United States Central Command", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 14, "span": [596, 600], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 15, "span": [608, 613], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 16, "span": [664, 680], "entity_id": "Q40469", "name": "Associated Press", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 17, "span": [727, 738], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 18, "span": [742, 747], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 19, "span": [809, 813], "entity_id": "Q30", "name": "United States of America", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 20, "span": [954, 965], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 21, "span": [979, 984], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 22, "span": [1022, 1027], "entity_id": "Q858", "name": "Syria", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 23, "span": [1070, 1089], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 24, "span": [1154, 1165], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 25, "span": [1210, 1212], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 26, "span": [1234, 1238], "entity_id": "Q796", "name": "Iraq", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 27, "span": [1314, 1316], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 28, "span": [1376, 1378], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 29, "span": [1551, 1562], "entity_id": "Q11211", "name": "United States Armed Forces", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 30, "span": [1609, 1611], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 31, "span": [1624, 1628], "entity_id": "Q794", "name": "Iran", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 32, "span": [1864, 1876], "entity_id": "Q239097", "name": "Deir ez-Zor", "parent": null, "children": [], "optional": false, "type": "Q27096213"}, {"id": 33, "span": [1892, 1896], "entity_id": "Q796", "name": "Iraq", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}, {"id": 34, "span": [1923, 1935], "entity_id": "Q941013", "name": "Lloyd Austin", "parent": null, "children": [], "optional": false, "type": "Q215627"}, {"id": 35, "span": [2113, 2139], "entity_id": "Q271110", "name": "Islamic Revolutionary Guard Corps", "parent": null, "children": [], "optional": false, "type": "Q43229"}, {"id": 36, "span": [2169, 2199], "entity_id": "Unknown", "name": "UnknownNoMapping", "parent": null, "children": [], "optional": false, "type": "OTHER"}, {"id": 37, "span": [2236, 2238], "entity_id": "Q2429253", "name": "Islamic State", "parent": null, "children": [], "optional": false, "type": "Q27096213|Q43229"}]}

I realized that other benchmark readers don't enforce this sorting, which is something I need to fix.

Thank you for reporting this in any case!

from elevant.

flackbash avatar flackbash commented on August 26, 2024

The sorting of the ground truth labels for two other benchmark readers was implemented in 59f97bf. I'll close this issue. Feel free to reopen it in case your problem was not solved.

from elevant.

Related Issues (14)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.