jd-coderepos / sota Goto Github PK

This repository hosts the official SimpleText Task4 @ CLEF'24, i.e. the SOTA? Tracking the State-of-the-Art in Scholarly Publications task corpus. The full corpus is released in the dataset repository organized as follows:

[dataset]/	
     |--- [train]/
	     |--- [article-id-folder]/
		 |    |--- [article-id].tex
		 |    |--- annotations.json	
	     |___ ...
     |--- [validation]/
	     |--- [article-id-folder]/
		 |    |--- [article-id].tex
		 |    |--- annotations.json	
	     |___ ...		 
     |--- [test1-few-shot-papers]/	
	     |--- [article-counter-folder]/
		 |    |--- [article-id].tei.xml
	     |___ ...
     |--- [test1-few-shot-annotations]/	   	   # hidden during the competition
	     |--- [article-counter-folder]/
		 |    |--- annotations.txt		
		 |    |--- code-link.txt	   # optional	
	     |___ ...		 
     |--- [test2-zero-shot-papers]/				
	     |--- [article-counter-folder]/
		 |    |--- [article-id].tei.xml
	     |___ ...
     |--- [test2-zero-shot-annotations]/	   # hidden during the competition
	     |--- [article-counter-folder]/
		 |    |--- annotations.txt		
		 |    |--- code-link.txt	   # optional	
	     |___ ...			 

The dataset dump originates from paperswithcode.com.

Each folder in the respective dump corresponds to a scholarly article originally downloaded in LaTeX format from arXiv.

There are 12,288+100 total papers in the train+validation sets,respectively. Furthermore, note each annotations.json file either contains (task, dataset, metric, score) annotations for all papers reporting model scores. Otherwise annotations.json contains the value "unanswerable." This is for those papers that do not report any model scores and therefore leaderboards cannot be populated from them. Models trained on our dataset should, in a first step, distinguish papers with leaderboards and those without, then for the former set of papers, extract their leaderboard tuples as annotations. The train dataset has 7,936 papers with leaderboard annotations and the remaining 4,352 papers without leaderboard annotations and therefore annotated as "unanswerable." The validation dataset has 51 papers with leaderboard and 49 papers without leaderboard annotations.

Below are provided some detailed statistics relevant to the leaderboard annotations in our dataset offering a glimpse into the corpus.

Ten most common Tasks, Datasets, and Metrics in the Train+Validation set:

Ten most common (Task, Dataset, Metric) triples in Train+Validation Set:

Since each paper is accompanied with an annotations file, this section concludes with statistics for each of the four types in the tuple, what proportion of those can actually be found in the accompanying full-text.

• for Tasks, 60.24% of the annotation labels can be found in the accompanying paper full-text.
• for Datasets, 45.48% of the annotation labels can be found in the accompanying paper full-text.
• for Metrics, 42.69% of the annotation labels can be found in the accompanying paper full-text.
• for Scores, 58.86% of the annotations can be found in the accompanying paper full-text.

Shield:

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.