zbmed-semtec / trec-doc-2-doc-relevance Goto Github PK

Please have a look to the description of the dataset for doc-2-doc assessment. For instance, for topic 100 I was expecting only 7 references documents (see https://github.com/zbmed-semtec/TREC-doc-2-doc-relevance/blob/main/code/process.md) but there are 22 (i.e., all of the topic relevant documents). Also, each reference document should have 20 documents to be assessed but that is not always the case. Thanks.

Subtasks in this issue

• Chose the right number of reference documents per topic
• Each reference article should have exactly 20 documents to be assessed (randomly select 6 to 9 from the topic relevant ones, randomly select 6 to 9 from the topic partially relevant ones, and the rest --to complete 20, from the topic non-relevant ones)
• recreate the database used by the app

Build Dataset for assessment backend

1. Build topic_reference_pmid.tsv
   - Two column tsv with Topic No. and PMID
   - includes randomly selected ref candidates - Amount in TREC Topics List
2. Build topic_reference_and_documents.tsv
   - for each reference article in topic_reference_pmid.tsv 20 articles for assessment are randomly selected (6-9 def relevant, 6-9 partially relevant, rest irrelevant)
   - order by the first column, then the second and then the third.

Once the app moves to a container, how can we get the data from the database corresponding to all users?
Options:

• Keep the database outside the container and start the container in such a way that it access the 'external' database
• Document a process that allows to access the instance while running to copy the database. This might be the simplest option, it might required to stop the app so the databased is not locked although a copy is a read-only action.

Not implemented yet

• Fix the algorithm and regenerate candidates only for those topics that Olga has not touched
• For those topics that Olga has already processed (even if incomplete), please manually remove the candidate article which is the same as the reference article. Choose the replacement randomly within the topic.
• Update the database with the manual changes from 2 and the new sets from 1

Create a a database to store everything

• Reduce doc-to-be-assessed from 20 to 15.
• 15 doc-to-be-assessed should have 4 to 6 definitely relevant articles, 4 to 6 partially and rest non-relevant.

• Create the login feature
• Store the users in the database

• Display arrows for next/prev articles
• Progressbar at assessment article
• Display progress for reference article (needs database)
• Display progress for overall topic (needs database upgrade