Is your feature request related to a problem? Please describe.
I'm always frustrated when I am aligning the system submissions and the text descriptions ("metadata") that people have submitted through the form like https://forms.gle/4MEq8pG9zn6GTTncA .
I would like OCELoT to collect not only system outputs but also these details, in a way which absolutely clearly records the correspondence between system runs and the metadata.
Describe the solution you'd like
My view of WMT news translation task is this (your view may be slightly different):
- Each participating institution can form one or more teams.
- Each team can submit many systems, but one has to be marked as primary.
To report clearly on this in the paper:
- Each team should report its institution name.
- The names of institutions are usually clear, but we should still ask for them, esp. when startups and other companies participate.
- Each team has to suggest its name (e.g. "Donald's Nephews") for the main text and also a short name (e.g. "Donalds") for the result tables.
- In the result tables, the team short name refers to the primary submission of that team.
- Each submitted system needs to have a unique name.
- The name can be implied, e.g. "Donald's Nephews Primary Submission", "Donald's Nephews Contrastive 1", "Donald's Nephews Contrastive 2".
- Each submitted system needs to receive metadata, i.e. the paragraph for the Findings and indication of the data and system features used.
- The metadata are most important for the primary submissions. For contrastive submissions, they can be abridged.
When submitting any system run (which is presumably assigned a system ID in OCELoT), the submitter should be redirected to a web form to fill all the metadata. The system ID should be pre-filled in the form, and so should be the team and institution name, if known.
Ideally, the system submission would not be deemed completed if this form was not filled. This could be achieved by a simple export of form results to a URL where OCELoT would check and highlight the submission green if the system ID had any entry there.
I suggest using Google Forms to collect the metadata, because they offer the flexibility we need across years. Also, they easily offer pre-filled URLs, e.g.:
https://docs.google.com/forms/d/e/1FAIpQLSeQ-R9DAiY56WGmzInggusVQXQDx8YvZnETWn26DEh5RkqQ0w/viewform?usp=pp_url&entry.1616043926=Ondrej+prefilled+his+name&entry.63843270=My+prefilled+system+name
Describe alternatives you've considered
We do it as before, circulate the form around and then frantically chase people to answer, and fully make up the few entries for which the metadata never arrives.