How much do the new automatic annotations differ from what a human listening test would conclude? about medleydb HOT 4 CLOSED

@carlthome

I'm looking forward to MedleyDB 2.0 (when will it be available?) and intend to use it for various MIR tasks.

A partial version (~90 tracks) is ready and I'm happy to share it with you. I'm waiting for the remaining tracks to be finished before fully releasing it to the wild. I plan to have it out by the end of the month ::fingers crossed::

However, I read that the new annotations are automatically generated and worry that this causes a chicken and egg problem. My hope was to use the annotations for training multi-f0 estimation models, but surely an upper bound on f-measure will be introduced by the fact that the annotations have been automatically generated themselves.

Yes, you're right. There's unfortunately no perfect solution. My hope is that as the dataset grows, automatic annotations that are found to be problematic can be human corrected and checked into this repository as a form of soft crowdsourcing.

Could you expand a little on how the new annotations have been developed?

They're generated using pyin, but naturally we can only do this on tracks with no bleed and that contain only one pitched note at a time. Sadly, we currently don't have annotations for instruments like piano and guitar. We're working on a solution for that, but for now the "multif0" annotations have to be paired with partial mixes, where the guitars/pianos/etc are removed.

How much do they differ compared to what human listeners would annotate?

I recently computed the automatic annotations generated in the same way on the original release and compared them with the human generated annotations. The overall accuracy was 80% on average.

Particularly multi-f0 annotations are difficult to get right but I'm also concerned about onset annotations (and even melody annotations to some extent).

Why are you concerned about the onset annotations? To me those are the easiest to estimate (when you have stems). Can you elaborate?

from medleydb.

I'm waiting for the remaining tracks to be finished before fully releasing it to the wild. I plan to have it out by the end of the month ::fingers crossed::

🥇 😃

My hope is that as the dataset grows, automatic annotations that are found to be problematic can be human corrected and checked into this repository as a form of soft crowdsourcing.

Sounds reasonable! Then the 20% accuracy gap could be reduced pretty quickly, I hope.

Sadly, we currently don't have annotations for instruments like piano and guitar.

😢

Why are you concerned about the onset annotations? To me those are the easiest to estimate (when you have stems). Can you elaborate?

While onset detection is easier than multi-f0 estimation I have problems with instruments with weak or no transients even with spectral flux or ConvNets. For drums and piano you'd probably get an accuracy over 90% without much trouble, but for violin and vocals it's often a lot lower compared to a human baseline, particularly with repeated notes like violin_example.ogg.zip. Therefore I'd be very cautious about looking at a global accuracy across all instruments. It would be good to also group f-measures per instrument type.

from medleydb.

Oh yeah, thanks for answering this issue by the way! 👍

from medleydb.

While onset detection is easier than multi-f0 estimation I have problems with instruments with weak or no transients even with spectral flux or ConvNets.

Ah I guess that's the discrepancy - we estimate range activations not onsets, targeted at the source id problem, not onset detection.

from medleydb.