abstract pytorch from eval so that it works at google about open_flamingo HOT 5 CLOSED

to be addressed after #72

from open_flamingo.

happy to take point on this, I am becoming passionate about cleaning up the eval codebase anyway, we need to clean it up a lot before things get too messy when adding more eval datasets

from open_flamingo.

cc @simonster , I will follow up on this once imagenet is set up and we can formulate a reasonable plan of attack

from open_flamingo.

I think this strategy should be compatible with any sane API, but my general plan for Flamingo eval is as follows:

1. Run the eval script once and save out all the data to TFRecords
2. Run bulk inference on the TFRecords using Flamingo and save the outputs
3. Run the eval script again and feed the Flamingo outputs in

This seems like the best option because it avoids trying to run the eval code in Google's monorepo, which would restrict what packages could be used, and I should already have bulk inference capabilities for these models. This strategy won't support a conversation between the model and evaluation script, but it should work for the datasets we are currently considering.

from open_flamingo.

SGTM. In this case, it sounds like we don't really need to "abstract away" the PyTorch so much as support dumping of the eval sets to tfrecords, which will actually be simpler anyway.

Whenever you have a chance @simonster (not urgent), feel free to share any info on the schema needed for the tfrecords to work with the bulk inference pipeline.

from open_flamingo.