openai / clip-featurevis Goto Github PK

Can someone make a colab notebook for this?

In the model file there is a self.graph_def referenced on line 38 but I can't seem to see where it is defined.

Dear OpenAI team,

Thank you for sharing this great implementation, I really kile it.

In the section 'Understanding Language', you mention about:

If we fix a neuron on the vision side, we can search for the text that maximizes the logit. We do this with a hill climbing algorithm to find what amounts to the text maximally corresponding to that neuron.

Would you mind specify how you do this? You use a dataset of sentences, or optimize the senences using pretrained language model? The vision model only connects to language model in the output part, how you get the gradient to do optimization? How you do text feature visualization?

Thank you for your help.

Best Wishes,

Alex

Imagen' key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model.

Theyalso find that while T5-XXL and CLIP text encoders perform similarly on simple benchmarks such as MS-COCO, human evaluators prefer T5-XXL encoders over CLIP text encoders in both image-text alignment and image fidelity on DrawBench, a set of challenging and compositional prompts.

So try to visualize the features of T5?

T5: https://github.com/google-research/text-to-text-transfer-transformer