Question on frozen embeddings about vec2text HOT 4 CLOSED

There's no difference. If you're training for multiple epochs then you can save time by precomputing the embeddings and using the frozen embeddings parameter. You can also compute them on-the-fly, obviously, and save some disk space, but then you'll be recomputing them every epoch and wasting FLOPs. This might be the only option for large datasets though since embeddings take up a non-negligible amount of disk space.

I was also previously experimenting with learning the full system end-to-end, including the embedder. That's only possible when you don't precompute the embeddings, and set embedder_no_grad parameter to False. Most people probably don't want to do this, though.

from vec2text.

That makes sense. Thank you so much!

from vec2text.

Hi @jxmorris12, I have a follow-up question here.
If I set embedder_no_grad=False and use_frozen_embeddings_as_input=False, this means that the embedder will be trained as well right? Will the trained embedder be saved somewhere after training?

from vec2text.

Yep, that's right. I think the trained embedder will be saved as part of the InversionModel (InversionModel.embedder) but I'd have to check to be sure. It should be obvious because the loss curves are way steeper and reach much lower values when the embedder is trainable.

from vec2text.