Comments (4)
There's no difference. If you're training for multiple epochs then you can save time by precomputing the embeddings and using the frozen embeddings parameter. You can also compute them on-the-fly, obviously, and save some disk space, but then you'll be recomputing them every epoch and wasting FLOPs. This might be the only option for large datasets though since embeddings take up a non-negligible amount of disk space.
I was also previously experimenting with learning the full system end-to-end, including the embedder. That's only possible when you don't precompute the embeddings, and set embedder_no_grad parameter to False. Most people probably don't want to do this, though.
from vec2text.
That makes sense. Thank you so much!
from vec2text.
Hi @jxmorris12, I have a follow-up question here.
If I set embedder_no_grad=False
and use_frozen_embeddings_as_input=False
, this means that the embedder will be trained as well right? Will the trained embedder be saved somewhere after training?
from vec2text.
Yep, that's right. I think the trained embedder will be saved as part of the InversionModel (InversionModel.embedder) but I'd have to check to be sure. It should be obvious because the loss curves are way steeper and reach much lower values when the embedder is trainable.
from vec2text.
Related Issues (20)
- Sample to Invert embeddings not working (anymore) HOT 2
- Why padding all text to max length in trainer.base. _get_decoded_sequences HOT 2
- Question about embedding_transform HOT 2
- Request for GTR Inverters Checkpoint HOT 1
- Reproducing results from paper HOT 17
- LLAMA-2 inversion models HOT 1
- Examining hypothesis embeddings HOT 3
- Were the gtr models trained without normalization? HOT 2
- Question about storage path HOT 1
- Using Initial Inversion Model Only HOT 7
- Requirements file has no versions pinned
- Support other embeddings
- Languages supported?
- llama-2-7b inversion
- about demo HOT 1
- Error with vec2text.invert_embeddings when set num_steps parameter HOT 3
- Accidental issue
- Multi-GPU Support HOT 1
- About hypothesis generation HOT 1
- Any limitation on the targeted embedding model? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vec2text.