Question: Can the number of "adaptable" layers be significantly reduced? Can training be optimized for "extremely consumer" grade GPUs like the 3060 Ti (8GB VRAM)? about llama-adapter HOT 3 OPEN

We are optimizing the training and inference GPU memory costs. Information will be updated soon.
Thanks for your interest in LLaMa-Adapter.

from llama-adapter.

I have an inverse question: can the number of adaptable layers or the adapter parameters be dynamic and thus increased dramatically?

from llama-adapter.

@teknium1 Very interesting question. We list some related research paper below:

Improving GANs with A Dynamic Discriminator
PonderNet: Learning to Ponder
Dynamic Early Exiting for Accelerating BERT Inference](https://arxiv.org/abs/2004.12993)

We will explore this interesting idea in the future.

from llama-adapter.