Comments (5)
@ken012git 🙏 do you want to try 0.2.4
? i think i found the issue 🤦♂️
from imagen-pytorch.
@ken012git forgot the residual 🤦 and also needed a feedforward after it anyways
from imagen-pytorch.
@ken012git thank you for the experiments! basically, in a lot of papers, researchers remove attention past a certain token length (1024 or 2048) since it is prohibitively expensive due to the quadratic compute. but i like to substitute them with linear attention, even if it is a bit weaker. my favorite linear attention remains https://arxiv.org/abs/1812.01243 , and here i am also giving it a depthwise conv recommended by the primer paper
from imagen-pytorch.
Sure! Thanks for your immediate response!
I would also like to know what causes the issue. =)
from imagen-pytorch.
Hi @lucidrains ,
I have tested v0.2.4 and the issue seems gone. Thanks!
# test model, resolution 64
unet1 = Unet(
dim = 32,
cond_dim = 512,
dim_mults = (1, 2, 4, 8),
num_resnet_blocks = (2, 2, 2, 2), # small
layer_attns = (False, False, False, True),
layer_cross_attns = (False, False, False, True),
# use_linear_attn = False,
use_linear_attn = True,
)
Loss curve, blue: use_linear_attn =False
,red: use_linear_attn =True
early stage results, left: use_linear_attn =False
,right: use_linear_attn =True
I am wondering we should use transformers or linear attention layers at this line that configured by use_linear_attn
.
Would you point me relevant papers? Thanks
from imagen-pytorch.
Related Issues (20)
- trainer.train_ Incorrect use of step in text to video
- how to train your inpainting model using my own dataset?? HOT 1
- App para contratar servicios de eventos
- Correct Way to train both unets HOT 1
- OutOfMemoryError HOT 2
- Assert prevents mixed_precision? HOT 3
- Question about retraining/fine-tuning EncoderModel with new words in `t5.get_tokenizer()` HOT 2
- Question about ConnectionError
- how ta activate shuffle for DataLoader?
- Question about reproducing training images using respective training texts
- text prompts
- Can anyone look over my script and see if I've made any errors?
- Triton Python 3.10 and up HOT 1
- Super-resolution not completely respecting input low resolution image
- Why concatenate lowres before last conv?
- GPU memory increases with the number of GPUs used
- Subprocess Error during Wheel Build: Missing File
- Backward problem with using DDP
- NameError: name "model_validator" is not defined with version 1.26.1 HOT 1
- Any specific reason sampling is not in FP16? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from imagen-pytorch.