Comments (2)
Not sure if this is too late to be helpful, but the following is about twice as fast as looping over a list of captions and seems to be what you want. I've cleaned it up form my own file, so I haven't had a chance to run it and there may be an error lurking somewhere. The basic idea is to replace tiling the tokens coming from a single prompt—see the multiplications by batch_size
in the original code—with additional tokens.
from PIL import Image
from glide_text2im.download import load_checkpoint
from glide_text2im.model_creation import (
create_model_and_diffusion,
model_and_diffusion_defaults,
model_and_diffusion_defaults_upsampler
)
import torch
import matplotlib.pyplot as plt
has_cuda = torch.cuda.is_available()
device = torch.device('cpu' if not has_cuda else 'cuda')
# Create base glide.
options = model_and_diffusion_defaults()
options['use_fp16'] = has_cuda
options['timestep_respacing'] = '100' # use 100 diffusion steps for fast sampling
glide, diffusion = create_model_and_diffusion(**options)
glide.eval()
if has_cuda:
glide.convert_to_fp16()
glide.to(device)
glide.load_state_dict(load_checkpoint('base', device))
print('total base parameters', sum(x.numel() for x in glide.parameters()))
guidance_scale = 3.0
upsample_temp = 0.997
# Create a classifier-free guidance sampling function
def model_fn(x_t, ts, **kwargs):
half = x_t[: len(x_t) // 2]
combined = torch.cat([half, half], dim=0)
model_out = glide(combined, ts, **kwargs)
eps, rest = model_out[:, :3], model_out[:, 3:]
cond_eps, uncond_eps = torch.split(eps, len(eps) // 2, dim=0)
half_eps = uncond_eps + guidance_scale * (cond_eps - uncond_eps)
eps = torch.cat([half_eps, half_eps], dim=0)
return torch.cat([eps, rest], dim=1)
def glide_generate(prompts):
"""Returns a tensor of images where the ith image is generated from the ith prompt in [prompts].
Args:
prompts -- list of string prompts
"""
batch_size = len(prompts)
tokens = [glide.tokenizer.encode(p) for p in prompts]
tokens_and_masks = [glide.tokenizer.padded_tokens_and_mask(t, options['text_ctx']) for t in tokens]
tokens = [t for t,_ in tokens_and_masks]
masks = [m for _,m in tokens_and_masks]
# Create the classifier-free guidance tokens (empty)
full_batch_size = batch_size * 2
uncond_tokens, uncond_mask = glide.tokenizer.padded_tokens_and_mask([], options['text_ctx'])
# Pack the tokens together into glide kwargs.
model_kwargs = dict(
tokens=torch.tensor(tokens + [uncond_tokens] * batch_size, device=device),
mask=torch.tensor(masks + [uncond_mask] * batch_size, dtype=torch.bool, device=device))
glide.del_cache()
samples = diffusion.p_sample_loop(
model_fn,
(full_batch_size, 3, options["image_size"], options["image_size"]),
device=device,
clip_denoised=True,
progress=True,
model_kwargs=model_kwargs,
cond_fn=None,
)[:batch_size]
glide.del_cache()
# Uncomment what's below to validate the function
# scaled = ((samples + 1)*127.5).round().clamp(0,255).to(torch.uint8).cpu()
# for s in scaled:
# plt.imshow(s.permute(1, 2, 0) )
# plt.show()
return (samples + 1) / 2
# Uncomment what's below to validate the function
# glide_generate(["a painting of a blue bird", "a painting of a red cat", "a painting of a purple apple"])
from glide-text2im.
Check the code of the notebook at:
from glide-text2im.
Related Issues (20)
- While running the clip_guided notebook in CPU mode I get: "RuntimeError - Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.FloatTensor instead" HOT 8
- Training parameters
- In GPU mode generated image is all black with NaN tensor values (no problems in CPU mode) HOT 8
- Better resolution images for inpainting HOT 4
- Has the inpainting Colab been developed with the CLIP-guided version?
- disappointed, looks the model is poor for unseen data HOT 2
- In the clipped guided version, were GLIDE (filtered) and CLIP trained together?
- Question about the CLIP model HOT 2
- Question about generating masks
- YouTube video walk-through of this codebase
- How could I load a mask generated by myself? HOT 2
- Ways to reduce number of failed inpaints?
- Fixing Random Seed
- About CLIP training on nosied images
- Experimental IS and FID values without classifier guidance and CLIP guidance
- Inpaiting fin-tune details HOT 1
- How can i use .py file to run project in pycharm?
- Something wrong with upsample-inpaint checkpoint
- Is the formula for CFG different from the reference?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from glide-text2im.