dribnet / clipit_old Goto Github PK
View Code? Open in Web Editor NEWThis project forked from nerdyrodent/vqgan-clip
VQGAN+CLIP with some additional tuning. For notebooks and the command line.
License: MIT License
This project forked from nerdyrodent/vqgan-clip
VQGAN+CLIP with some additional tuning. For notebooks and the command line.
License: MIT License
Hey! I'm trying to run clipit on my own hardware (with an RTX A5000). I've got the environment set up more or less like the Colab notebook. I'm running this:
import sys
sys.path.append("clipit")
import clipit
clipit.reset_settings()
clipit.add_settings(prompts="Beirut skyline", aspect="widescreen")
clipit.add_settings(quality="better", scale=2.5)
clipit.add_settings(use_pixeldraw=True)
clipit.add_settings(num_cuts=10)
settings = clipit.apply_settings()
clipit.do_init(settings)
clipit.do_run(settings)
And getting a mysterious error from when calling loss.backward()
in the train function:
(lightning) root@pytorch-6dc4c57d8d-l2h26:/app# python tmp.py
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/comet_ml/monkey_patching.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/pkg_resources/__init__.py:1131: DeprecationWarning: Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.
self, resource_name
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/mlflow/types/schema.py:49: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
binary = (7, np.dtype("bytes"), "BinaryType", np.object)
Using device: cuda:0
Optimising using: AdamP
Using text prompts: ['Beirut skyline']
Using seed: 6415327989839999698
0it [00:00, ?it/s]/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/nn/functional.py:3613: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode)
/root/miniconda3/envs/lightning/lib/python3.7/site-packages/kornia/utils/helpers.py:96: UserWarning: torch.solve is deprecated in favor of torch.linalg.solveand will be removed in a future PyTorch release.
torch.linalg.solve has its arguments reversed and does not return the LU factorization.
To get the LU factorization see torch.lu, which can be used with torch.lu_solve or torch.lu_unpack.
X = torch.solve(B, A).solution
should be replaced with
X = torch.linalg.solve(A, B) (Triggered internally at /opt/conda/conda-bld/pytorch_1623448265233/work/aten/src/ATen/native/BatchLinearAlgebra.cpp:760.)
out1, out2 = torch.solve(input.to(dtype), A.to(dtype))
iter: 0, loss: 2.97693, losses: 1.04082, 0.961298, 0.974818
0it [00:01, ?it/s]Oops: runtime error:
Try reducing --num-cuts to save memory
0it [00:01, ?it/s]
Traceback (most recent call last):
File "tmp.py", line 11, in <module>
clipit.do_run(settings)
File "clipit/clipit.py", line 933, in do_run
raise e
File "clipit/clipit.py", line 925, in do_run
train(args, cur_iteration)
File "clipit/clipit.py", line 845, in train
loss.backward()
File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/_tensor.py", line 255, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/autograd/__init__.py", line 149, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/torch/autograd/function.py", line 87, in apply
return self._forward_cls.backward(self, *args) # type: ignore[attr-defined]
File "/root/miniconda3/envs/lightning/lib/python3.7/site-packages/diffvg-0.0.1-py3.7-linux-x86_64.egg/pydiffvg/render_pytorch.py", line 709, in backward
eval_positions.shape[0])
RuntimeError
I'm kind of at a loss, here (pun intended), since the RuntimeError doesn't have any message associated with it, and the function in question seems to be a compiled C function. Any pointers would be appreciated.
Currently clipit is a fork so the code can't be searched and it won't show up in searches on Github either. It's not even similar to VQGAN-CLIP so it's definitely beyond the usual idea of a fork.
Before things progress too much further it'd be for the best if this was remade so it can be searched properly, it's making it difficult trying to find what controls the color of diffvg and where it is, and it'll only get worse.
With windows 10 + VS2017/2019 (I tried both), I got the folling error preventing me from installing diffvg.
diffvg.cpp
c:\[...]\diffvg\pybind11\include\pybind11\cast.h(1405): error : too few arguments for template template pa
rameter "Tuple" [C:[...]\diffvg\build\temp.win-amd64-3.9\Release\diffvg.vcxproj]
detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=
std::pair, Ts=<T1, T2>]"
(1483): here
Is there anyway to make this work...?
sometimes it's useful to have static resources associated with the project. this is where they will go.
#@title Parameters
prompts = "trending on artstation" #@param {type:"string"}
#@markdown you can trade off speed for quality
quality = "best" #@param ["draft", "normal", "better", "best"]
#@markdown some people actually like square? ¯\(ツ)/¯
aspect = "square" #@param ["widescreen", "square"]
init_image = "1.jpeg"
#@markdown When you have the settings you want, press the play button on the left.
#@markdown The system will start drawing and show you results as it goes along.
#@markdown (I can get "better" running on the colabs, but not "best".
#@markdown Sometimes it is necessary to "Restart Runtime" to clear memory.)
import clipit
clipit.reset_settings()
clipit.add_settings(prompts=prompts, aspect=aspect, quality=quality,init_image=init_image)
settings = clipit.apply_settings()
clipit.do_init(settings)
clipit.do_run(settings)
The PixelDrawer currently has no support for limiting the colors - it only support full three channel RGB. It would be preferable to be able to limit the colorspace to either subsets of RGB or to a fixed color palette.
As an initial step, I would like to create a version of the PixelDrawer class that supports a grayscale image where R=G=B. In principle this should be possible by optimizing only one tensor per pixel and then having that tensor be connected to the R, G, and B channels.
The hope is that the PixelDrawer would be able to evolve from there to next support a discrete set of fixed colors, and ultimately also to optionally optimize the color lookup table as it also optimizes the color assignments of the pixels.
Note also that this issue is related up upstream issue BachiLi/diffvg#23
braceexpand
perlin_numpy
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.