luciddreamer-cvlab / luciddreamer Goto Github PK

I am using ubuntu 22.04, I had many errors and spent a whole day getting this to work. I will outline the steps I needed to take to get it working. hopefully this helps if you get any of the errors mentioned in the title.

steps:

• conda create -n lucid python=3.9
• conda activate lucid

• conda install cuda -c nvidia/label/cuda-11.3.0

• pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d torch==2.0.1 torchvision==0.15.2 gradio omegaconf
• pip install timm==0.6.7
• pip install plyfile==0.8.1

• conda install gcc_linux-64=10 gxx_linux-64=10
• export CC=$CONDA_PREFIX/bin/gcc
• export CXX=$CONDA_PREFIX/bin/g++
• $CONDA_PREFIX/bin/g++ --version (checks it's installed)
• echo $CC (checks it's installed within conda)
• echo $CXX

• conda install -c conda-forge glm
• ls $CONDA_PREFIX/include | grep glm

• cd submodules/depth-diff-gaussian-rasterization-min
• python setup.py install

• cd ../simple-knn
• python setup.py install

My VPN doesn't open the website, can you provide an offline version of the code?

File "D:\Program Files\Python\Python39\LucidDreamer\scene\gaussian_model.py", line 26, in
from simple_knn._C import distCUDA2
ModuleNotFoundError: No module named 'simple_knn'

EDIT: I think I must have missed an error message on the first install attempt. Wiped everything, started over, reinstalled C++ and Cuda 11.8. No errors. 👍

EDIT: Closing, rather than deleting, because results I found when searching for answers related to this took me on an adventure. It could be helpful for someone to see the error and that reinstalling fixed it.

│ ❱ 376 │ W, H = traindata["frames"][0]["image"].size │
│ 377 │ # W, H = traindata["W"], traindata["H"] │
│ 378 │ │
│ 379 │ for camkey in presetdata: │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: list index out of range

num_rendered, color, depth, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 506.54 GiB (GPU 0; 24.00 GiB total capacity; 5.73 GiB already allocated; 15.99 GiB free; 5.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\ws3-01\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 493, in process_events
response = await self.call_prediction(awake_events, batch)
File "C:\Users\ws3-01\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 464, in call_prediction
raise Exception(str(error) if show_error else None) from error
Exception: None

What is the minimum configuration for operation?
Or am I encountering other issues...

Hello, I have run into an issue on windows

It happens after second "aligning" iteration finishes. Seems it cannot find output folder somehow. But I can't figure out what is wrong. os. path is fine, I have printed it out just now.

Would much appreciate any information.

log as follows:

Reading Training Transforms
Loading Training Cameras
Loading Preset Cameras
Number of points at initialisation :  1615299
Traceback (most recent call last):
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "c:\aigc\LucidDreamer\luciddreamer.py", line 170, in run
    gaussians = self.create(
  File "c:\aigc\LucidDreamer\luciddreamer.py", line 191, in create
    outfile = self.save_ply(os.path.join(self.save_dir, 'gsplat.ply'))
  File "C:\Users\..\anaconda3\envs\lucid\lib\ntpath.py", line 78, in join
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1522, in process_api
    result = await self.call_function(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
  File "c:\aigc\LucidDreamer\luciddreamer.py", line 170, in run
    gaussians = self.create(
  File "c:\aigc\LucidDreamer\luciddreamer.py", line 191, in create
    outfile = self.save_ply(os.path.join(self.save_dir, 'gsplat.ply'))
  File "C:\Users\.\anaconda3\envs\lucid\lib\ntpath.py", line 78, in join
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
  File "C:\Users\..\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None```

Based on the example below in the pubilc URL, I successfully ran the result once, but when I tried to run a new example, I kept getting the following error:

And here on the colab side of the code, the display is running:

However, all three screens that should have produced results reported "ERROR", and when I tried to run again, the code simply stopped running:

What I can determine is whether the colab linked device is the problem, because I have been observing the RAM condition of the linked device while running, and it is normally linked.
I also tried "disconnect and delete runtime", tried to run the entire colab code again, and found that the same error occurred, that is, a device would only run successfully the first time (I used colab to run once on another device, and the second time also reported an error), However, such a link error will occur the second time and thereafter.

(lucid) C:\DATA\LucidDreamer>python app.py
Traceback (most recent call last):
File "C:\DATA\LucidDreamer\app.py", line 55, in
from luciddreamer import LucidDreamer
File "C:\DATA\LucidDreamer\luciddreamer.py", line 41, in
from gaussian_renderer import render
File "C:\DATA\LucidDreamer\gaussian_renderer_init_.py", line 15, in
from depth_diff_gaussian_rasterization_min import GaussianRasterizationSettings, GaussianRasterizer
ModuleNotFoundError: No module named 'depth_diff_gaussian_rasterization_min'

Can I get a 3D Gaussian point cloud for real-time rendering? If I just generate a short RGB video, what is the use of it?

I tried running the colab notebook from here:
https://colab.research.google.com/github/camenduru/LucidDreamer-Gaussian-colab/blob/main/LucidDreamer_Gaussian_colab.ipynb

But after the first cell completed running, there wasn't any UI. Is there supposed to be a gradio link or something?

when i run python app.py this error and warning occurs

Can you get scene mesh output with texture colors?

An error occurred during the generation process :

The error seems to occur when the jpg image size is too large.
I got an error when the image was about 600KB.
If I downsize it to 70KB or so, it works fine.

Traceback (most recent call last):
File "/home/h/LucidDreamer/run.py", line 55, in
ld.create(rgb_cond, txt_cond, neg_txt_cond, args.campath_gen, args.seed, args.diff_steps)
File "/home/h/LucidDreamer/luciddreamer.py", line 65, in create
self.traindata = self.generate_pcd(rgb_cond, txt_cond, neg_txt_cond, pcdgenpath, seed, diff_steps)
File "/home/h/LucidDreamer/luciddreamer.py", line 375, in generate_pcd
imagej = interp_grid(pixel_coord_camj.transpose(1,0), pts_colors[valid_idxj], grid, method='linear', fill_value=0).reshape(H,W,3)
File "/home/h/miniconda3/envs/lucid/lib/python3.9/site-packages/scipy/interpolate/_ndgriddata.py", line 273, in griddata
ip = LinearNDInterpolator(points, values, fill_value=fill_value,
File "interpnd.pyx", line 290, in scipy.interpolate.interpnd.LinearNDInterpolator.init
File "_qhull.pyx", line 1827, in scipy.spatial._qhull.Delaunay.init
File "_qhull.pyx", line 277, in scipy.spatial._qhull._Qhull.init
ValueError: No points given

Hello everyone, have you ever encountered this problem

Unable to parse import in this file （gaussian_renderer/init.py）

……
65 / 70
66 / 70
67 / 70
68 / 70
69 / 70
70 / 70
Reading Training Transforms
Loading Training Cameras
Loading Preset Cameras
Number of points at initialisation : 1283959
Traceback (most recent call last):
File "C:\DATA\LucidDreamer\run.py", line 53, in
ld.create(rgb_cond, txt_cond, neg_txt_cond, args.campath_gen, args.seed, args.diff_steps)
File "C:\DATA\LucidDreamer\luciddreamer.py", line 188, in create
self.scene = Scene(self.traindata, self.gaussians, self.opt)
File "C:\DATA\LucidDreamer\scene_init_.py", line 33, in init
self.gaussians.create_from_pcd(info.point_cloud, self.cameras_extent)
File "C:\DATA\LucidDreamer\scene\gaussian_model.py", line 136, in create_from_pcd
dist2 = torch.clamp_min(distCUDA2(torch.from_numpy(np.asarray(pcd.points)).float().cuda()), 0.0000001)
MemoryError: bad allocation: cudaErrorMemoryAllocation: out of memory

Display card 3090 24GB memory

python run.py
Traceback (most recent call last):
File "/media/ma/新加卷/workspace/全景拼接毕设/LucidDreamer/run.py", line 5, in
from luciddreamer import LucidDreamer
File "/media/ma/新加卷/workspace/全景拼接毕设/LucidDreamer/luciddreamer.py", line 36, in
import gradio as gr
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/init.py", line 3, in
import gradio._simple_templates
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/_simple_templates/init.py", line 1, in
from .simpledropdown import SimpleDropdown
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/_simple_templates/simpledropdown.py", line 6, in
from gradio.components.base import FormComponent
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/components/init.py", line 1, in
from gradio.components.annotated_image import AnnotatedImage
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/components/annotated_image.py", line 12, in
from gradio.components.base import Component
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/components/base.py", line 21, in
from gradio.blocks import Block, BlockContext
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/blocks.py", line 28, in
from gradio import (
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/networking.py", line 20, in
from gradio.routes import App
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/gradio/routes.py", line 116, in
client = httpx.AsyncClient()
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/httpx/_client.py", line 1395, in init
proxy_map = self._get_proxy_map(proxies, allow_env_proxies)
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/httpx/_client.py", line 216, in _get_proxy_map
return {
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/httpx/_client.py", line 217, in
key: None if url is None else Proxy(url=url)
File "/home/ma/anaconda3/envs/lucid/lib/python3.9/site-packages/httpx/_config.py", line 334, in init
raise ValueError(f"Unknown scheme for proxy URL {url!r}")
ValueError: Unknown scheme for proxy URL URL('socks://0.0.0.0:7891/')

Crashing to this one:

Traceback (most recent call last): File "/content/LucidDreamer-Gaussian/app.py", line 14, in from luciddreamer import LucidDreamer File "/content/LucidDreamer-Gaussian/luciddreamer.py", line 34, in from gaussian_renderer import render File "/content/LucidDreamer-Gaussian/gaussian_renderer/init.py", line 14, in from depth_diff_gaussian_rasterization_min import GaussianRasterizationSettings, GaussianRasterizer ModuleNotFoundError: No module named 'depth_diff_gaussian_rasterization_min'

According to the paper, the model can take an image or text as input. But in the code, do we have to take both?

I tried using the Gradio interface locally but I am running into an error:

`50 / 50
Reading Training Transforms
Loading Training Cameras
Loading Preset Cameras
Number of points at initialisation : 1976286
Traceback (most recent call last):
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 455, in call_prediction
output = await route_utils.call_process_api(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1533, in process_api
result = await self.call_function(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1151, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\utils.py", line 678, in wrapper
response = f(*args, **kwargs)
File "C:\Users\info\LucidDreamer\luciddreamer.py", line 191, in create
outfile = self.save_ply(os.path.join(self.save_dir, 'gsplat.ply'))
File "C:\Users\info\anaconda3\envs\lucid\lib\ntpath.py", line 78, in join
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\queueing.py", line 455, in call_prediction
output = await route_utils.call_process_api(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1533, in process_api
result = await self.call_function(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\blocks.py", line 1151, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\anyio\to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\anyio_backends_asyncio.py", line 807, in run
result = context.run(func, *args)
File "C:\Users\info\anaconda3\envs\lucid\lib\site-packages\gradio\utils.py", line 678, in wrapper
response = f(*args, **kwargs)
File "C:\Users\info\LucidDreamer\luciddreamer.py", line 191, in create
outfile = self.save_ply(os.path.join(self.save_dir, 'gsplat.ply'))
File "C:\Users\info\anaconda3\envs\lucid\lib\ntpath.py", line 78, in join
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

The above exception was the direct cause of the following exception:`

When I use the commandline script run.py, it does work locally.

Running the gradio app locally on Windows under it's own virtual enviroment.
It starts fine and the examples work.
When I select another image (example attached) I get this error...

Traceback (most recent call last):
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\gradio\queueing.py", line 489, in call_prediction
    output = await route_utils.call_process_api(
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\gradio\blocks.py", line 1561, in process_api
    result = await self.call_function(
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\gradio\blocks.py", line 1179, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\anyio\to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\anyio\_backends\_asyncio.py", line 2134, in run_sync_in_worker_thread
    return await future
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\anyio\_backends\_asyncio.py", line 851, in run
    result = context.run(func, *args)
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\gradio\utils.py", line 678, in wrapper
    response = f(*args, **kwargs)
  File "D:\Tests\LucidDreamer\luciddreamer.py", line 170, in run
    gaussians = self.create(
  File "D:\Tests\LucidDreamer\luciddreamer.py", line 187, in create
    self.traindata = self.generate_pcd(rgb_cond, txt_cond, neg_txt_cond, pcdgenpath, seed, diff_steps)
  File "D:\Tests\LucidDreamer\luciddreamer.py", line 352, in generate_pcd
    depth_curr = self.d(image_curr)
  File "D:\Tests\LucidDreamer\luciddreamer.py", line 157, in d
    return self.d_model.infer_pil(im)
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\depth_model.py", line 141, in infer_pil
    out_tensor = self.infer(x, pad_input=pad_input, with_flip_aug=with_flip_aug, **kwargs)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\depth_model.py", line 126, in infer
    return self.infer_with_flip_aug(x, pad_input=pad_input, **kwargs)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\depth_model.py", line 110, in infer_with_flip_aug
    out = self._infer_with_pad_aug(x, pad_input=pad_input, **kwargs)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\depth_model.py", line 88, in _infer_with_pad_aug
    out = self._infer(x)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\depth_model.py", line 55, in _infer
    return self(x)['metric_depth']
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\zoedepth\zoedepth_v1.py", line 144, in forward
    rel_depth, out = self.core(x, denorm=denorm, return_rel_depth=True)
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\base_models\midas.py", line 262, in forward
    x = self.prep(x)
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\base_models\midas.py", line 186, in __call__
    return self.normalization(self.resizer(x))
  File "D:\Tests\LucidDreamer\./ZoeDepth\zoedepth\models\base_models\midas.py", line 173, in __call__
    return nn.functional.interpolate(x, (height, width), mode='bilinear', align_corners=True)
  File "D:\Tests\LucidDreamer\voc_luciddreamer\lib\site-packages\torch\nn\functional.py", line 3924, in interpolate
    raise TypeError(
TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.int32'>, <class 'numpy.int32'>]

Dear LucidDreamer Contributors,

I hope this message finds you well. I am reaching out to inquire about the possibility of integrating custom 3D models within the LucidDreamer framework, specifically in relation to the Gaussian splatting scenes generation process.

Having perused your impressive repository and the accompanying paper, I am intrigued by the potential applications of your domain-free generation technique. My interest lies in exploring how LucidDreamer might accommodate bespoke 3D models, which could be used in conjunction with the Gaussian splatting method to create unique and personalised scenes.

Would you be so kind as to provide guidance on whether the current implementation supports such an integration? If so, I would be most grateful for any documentation or examples that could assist me in understanding the process. Additionally, should this functionality not be present, might you consider it a worthwhile addition for future updates?

I am keen to experiment with LucidDreamer for an upcoming project that would greatly benefit from this capability. Your insights and advice would be invaluable in helping me to realise this endeavour.

Thank you for your time and the remarkable work you have shared with the community. I look forward to your response.

Best regards,
yihong1120

Thanks for the project ❤️ I made a colab. 🥳 I hope you like it. https://github.com/camenduru/LucidDreamer-Gaussian-colab

Everything appears to be working, but I keep getting a vertical Chinese scroll painting surrounded by other paintings, regardless of image input, prompts, or seed. The only thing that makes any difference is camera path.

Result I've been getting:
https://files.catbox.moe/jsdy2w.mp4

Note that this result only made sense the very first time I ran LucidDreamer. Since that time, I've tried many different prompts that should not produce anything like this.

EDIT: I tried outputting text from the command prompt window to a log file, which yielded error messages.
Log: https://pastebin.com/cX5ZzsBc

EDIT: Shortened and clarified initial post.

EDIT: The log mentioned "Using cache found in C:\Users\stupi/.cache\torch\hub\intel-isl_MiDaS_master" so I tried deleting that cache. No effect.

EDIT: Tried deleting all the pycache directories. No effect.

Hi,
The output ply file does not have the color information, right?

Can SD checkpoint useSDXL checkpoints?

Will it be possible to build it for Windows

See depth backward codes

I wonder why use projmatrix to compute gradients ?

In fact, the cam.z is computed from p_view.z, see depth forward.

Can you verify this potential bug ?

As I tested the back of the scene does not have much details, I hope adding more images would help solve that problem as the original gaussian splatting does. I do like your build for the gaussian splatting.

(lucid) PS C:\Users\xxxx\Deep\LucidDreamer> CUDA_VISIBLE_DEVICES=0 python app.py
CUDA_VISIBLE_DEVICES=0 : The term 'CUDA_VISIBLE_DEVICES=0' is not recognized as the name of a cmdlet, function, script
file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct
and try again.
At line:1 char:1

• CUDA_VISIBLE_DEVICES=0 python app.py

•   + CategoryInfo          : ObjectNotFound: (CUDA_VISIBLE_DEVICES=0:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException
  

I get the following error when I try to run app.py on Windows 10

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ Y:\ai\LucidDreamer\app.py:318 in │
│ │
│ 315 │
│ 316 │
│ 317 if name == 'main': │
│ ❱ 318 │ demo.launch() │
│ 319 │
│ │
│ C:\Python310\lib\site-packages\gradio\blocks.py:1727 in launch │
│ │
│ 1724 │ │ if not isinstance(self.blocked_paths, list): │
│ 1725 │ │ │ raise ValueError("blocked_paths must be a list of directories.") │
│ 1726 │ │ │
│ ❱ 1727 │ │ self.validate_queue_settings() │
│ 1728 │ │ │
│ 1729 │ │ self.config = self.get_config_file() │
│ 1730 │ │ self.max_threads = max( │
│ │
│ C:\Python310\lib\site-packages\gradio\blocks.py:1563 in validate_queue_settings │
│ │
│ 1560 │ │
│ 1561 │ def validate_queue_settings(self): │
│ 1562 │ │ if not self.enable_queue and self.progress_tracking: │
│ ❱ 1563 │ │ │ raise ValueError("Progress tracking requires queuing to be enabled.") │
│ 1564 │ │ │
│ 1565 │ │ for fn_index, dep in enumerate(self.dependencies): │
│ 1566 │ │ │ if not self.enable_queue and self.queue_enabled_for_fn(fn_index): │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Progress tracking requires queuing to be enabled.

Do you have plans to make Marigold compatible for depth map generation? In addition to ZoeDepth, it would be great to have the ability to generate depth maps using Marigold.

https://github.com/prs-eth/Marigold

The HuggingFace demo just throws an error, almost immediately when I try anything ... is there a problem here, or how I'm trying to use it?

Hello, and thank you for showcasing your impressive work.

I have an image that I would like to use as input, and I'm interested in understanding how to utilize the run.py script for generating 3D grayscale images (3d-gs) without providing a text prompt.

Could you please inform me on how to modify the run.py or the command I need to use to achieve this with the provided image?

Thank you for your assistance.

What kind of text prompts and negative prompts should I input to generate the expected results when I input a scene photo I took.

Hi team, I follow the instructions to install and run under the Ubuntu environment. However, I am encountering an error when running python setup.py install in the path submodules/depth-diff-gaussian-rasterization-min. The error message suggests the CUDA version is mismatched:
RuntimeError: The detected CUDA version (12.2) mismatches the version that was used to compile PyTorch (11.7). Please make sure to use the same CUDA versions.

I have pytorch 2.0.1 installed. is there any solution to fix this without downgrading the system CUDA? Thanks.

I'ts some kind of issue about the rendering. Sometimes I wait for rendering about 2-3 hours on i9 13900/3090 and I have just bright splash with strange ply file and some times using difirent models / images / promt I wait for 1.5-2 hours and get awarange results. I wondering maybe it's some kind a problem with my cuda 11.8 or drivers studio (latest) ? 1/105 2/105 caunter loading and even stugger sometimes to continue the render I pressing enter. CPU load 100% GPU 6-50% mostly 8% with about 9gb vram usage. One more question is it possible to save glb mesh somehow? Thank you.

when I execute the first command setup.py install, I get the following error, how should I solve, I have been troubled by this problem for a long time

Hi,

I met the follow error when loading sd inpainting:
OSError: Unable to load weights from pytorch checkpoint file for '../../ckpt/diff_inpaint/safety_checker/pytorch_model.bin' at '../../ckpt/diff_inpaint/safety_checker/pytorch_model.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

Do you have any idea about it?

Best,

In the Windows 10 system, the following error message appears when clicking run:

num_rendered, color, depth, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 270.43 GiB

When I clicked on the "Running on public URL" link, I clicked on the example below and entered the default images and parameters, but it didn't work.

The error output is:

Getting this error on second par of install ...

cd submodules/depth-diff-gaussian-rasterization-min
python setup.py install

(luciddreamer) PS C:\Users\xxxx\Deep\LucidDreamer\submodules\depth-diff-gaussian-rasterization-min> python setup.py install
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8'
running install
C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

    ********************************************************************************
    Please avoid running ``setup.py`` directly.
    Instead, use pypa/build, pypa/installer or other
    standards-based tools.

    See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
    ********************************************************************************

!!
self.initialize_options()
C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

    ********************************************************************************
    Please avoid running ``setup.py`` and ``easy_install``.
    Instead, use pypa/build, pypa/installer or other
    standards-based tools.

    See https://github.com/pypa/setuptools/issues/917 for details.
    ********************************************************************************

!!
self.initialize_options()
running bdist_egg
running egg_info
creating depth_diff_gaussian_rasterization_min.egg-info
writing depth_diff_gaussian_rasterization_min.egg-info\PKG-INFO
writing dependency_links to depth_diff_gaussian_rasterization_min.egg-info\dependency_links.txt
writing top-level names to depth_diff_gaussian_rasterization_min.egg-info\top_level.txt
writing manifest file 'depth_diff_gaussian_rasterization_min.egg-info\SOURCES.txt'
C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\torch\utils\cpp_extension.py:476: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file 'depth_diff_gaussian_rasterization_min.egg-info\SOURCES.txt'
adding license file 'LICENSE.md'
writing manifest file 'depth_diff_gaussian_rasterization_min.egg-info\SOURCES.txt'
installing library code to build\bdist.win-amd64\egg
running install_lib
running build_py
creating build
creating build\lib.win-amd64-cpython-39
creating build\lib.win-amd64-cpython-39\depth_diff_gaussian_rasterization_min
copying depth_diff_gaussian_rasterization_min_init_.py -> build\lib.win-amd64-cpython-39\depth_diff_gaussian_rasterization_min
running build_ext
C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
File "C:\Users\xxxx\Deep\LucidDreamer\submodules\depth-diff-gaussian-rasterization-min\setup.py", line 17, in
setup(
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_init_.py", line 107, in setup
return distutils.core.setup(**attrs)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\core.py", line 185, in setup
return run_commands(dist)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\core.py", line 201, in run_commands
dist.run_commands()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\dist.py", line 969, in run_commands
self.run_command(cmd)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\dist.py", line 1234, in run_command
super().run_command(command)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\command\install.py", line 80, in run
self.do_egg_install()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\command\install.py", line 129, in do_egg_install
self.run_command('bdist_egg')
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\dist.py", line 1234, in run_command
super().run_command(command)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\command\bdist_egg.py", line 164, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\command\bdist_egg.py", line 150, in call_command
self.run_command(cmdname)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\dist.py", line 1234, in run_command
super().run_command(command)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
self.build()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\command\install_lib.py", line 111, in build
self.run_command('build_ext')
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\dist.py", line 1234, in run_command
super().run_command(command)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\dist.py", line 988, in run_command
cmd_obj.run()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools\command\build_ext.py", line 84, in run
_build_ext.run(self)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 345, in run
self.build_extensions()
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
_check_cuda_version(compiler_name, compiler_version)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\torch\utils\cpp_extension.py", line 383, in _check_cuda_version
torch_cuda_version = packaging.version.parse(torch.version.cuda)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\pkg_resources_vendor\packaging\version.py", line 52, in parse
return Version(version)
File "C:\Users\xxxx\anaconda3\envs\luciddreamer\lib\site-packages\pkg_resources_vendor\packaging\version.py", line 196, in init
match = self._regex.search(version)
TypeError: expected string or bytes-like object
(luciddreamer) PS C:\Users\xxxx\Deep\LucidDreamer\submodules\depth-diff-gaussian-rasterization-min>

Congrats on this amazing work and thank you for the quick release of the code!

I am running the code with the rotate360 + headbanging setting using images I prepared. But, sometimes I encounter a phenomenon like the attached photo and video. Are there any tips or appropriate hyperparameter values to avoid this?

https://github.com/prs-eth/marigold

C:\Users\sd163\Anaconda3\envs\lucid\Lib\site-packages\torch\include\pybind11\cast.h(628): error: too few arguments for template template parameter "Tuple"
detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair, Ts=<T1, T2>]"
(725): here

C:\Users\sd163\Anaconda3\envs\lucid\Lib\site-packages\torch\include\pybind11\cast.h(721): error: too few arguments for template template parameter "Tuple"
detected during instantiation of class "pybind11::detail::tuple_caster<Tuple, Ts...> [with Tuple=std::pair, Ts=<T1, T2>]"
(725): here

2 errors detected in the compilation of "rasterize_points.cu".
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\bin\nvcc.exe' failed with exit code 1

The whole error is

ImportError: /home/hanhechong/anaconda3/envs/gaussian/lib/python3.7/site-packages/depth_diff_gaussian_rasterization_min-0.0.0-py3.7-linux-x86_64.egg/depth_diff_gaussian_rasterization_min/_C.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops7resize_4callERKNS_6TensorEN3c108ArrayRefIlEENS5_8optionalINS5_12MemoryFormatEEE

I think that is due to the incorrect installation of the submodule depth-diff-gaussian-rasterization-min. So I install it once again. The only warning is the CUDA version is 11.6 while the version to compile Pytorch is 11.7, which does not influence the installation process. So what's the problem?
By the way, how to use pip to install these submodules as the system suggests this way.
Thank you! Any help will be appreciated.

Thank you very much for your work. This is a great project, but i have a question about the code.

Why did only the transition vector of the first pose change, while the rest remained at 0.