saharmor / dalle-playground Goto Github PK

The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html

Traceback (most recent call last):
  File "/tmp/pip-install-bjfk8s96/pillow/setup.py", line 996, in <module>
    zip_safe=not (debug_build() or PLATFORM_MINGW),
  File "/home/anyuser/Projects/IA/dalle-playground/backend/mydalee_env/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/anyuser/Projects/IA/dalle-playground/backend/mydalee_env/lib/python3.7/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/anyuser/Projects/IA/dalle-playground/backend/mydalee_env/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 84, in run
    _build_ext.run(self)
  File "/home/anyuser/anaconda3/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/tmp/pip-install-bjfk8s96/pillow/setup.py", line 804, in build_extensions
    raise RequiredDependencyException(f)
__main__.RequiredDependencyException: zlib

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-install-bjfk8s96/pillow/setup.py", line 1009, in <module>
    raise RequiredDependencyException(msg)
__main__.RequiredDependencyException:

The headers or library files could not be found for zlib,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
   https://pillow.readthedocs.io/en/latest/installation.html


----------------------------------------

sudo npm install -g localtunnel
lt --port 8080

your url is: https://*********.loca.lt/
2022-04-11 17:49:18.477391: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
wandb: Currently logged in as: anony-mouse-234245 (use `wandb login --relogin` to force relogin)
wandb: Tracking run with wandb version 0.12.14
wandb: Run data is saved locally in /content/wandb/run-20220411_174918-1wn83s1g
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run different-dust-2
wandb: ⭐️ View project at https://wandb.ai/anony-mouse-234245/uncategorized?apiKey=d234eca2f0b786a223ecace15cce6228271a435d
wandb: 🚀 View run at https://wandb.ai/anony-mouse-234245/uncategorized/runs/1wn83s1g?apiKey=d234eca2f0b786a223ecace15cce6228271a435d
wandb: WARNING Do NOT share these links with anyone. They can be used to claim your runs.
wandb: Downloading large artifact model-mheh9e55:latest, 1673.44MB. 7 files... Done. 0:0:0
tcmalloc: large alloc 1751359488 bytes == 0x5566ab700000 @  0x7f29c23b01e7 0x55669ed79338 0x55669ed43ad7 0x55669eec2575 0x55669ee5c8b8 0x55669ed473a2 0x55669ee25c6e 0x55669ed46ff9 0x55669ee38d4d 0x55669edbaec8 0x55669edb5a2e 0x55669ed4913c 0x55669ed49341 0x55669edb7ff1 0x55669edb5cdd 0x55669ed4888a 0x55669edb7719 0x55669edb5a2e 0x55669edb5723 0x55669ee7f812 0x55669ee7fb8d 0x55669ee7fa36 0x55669ee57183 0x55669ee56e2c 0x7f29c119ac87 0x55669ee56d0a
wandb: Downloading large artifact model-mheh9e55:latest, 1673.44MB. 7 files... Done. 0:0:0
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
INFO:werkzeug: * Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
INFO:werkzeug:127.0.0.1 - - [11/Apr/2022 17:50:33] "OPTIONS / HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [11/Apr/2022 17:51:42] "OPTIONS / HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [11/Apr/2022 17:51:42] "GET / HTTP/1.1" 200 -
INFO:werkzeug:127.0.0.1 - - [11/Apr/2022 17:51:50] "OPTIONS /dalle HTTP/1.1" 200 -
  0%|          | 0/2 [00:00<?, ?it/s]

2022-06-07 18:40:23,810 DEBUG   HandlerThread:20932 [meta.py:probe():209] probe
2022-06-07 18:40:23,821 DEBUG   HandlerThread:20932 [git.py:repo():29] git repository is invalid

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.

WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
--> Starting DALL-E Server. This might take up to two minutes.

2022-06-06 13:07:53.761716: I external/org_tensorflow/tensorflow/core/util/util.cc:168] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-06-06 13:07:53.761954: I external/org_tensorflow/tensorflow/core/tpu/tpu_initializer_helper.cc:259] Libtpu path is: libtpu.so
2022-06-06 13:07:53.785866: I external/org_tensorflow/tensorflow/core/util/util.cc:168] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-06-06 13:07:54.419499: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2022-06-06 13:07:55.545835: I external/org_tensorflow/tensorflow/compiler/xla/service/service.cc:174] XLA service 0x555e2489cb80 initialized for platform Interpreter (this does not guarantee that XLA will be used). Devices:
2022-06-06 13:07:55.545853: I external/org_tensorflow/tensorflow/compiler/xla/service/service.cc:182]   StreamExecutor device (0): Interpreter, <undefined>
2022-06-06 13:07:55.547821: I external/org_tensorflow/tensorflow/compiler/xla/pjrt/tfrt_cpu_pjrt_client.cc:176] TfrtCpuClient created.
2022-06-06 13:07:55.548231: I external/org_tensorflow/tensorflow/stream_executor/tpu/tpu_platform_interface.cc:74] No TPU platform found.
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
--> Starting DALL-E Server. This might take up to two minutes.

$ python
Python 3.9.12 (main, Apr  5 2022, 06:56:58) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.

2022-06-04 13:02:17.271909: W external/org_tensorflow/tensorflow/core/common_runtime/bfc_allocator.cc:479] Allocator (GPU_0_bfc) ran out of memory trying to allocate 768.00MiB (rounded to 805306368)requested by op 
2022-06-04 13:02:17.272293: W external/org_tensorflow/tensorflow/core/common_runtime/bfc_allocator.cc:491] *********************************************************************_****************************__
2022-06-04 13:02:17.273116: E external/org_tensorflow/tensorflow/compiler/xla/pjrt/pjrt_stream_executor_client.cc:2141] Execution of replica 0 failed: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 805306368 bytes.
BufferAssignment OOM Debugging.
BufferAssignment stats:
             parameter allocation:    16.4KiB
              constant allocation:        64B
        maybe_live_out allocation:    5.25GiB
     preallocated temp allocation:  288.04MiB
  preallocated temp fragmentation:       496B (0.00%)
                 total allocation:    5.53GiB
              total fragmentation:     5.3KiB (0.00%)

Error querying DALL-E service. Check your backend server logs.

Access to fetch at 'https://empty-oranges-juggle-*-*-*-222.loca.lt/dalle' from origin 'https://saharmor.me' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

--> Starting DALL-E Server. This might take up to two minutes.
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Some of the weights of DalleBart were initialized in float16 precision from the model checkpoint at /var/folders/96/q4j34l5j1k14yq56kcys66q00000gn/T/tmptpfziy1j:
[('lm_head', 'kernel'), ('model', 'decoder', 'embed_positions', 'embedding'), ('model', 'decoder', 'embed_tokens', 'embedding'), ('model', 'decoder', 'final_ln', 'bias'), ('model', 'decoder', 'layernorm_embedding', 'bias'), ('model', 'decoder', 'layernorm_embedding', 'scale'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'k_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'out_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'q_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'v_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'k_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'out_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'q_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'v_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'Dense_0', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'Dense_1', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'Dense_2', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'LayerNorm_0', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'LayerNorm_1', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_0', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_1', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_1', 'scale'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_2', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_3', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_3', 'scale'), ('model', 'encoder', 'embed_positions', 'embedding'), ('model', 'encoder', 'embed_tokens', 'embedding'), ('model', 'encoder', 'final_ln', 'bias'), ('model', 'encoder', 'layernorm_embedding', 'bias'), ('model', 'encoder', 'layernorm_embedding', 'scale'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'k_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'out_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'q_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'v_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'Dense_0', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'Dense_1', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'Dense_2', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'LayerNorm_0', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'LayerNorm_1', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'LayerNorm_0', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'LayerNorm_1', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'LayerNorm_1', 'scale')]
You should probably UPCAST the model weights to float32 if this was not intended. See [`~FlaxPreTrainedModel.to_fp32`] for further information on how to do this.

OS: macOS 12.4 21F79 x86_64
Host: MacBookPro12,1
Kernel: 21.5.0
Shell: zsh 5.8.1
CPU: Intel i7-5557U (4) @ 3.10GHz
GPU: Intel Iris Graphics 6100
Memory: 4535MiB / 16384MiB

~/Desktop/dalle-playground/backend$ python3 app.py 8000
--> Starting DALL-E Server. This might take up to two minutes.
2022-06-12 13:16:41.035513: I external/org_tensorflow/tensorflow/core/tpu/tpu_initializer_helper.cc:259] Libtpu path is: libtpu.so
2022-06-12 13:16:51.303810: I external/org_tensorflow/tensorflow/compiler/xla/service/service.cc:174] XLA service 0x90bd0b0 initialized for platform Interpreter (this does not guarantee that XLA will be used). Devices:
2022-06-12 13:16:51.303839: I external/org_tensorflow/tensorflow/compiler/xla/service/service.cc:182]   StreamExecutor device (0): Interpreter, <undefined>
2022-06-12 13:16:51.305845: I external/org_tensorflow/tensorflow/compiler/xla/pjrt/tfrt_cpu_pjrt_client.cc:176] TfrtCpuClient created.
2022-06-12 13:16:51.306543: I external/org_tensorflow/tensorflow/stream_executor/tpu/tpu_platform_interface.cc:74] No TPU platform found.
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorflow.python.client import device_lib
>>> print(device_lib.list_local_devices())
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 9644359123142212818
xla_global_id: -1
, name: "/device:GPU:0"
device_type: "GPU"
memory_limit: 10567155712
locality {
  bus_id: 1
  links {
  }
}
incarnation: 80875608830553824
physical_device_desc: "device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:01:00.0, compute capability: 8.6"
xla_global_id: 416903419
]
>>>

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

ERROR: Cannot install flax because these package versions have conflicting dependencies.

The conflict is caused by:
    optax 0.1.2 depends on jaxlib>=0.1.37
    optax 0.1.1 depends on jaxlib>=0.1.37
    optax 0.1.0 depends on jaxlib>=0.1.37
    optax 0.0.91 depends on jaxlib>=0.1.37
    optax 0.0.9 depends on jaxlib>=0.1.37
    optax 0.0.8 depends on jaxlib>=0.1.37
    optax 0.0.6 depends on jaxlib>=0.1.37
    optax 0.0.5 depends on jaxlib>=0.1.37
    optax 0.0.3 depends on jaxlib>=0.1.37
    optax 0.0.2 depends on jaxlib>=0.1.37
    optax 0.0.1 depends on jaxlib>=0.1.37

Traceback (most recent call last):

  File "app.py", line 47, in <module>

    dalle_model = DalleModel(dalle_version)

  File "/app/dalle_model.py", line 62, in __init__

    self.model, params = DalleBart.from_pretrained(

  File "/usr/local/lib/python3.8/dist-packages/dalle_mini/model/utils.py", line 25, in from_pretrained

    return super(PretrainedFromWandbMixin, cls).from_pretrained(

  File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_flax_utils.py", line 603, in from_pretrained

    state = from_bytes(cls, state_f.read())

  File "/usr/local/lib/python3.8/dist-packages/flax/serialization.py", line 368, in from_bytes

    state_dict = msgpack_restore(encoded_bytes)

  File "/usr/local/lib/python3.8/dist-packages/flax/serialization.py", line 350, in msgpack_restore

    state_dict = msgpack.unpackb(

  File "msgpack/_unpacker.pyx", line 205, in msgpack._cmsgpack.unpackb

ValueError: Unpack failed: incomplete input

> D:\dalle\dalle-playground\backend>python app.py --port 8080 --model_version mega --save_to_disk true

--> Starting DALL-E Server. This might take up to two minutes.
wandb: Currently logged in as: anony-moose-315442. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.12.18
wandb: Run data is saved locally in D:\dalle\dalle-playground\backend\wandb\run-20220615_222754-1tbnwomv
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run lilac-field-9
wandb:  View project at https://wandb.ai/anony-moose-315442/dalle-playground-backend?apiKey=cb614629a73707160e3c82c72823ef954bcad91c
wandb:  View run at https://wandb.ai/anony-moose-315442/dalle-playground-backend/runs/1tbnwomv?apiKey=cb614629a73707160e3c82c72823ef954bcad91c
wandb: WARNING Do NOT share these links with anyone. They can be used to claim your runs.
wandb: Downloading large artifact mega-1-fp16:latest, 4938.53MB. 7 files... Done. 0:0:5.4
Some of the weights of DalleBart were initialized in float16 precision from the model checkpoint at C:\Users\blood\AppData\Local\Temp\tmp_nwoes5j:
[('lm_head', 'kernel'), ('model', 'decoder', 'embed_positions', 'embedding'), ('model', 'decoder', 'embed_tokens', 'embedding'), ('model', 'decoder', 'final_ln', 'bias'), ('model', 'decoder', 'layernorm_embedding', 'bias'), ('model', 'decoder', 'layernorm_embedding', 'scale'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'k_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'out_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'q_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_0', 'v_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'k_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'out_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'q_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'FlaxBartAttention_1', 'v_proj', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'Dense_0', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'Dense_1', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'Dense_2', 'kernel'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'LayerNorm_0', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'GLU_0', 'LayerNorm_1', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_0', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_1', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_1', 'scale'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_2', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_3', 'bias'), ('model', 'decoder', 'layers', 'FlaxBartDecoderLayers', 'LayerNorm_3', 'scale'), ('model', 'encoder', 'embed_positions', 'embedding'), ('model', 'encoder', 'embed_tokens', 'embedding'), ('model', 'encoder', 'final_ln', 'bias'), ('model', 'encoder', 'layernorm_embedding', 'bias'), ('model', 'encoder', 'layernorm_embedding', 'scale'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'k_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'out_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'q_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'FlaxBartAttention_0', 'v_proj', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'Dense_0', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'Dense_1', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'Dense_2', 'kernel'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'LayerNorm_0', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'GLU_0', 'LayerNorm_1', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'LayerNorm_0', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'LayerNorm_1', 'bias'), ('model', 'encoder', 'layers', 'FlaxBartEncoderLayers', 'LayerNorm_1', 'scale')]
You should probably UPCAST the model weights to float32 if this was not intended. See [`~FlaxPreTrainedModel.to_fp32`] for further information on how to do this.
wandb: Downloading large artifact mega-1-fp16:latest, 4938.53MB. 7 files... Done. 0:0:2.7
Traceback (most recent call last):
  File "D:\dalle\dalle-playground\backend\app.py", line 61, in <module>
    dalle_model.generate_images("warm-up", 1)
  File "D:\dalle\dalle-playground\backend\dalle_model.py", line 92, in generate_images
    key, subkey = jax.random.split(key)
  File "C:\Python\Python39\site-packages\jax\_src\random.py", line 194, in split
    return _return_prng_keys(wrapped, _split(key, num))
  File "C:\Python\Python39\site-packages\jax\_src\random.py", line 180, in _split
    return key._split(num)
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 203, in _split
    return PRNGKeyArray(self.impl, self.impl.split(self._keys, num))
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 474, in threefry_split
    return _threefry_split(key, int(num))  # type: ignore
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 479, in _threefry_split
    return lax.reshape(threefry_2x32(key, counts), (num, 2))
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 467, in threefry_2x32
    x = threefry2x32_p.bind(key1, key2, x[0], x[1])
jax._src.source_info_util.JaxStackTraceBeforeTransformation: ValueError: DenseElementsAttr could not be constructed from the given buffer. This may mean that the Python buffer layout does not match that MLIR expected layout and is a bug.

The preceding stack trace is the source of the JAX operation that, once transformed by JAX, triggered the following exception.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\dalle\dalle-playground\backend\app.py", line 61, in <module>
    dalle_model.generate_images("warm-up", 1)
  File "D:\dalle\dalle-playground\backend\dalle_model.py", line 92, in generate_images
    key, subkey = jax.random.split(key)
  File "C:\Python\Python39\site-packages\jax\_src\random.py", line 194, in split
    return _return_prng_keys(wrapped, _split(key, num))
  File "C:\Python\Python39\site-packages\jax\_src\random.py", line 180, in _split
    return key._split(num)
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 203, in _split
    return PRNGKeyArray(self.impl, self.impl.split(self._keys, num))
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 474, in threefry_split
    return _threefry_split(key, int(num))  # type: ignore
  File "C:\Python\Python39\site-packages\jax\_src\traceback_util.py", line 162, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "C:\Python\Python39\site-packages\jax\_src\api.py", line 473, in cache_miss
    out_flat = xla.xla_call(
  File "C:\Python\Python39\site-packages\jax\core.py", line 1765, in bind
    return call_bind(self, fun, *args, **params)
  File "C:\Python\Python39\site-packages\jax\core.py", line 1781, in call_bind
    outs = top_trace.process_call(primitive, fun_, tracers, params)
  File "C:\Python\Python39\site-packages\jax\core.py", line 678, in process_call
    return primitive.impl(f, *tracers, **params)
  File "C:\Python\Python39\site-packages\jax\_src\dispatch.py", line 182, in _xla_call_impl
    compiled_fun = _xla_callable(fun, device, backend, name, donated_invars,
  File "C:\Python\Python39\site-packages\jax\linear_util.py", line 285, in memoized_fun
    ans = call(fun, *args)
  File "C:\Python\Python39\site-packages\jax\_src\dispatch.py", line 230, in _xla_callable_uncached
    return lower_xla_callable(fun, device, backend, name, donated_invars, False,
  File "C:\Python\Python39\site-packages\jax\_src\profiler.py", line 206, in wrapper
    return func(*args, **kwargs)
  File "C:\Python\Python39\site-packages\jax\_src\dispatch.py", line 340, in lower_xla_callable
    module, keepalive = mlir.lower_jaxpr_to_module(
  File "C:\Python\Python39\site-packages\jax\interpreters\mlir.py", line 556, in lower_jaxpr_to_module
    lower_jaxpr_to_fun(
  File "C:\Python\Python39\site-packages\jax\interpreters\mlir.py", line 810, in lower_jaxpr_to_fun
    out_vals, tokens_out = jaxpr_subcomp(ctx.replace(name_stack=callee_name_stack),
  File "C:\Python\Python39\site-packages\jax\interpreters\mlir.py", line 929, in jaxpr_subcomp
    ans = rule(rule_ctx, *map(_unwrap_singleton_ir_values, in_nodes),
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 400, in _threefry2x32_gpu_lowering
    return threefry2x32_lowering(
  File "C:\Python\Python39\site-packages\jaxlib\cuda_prng.py", line 76, in threefry2x32_lowering
    layout = ir.DenseIntElementsAttr.get(np.arange(ndims - 1, -1, -1),
jax._src.traceback_util.UnfilteredStackTrace: ValueError: DenseElementsAttr could not be constructed from the given buffer. This may mean that the Python buffer layout does not match that MLIR expected layout and is a bug.

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\dalle\dalle-playground\backend\app.py", line 61, in <module>
    dalle_model.generate_images("warm-up", 1)
  File "D:\dalle\dalle-playground\backend\dalle_model.py", line 92, in generate_images
    key, subkey = jax.random.split(key)
  File "C:\Python\Python39\site-packages\jax\_src\random.py", line 194, in split
    return _return_prng_keys(wrapped, _split(key, num))
  File "C:\Python\Python39\site-packages\jax\_src\random.py", line 180, in _split
    return key._split(num)
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 203, in _split
    return PRNGKeyArray(self.impl, self.impl.split(self._keys, num))
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 474, in threefry_split
    return _threefry_split(key, int(num))  # type: ignore
  File "C:\Python\Python39\site-packages\jax\_src\prng.py", line 400, in _threefry2x32_gpu_lowering
    return threefry2x32_lowering(
  File "C:\Python\Python39\site-packages\jaxlib\cuda_prng.py", line 76, in threefry2x32_lowering
    layout = ir.DenseIntElementsAttr.get(np.arange(ndims - 1, -1, -1),
ValueError: DenseElementsAttr could not be constructed from the given buffer. This may mean that the Python buffer layout does not match that MLIR expected layout and is a bug.
wandb: Waiting for W&B process to finish... (failed 1). Press Ctrl-C to abort syncing.
wandb:
wandb: Synced lilac-field-9: https://wandb.ai/anony-moose-315442/dalle-playground-backend/runs/1tbnwomv?apiKey=cb614629a73707160e3c82c72823ef954bcad91c
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: .\wandb\run-20220615_222754-1tbnwomv\logs

dalle-backend      | Original error: INTERNAL: All algorithms tried for %cudnn-conv = (f32[1,256,256,128]{2,1,3,0}, u8[0]{0}) custom-call(f32[1,256,256,3]{2,1,3,0} %copy, f32[3,3,3,128]{1,0,2,3} %copy.1), window={size=3x3 pad=1_1x1_1}, dim_labels=b01f_01io->b01f, custom_call_target="__cudnn$convForward", metadata={op_name="jit(conv_general_dilated)/jit(main)/conv_general_dilated[window_strides=(1, 1) padding=((1, 1), (1, 1)) lhs_dilation=(1, 1) rhs_dilation=(1, 1) dimension_numbers=ConvDimensionNumbers(lhs_spec=(0, 3, 1, 2), rhs_spec=(3, 2, 0, 1), out_spec=(0, 3, 1, 2)) feature_group_count=1 batch_group_count=1 lhs_shape=(1, 256, 256, 3) rhs_shape=(3, 3, 3, 128) precision=None preferred_element_type=None]" source_file="/usr/local/lib/python3.8/dist-packages/flax/linen/linear.py" source_line=406}, backend_config="{\"conv_result_scale\":1,\"activation_mode\":\"0\",\"side_input_scale\":0}" failed. Falling back to default algorithm.  Per-algorithm errors:
dalle-backend      |   Profiling failure on cuDNN engine eng30{k4=1,k5=0,k6=0,k7=0,k2=2}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng34{k4=1,k5=1,k6=0,k7=0,k2=0,k19=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng30{k2=1,k4=2,k5=1,k6=0,k7=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng34{k2=2,k19=0,k4=1,k5=0,k6=0,k7=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng34{k2=0,k19=0,k4=2,k5=1,k6=0,k7=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng34{k2=2,k19=0,k4=2,k5=0,k6=0,k7=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng30{k4=2,k5=0,k6=0,k7=0,k2=2}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng30{k2=0,k4=2,k5=1,k6=0,k7=0}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng30{k4=3,k5=1,k6=0,k7=0,k2=1}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng42{k4=1,k5=0,k6=0,k7=0,k2=2}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng0{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng1{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng2{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng4{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng6{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng12{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng28{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng30{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng31{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng34{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'
dalle-backend      |   Profiling failure on cuDNN engine eng42{}: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
dalle-backend      | in external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_dnn.cc(4369): 'status'