How long does it take to run laion_cloudwriter.py and precompute_latents.py for aesthetic-4.5 about diffusion HOT 1 CLOSED

Are you downloading the data and running laion_cloudwriter.py at the same time? It does take a lot of time to run, but this will depend on the number of CPUs, network bandwidth, disk I/O, etc. We need to dig up the exact time it took us, but it will vary greatly based on the machine specs you are using.

Since the parquets can be processed in parallel, we divided the parquets into 4 subsets and processed each subset on a different machine. If you have access to more than one machine, I highly recommend this strategy. Our Streaming library can seamlessly use multiple streaming datasets, so no need to combine separately processed streaming datasets.

from diffusion.