nperraud / download-celeba-hq Goto Github PK

Script was working until this error:

Traceback (most recent call last):
File "/workspace/make_HQ_images.py", line 164, in pool.map(do_the_work, list(range(expected_dat)))
File "/opt/conda/lib/python3.5/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/conda/lib/python3.5/multiprocessing/pool.py", line 644, in get raise self._value
OSError: 3145728 requested and 1671040 written

It successfully loaded Celeba and Celeba-HQ deltas, then it crashed because "image_list.txt" doesn't exist

Loading CelebA from ./celebA/Img/img_celeba Loading CelebA-HQ deltas from ./celebA-HQ Traceback (most recent call last): File "/workspace/make_HQ_images.py", line 45, in with open(os.path.join('image_list.txt'), 'rt') as file: FileNotFoundError: [Errno 2] No such file or directory: 'image_list.txt'

The above exception was raised in running download_celebA_HQ.py

Command:

python download_celebA_HQ.py ./img_align_celeba

Output log:

Deal with file: deltas00000.zip
[*] ./img_align_celeba/deltas00000.zip already exists
Traceback (most recent call last):
File "download_celebA_HQ.py", line 93, in
with zipfile.ZipFile(save_path) as zf:
File "/u/jliang/anaconda3/envs/tf_gpu/lib/python3.6/zipfile.py", line 1108, in init
self._RealGetContents()
File "/u/jliang/anaconda3/envs/tf_gpu/lib/python3.6/zipfile.py", line 1175, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

Given that this issue hasn't seemed to occur previously, is this an issue associated with the file itself? Or is it due to my setup?

Hello, and thanks for your work! Could you tell me what the total size of the dataset is? I need to know on what kind of hard drive I should run the program. Maybe it would be a useful information to display somewhere on the README.

Hello, I am currently unable to use dropbox links in download_celebA.py file, are there any alternatives for those or how else I can get the dataset?

Thank you.

Dropbox is limited in term of download. Replace it with google drive in the script download_celebA.py.

Some of it has already been done.

sum([1 for file in os.listdir(img_dir) if file[-4:] == '.jpg'])

could be simplified in a bit efficient way

from glob import glob
len(glob(os.path.join(img_dir,'*.jpg')))

Hello and thanks for sharing your code! 😄

I've been browsing in the generated images and most of them are of high quality.
However, I found that images 70 and 2815 are not that great.

Image 70 - half of the face is cut

Image 2815 - there is an eye in the mouth

Is it the same for you? Is it the same from the original repo?(NIVIDIA)
Did you find other images with small flaws?

I've got the following exception during the downloading

if os.name != 'posix':
raise NotImplementedError('Only Linux and Mac OS X support .7z '

It downloads and unpacks the first 200K images easily but when it comes to archives like
img_celeba.7z.001 it raises the exception.

What should I do with Windows 10 then? 7z is already installed.

I get stuck at the same error whether I try the Docker approach or follow the instructions myself.

PackagesNotFoundError: The following packages are not available from current channels:

  - pillow==3.1.1
  - jpeg=8d 

Am I doing something wrong? Or does the script need to be udpated?

Hi,

thank you for this code !
I have an error in the make_celeba_hQ.py

Traceback (most recent call last):
File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "make_HQ_images.py", line 157, in do_the_work
img = process_func(img_num)
File "make_HQ_images.py", line 115, in process_func
img = np.pad(np.float32(img), ((pad[1], pad[3]), (pad[0], pad[2]), (0, 0)), 'reflect')
TypeError: float() argument must be a string or a number, not 'Image'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "make_HQ_images.py", line 167, in
pool.map(do_the_work, list(range(expected_dat)))
File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/multiprocessing/pool.py", line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.5.4/lib/python3.5/multiprocessing/pool.py", line 644, in get
raise self._value
TypeError: float() argument must be a string or a number, not 'Image'

Do you know where it can come from??

Thanks in advance

I want the celeba hq dataset for some task in which I just want 2000 images.
So I have downloaded the celeba dataset, but please tell me which delta should I download ( Because I have slow internet connection ) to convert it to HQ, and what will be the process

I was trying to run this with docker, it didn't work:

After clone:
$ docker build -t celeba_hq . && docker run -it -v $(pwd):/data celeba_hq
...
PackagesNotFoundError: The following packages are not available from current channels:

• pillow==3.1.1
• jpeg=8d

Current channels:

• https://repo.anaconda.com/pkgs/main/linux-64
• https://repo.anaconda.com/pkgs/main/noarch
• https://repo.anaconda.com/pkgs/r/linux-64
• https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.
...

Hi nperraud,
Thanks for your code, I'm running the make_HQ_images.py without any problem, but the result is not a single subject 1024by1024 image but a 3by3 replication of the subject(and the image is in grey scale).
Is this normal or I missed some steps? Here is an example:

conda install jpeg=8d tqdm requests pillow==3.1.1 urllib3 numpy cryptography scipy
results in dependency conflict.
conda install pillow==3.1.1 alone causes anaconda to demand switching to python2

doing
conda install -c conda-forge tqdm (first anaconda result after google 'conda tqdm') and same for pillow (without setting version) numpy, criptography, scipy,

and also comment out the the warnings works

I am using ubuntu 19
I ran the first two python files, than found that i didn't setup conda environment correctly and was installing onto the default env, so removed anaconda and reinstalled, than roughly followed the above procedure and it worked.

Command:
cd download-celebA-HQ/ && python download_celebA.py ./

Output:

Downloading img_align_celeba.zip to ./celebA/img_align_celeba.zip
./celebA/img_align_celeba.zip: 1.00B [00:00, 847B/s]
Done!
Check SHA1 ./celebA/img_align_celeba.zip
Traceback (most recent call last):
File "download_celebA.py", line 219, in
download_celabA(dataset_dir)
File "download_celebA.py", line 183, in download_celabA
filepaths = download_and_check(_ALIGNED_IMGS_DRIVE, dataset_dir)
File "download_celebA.py", line 106, in download_and_check
raise RuntimeError('Checksum mismatch for %s.' % save_path)
RuntimeError: Checksum mismatch for ./celebA/img_align_celeba.zip.

To get this to work on Windows 10 I had to do the following:

Create a new environement

conda activate celebaHQ

Install the packages

conda  install -c anaconda jpeg=8d tqdm requests pillow==3.1.1 urllib3 numpy cryptography scipy

I had to add a if __name__ == '__main__': line just before num_workers.

if __name__ == '__main__':

    num_workers = mp.cpu_count() - 1
    print('Starting a pool with {} workers'.format(num_workers))
    with mp.Pool(processes=num_workers) as pool:
        pool.map(do_the_work, list(range(expected_dat)))
    if len(glob.glob(os.path.join(delta_dir, '*.npy'))) != 30000:
        raise ValueError('Expected to find {} npy files\n Something went wrong!'.format(30000))
# Remove the dat files
    for filepath in glob.glob(os.path.join(delta_dir, '*.dat')):
        os.remove(filepath)
    print('All done! Congratulations!')

As in the title, can't get the data thru codes

urllib.error.HTTPError: HTTP Error 429: Too Many Requests

Looking forward to ur response!