Duplicate Images about bing_image_downloader HOT 8 OPEN

My first attempt to filter out duplicates would be to subtract two possible duplicated images and check if the difference is close to zero.

from bing_image_downloader.

I'm getting the same. Downloaded 10000 pictures and 9789 of them were duplicates. Is this a nature of Bing image search, or particular to this downloader?

from bing_image_downloader.

When I scrape 100 photos, after the first 85 to 90 images, they start to repeat, and the rest are all duplicates.
When I scrape 500 photos, 370 of them are duplicates :(
Other than this it works great, so I really hope this issue can get fixed.

from bing_image_downloader.

Ya I also faced same issue it was due to how its programed i.e there is no next page in bing so instead first=pagecounter -> do first len of total url visited
also added ignore duplicates if same url is already visited
i will also pull the code or you can visit https://github.com/AbhiDhariwal/bing_image_downloader

from bing_image_downloader.

I successfully avoided duplicated images with the following code. But now it will search forever. So yeah,
maybe we need a next button for more images.

`
self.duplicates = set()

def save_image(self, link, file_path):
    request = urllib.request.Request(link, None, self.headers)
    image = urllib.request.urlopen(request, timeout=self.timeout).read()
    
    if not imghdr.what(None, image) or image in self.duplicates:
        print('[Error]Invalid image, not saving {}\n'.format(link))
        raise
    else:
        self.duplicates.add(image)

    with open(file_path, 'wb') as f:
        f.write(image)

`

from bing_image_downloader.

Remove duplicates PR#20

from bing_image_downloader.

Bumping this as an issue. The fix above looks like it works and would be great if merged. Thanks!

from bing_image_downloader.

Please close this issue

from bing_image_downloader.