visual-layer / fastdup Goto Github PK

OpenCV(4.5.5) 👎 error: (-5:Bad argument) in function 'imencode'

Overload resolution failed:

• img data type = 17 is not supported
• Expected Ptrcv::UMat for argument 'img'

fastdup/image.py in imageformatter at line 112

After installing fastdup with

python3 -m pip install fastdup

,

import fastdup

fails on

ModuleNotFoundError: No module named 'packaging'
ModuleNotFoundError: No module named 'requests'

pip-installing those missing requirements solves the issue.

The docker image has been built from this repo's Dockerfile.

import fastdup returns an error:

root@a5a6305aaaf2:/opt# python3.8 
Python 3.8.10 (default, Mar 15 2022, 12:22:08) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastdup
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/fastdup/__init__.py", line 5, in <module>
    import cv2
  File "/usr/local/lib/python3.8/dist-packages/cv2/__init__.py", line 181, in <module>
    bootstrap()
  File "/usr/local/lib/python3.8/dist-packages/cv2/__init__.py", line 153, in bootstrap
    native_module = importlib.import_module("cv2")
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

• if max_width is not None and spriteimage.width > max_width:

•    factor = max_width / spriteimage.width
  

•    spriteimage = spriteimage.resize((int(spriteimage.width * factor), int(spriteimage.height * factor)))
  

  if isinstance(img_path[0], str):
  if alternative_filename is not None:

We have been running fastdup inside of a python:3.8-buster docker image for several months now. As of this morning with the latest v0.165 release we have started to see a library error when trying to import fastdup that we weren't seeing previously. We've now pinned to v0.163 to resolve this but still wanted to bring it to your attention.

Error seen:

venv/lib/python3.8/site-packages/fastdup/__init__.py:70: in <module>
    dll = CDLL(so_file)
/usr/local/lib/python3.8/ctypes/__init__.py:373: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: libunity_shared.so: cannot open shared object file: No such file or directory

includes: duplicates_gallery, similarity_gallery, ...

os.path.basename(x)

Hi,
Thank you for looking into my issue. I was trying to run the following code:
import fastdup
fastdup.run_kmeans(input_dir='./data', work_dir = ./out')
then it shows the following error

Canceled future for execute_request message before replies were done
The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click here for more info. View Jupyter log for further details.

The log is as below:

Found total 38 images to run on

error 13:19:39.756: Disposing session as kernel process died ExitCode: undefined, Reason:
/anaconda3/envs/scan/lib/python3.7/site-packages/traitlets/traitlets.py:2415: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
FutureWarning,
/anaconda3/envs/scan/lib/python3.7/site-packages/traitlets/traitlets.py:2369: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use '298cec57-193d-4bb0-b39b-bad7277157b3' instead of 'b"298cec57-193d-4bb0-b39b-bad7277157b3"'.
FutureWarning,

error 13:19:39.757: Raw kernel process exited code: undefined
error 13:19:39.759: Error in waiting for cell to complete [Error: Canceled future for execute_request message before replies were done
at t.KernelShellFutureHandler.dispose (/.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:1376085)
at /.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:1395137
at Map.forEach ()
at y._clearKernelState (/.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:1395122)
at y.dispose (/.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:1388604)
at /.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:578260
at t.swallowExceptions (/.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:1001797)
at dispose (/.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:578238)
at t.RawSession.dispose (/.vscode-server/extensions/ms-toolsai.jupyter-2022.5.1001601848/out/extension.node.js:2:583202)
at runMicrotasks ()
at processTicksAndRejections (node:internal/process/task_queues:96:5)]
warn 13:19:39.759: Cell completed with errors {
message: 'Canceled future for execute_request message before replies were done'
}
info 13:19:39.760: Cancel all remaining cells true || Error || undefined
info 13:19:39.760: Cancel pending cells
info 13:19:39.761: Cell 4 executed with state Error

Hi !
I am sort of newbie to python...

Installed fastdup successfully on ubuntu 20.04 by creating a separate environment with python 3.8
Activated the environment ....copied the code as mentioned in your website:
import fastdup fastdup.run(input_dir="/path/to/your/folder", work_dir='out') #main running function fastdup.create_duplicates_gallery('out/similarity.csv', save_path='.') #create a visual gallery of found duplicates fastdup.create_outliers_gallery('out/outliers.csv', save_path='.') #create a visual gallery of anomalies fastdup.create_components_gallery('out', save_path='.') #create visualiaiton of connected components

in file fast.py but when I execute python3 fastdup.py I get error:

Traceback (most recent call last):
File "fast.py", line 2, in
fastdup.run(input_dir="/home/rana/Fast-Duplicate/pics", work_dir='/home/rana/Fast-Duplicate/out') #main running function
AttributeError: module 'fastdup' has no attribute 'run'

FYI:
Fast-Duplicate is the folder inside which python environment was created and fastdup was installed successfully.
i have copied the folder named pics inside Fast-Duplicate....& created out folder there.
pics is the folder containing many images, all inside categorized folders.....say it has folders food, bicycle, bus etc containing many respective images.

I executed the following command

fastdup.run(input_dir=train_imgs_list,
            work_dir=output_folder,
            test_dir=test_imgs_list,)

Despite I set work_dir output folder, the code saves files as .sentry-native/, files.txt and testfiles.txt in the root path

First I want to say its a great tool and super fast! I used to analyze 19,000 images. The install was fast and the analysis super fast.
Entering a wrong path (in the input dir) causes a complete kernel crash. I noticed my typo and fixed it. It would be good if the library would return a clearer exception specifying the path doesn't exist. I haven't tested inputting a none existing dir in the output one.

I get the below exception when I try to run the following code (the example on the README.md):

import fastdup
fastdup.run(input_dir="/path/to/your/folder", work_dir='out', nearest_neighbors_k=5, turi_param='ccthreshold=0.96')    #main running function.
fastdup.create_duplicates_gallery('out/similarity.csv', save_path='.')     #create a visual gallery of found duplicates
fastdup.create_outliers_gallery('out/outliers.csv',   save_path='.')       #create a visual gallery of anomalies
fastdup.create_components_gallery('out', save_path='.')                    #create visualiaiton of connected components
fastdup.create_stats_gallery('out', save_path='.', metric='blur')          #create visualization of images stastics (for example blur)
fastdup.create_similarity_gallery('out', save_path='.',get_label_func=lambda x: x.split('/')[-2])     #create visualization of top_k similar images assuming data have labels which are in the folder name
fastdup.create_aspect_ratio_gallery('out', save_path='.')                  #create aspect ratio gallery

Exception:

qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/usr/miniconda3/envs/fastdup/lib/python3.9/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Qt is installed. Also tried reinstalling it but it didn't seem to work.

Can you add support for The Resnet50 backbone used in fast mask RCNN on the CityScape dataset or the BDD100K dataset.

In the Detailed Run section, it says path to ONNX file should not be used so it seems that its not yet supported.
I am not talking about full ability to add weights, but maybe some presets, of common weight backbones in different tasks (as a simpler interface to actually completely setting an onnx file) . So the API can be

fastdup.run(input_dir="./tagged_sliced", work_dir='out', backbone = 'resnet.cityscape') 

It would be nice if you can add a progress bar.

in the gallery

Hi,

I am unable to install the package on windows for some reason. I could install the package in Linux os.

I checked if I can install other packages and it works, is it some registry issue?

please help

Hi, can I check what is used as your feature extractor?

I run this tool over 999_999 images, and it found these duplicates:

visually they are not duplicate. something is wrong in the indexes of filename to index?

fastdup.create_stats_gallery(os.path.join(results_dir, 'atrain_stats.csv'), save_path=results_dir, metric='size')
fails on missing matplotlib. Is it by design?

fastdup version: 0.190
(Info about fastdup installation method, python, pip and machine versions can be found in #47)

Thanks!

P.S. BTW, Dockerfile lists matplotlib explicitly.

Hi, I'm trying to install your package with Poetry (package manager) on WSL (ubuntu) with Python3.8 but it fails :(.

I tried the regular way: poetry add fastdup

  • Installing fastdup (0.180): Failed

  RuntimeError

  Unable to find installation candidates for fastdup (0.180)

This is what I get when I tried to add from git directly

poetry add git+https://github.com/visual-layer/fastdup.git#main

Please let me know if there is something I can do.
Thanks.

Unable to determine package info for path: /home/kikohs/.cache/pypoetry/virtualenvs/rts-DIiCth0j-py3.8/src/fastdup

Command ['/tmp/tmpzgduc58e/.venv/bin/python', '-'] errored with the following return code 1, and output: 
Traceback (most recent call last):
  File "<stdin>", line 9, in <module>
  File "/tmp/tmpzgduc58e/.venv/lib/python3.8/site-packages/build/__init__.py", line 208, in __init__
    _validate_source_directory(srcdir)
  File "/tmp/tmpzgduc58e/.venv/lib/python3.8/site-packages/build/__init__.py", line 109, in _validate_source_directory
    raise BuildException(f'Source {srcdir} does not appear to be a Python project: no pyproject.toml or setup.py')
build.BuildException: Source /home/kikohs/.cache/pypoetry/virtualenvs/rts-DIiCth0j-py3.8/src/fastdup does not appear to be a Python project: no pyproject.toml or setup.py
input was : import build
import build.env
import pep517

source = '/home/kikohs/.cache/pypoetry/virtualenvs/rts-DIiCth0j-py3.8/src/fastdup'
dest = '/tmp/tmpzgduc58e/dist'

with build.env.IsolatedEnvBuilder() as env:
    builder = build.ProjectBuilder(
        srcdir=source,
        scripts_dir=env.scripts_dir,
        python_executable=env.executable,
        runner=pep517.quiet_subprocess_runner,
    )
    env.install(builder.build_system_requires)
    env.install(builder.get_requires_for_build('wheel'))
    builder.metadata_path(dest)

No fallback setup.py file was found to generate egg_info.

Running in docker again.

Your system is missing some depdencies, please pip install matplotlib matplotlib-inline torchvision
Traceback (most recent call last):
  File "./run.py", line 5, in <module>                                                                                                                        
    fastdup.create_components_gallery('out', save_path='.')                    #create visualiaiton of connected components
  File "/usr/local/lib/python3.8/dist-packages/fastdup/__init__.py", line 733, in create_components_gallery
    for i,(j,row) in tqdm(enumerate(subdf.iterrows()), total=num_images):      
AttributeError: 'int' object has no attribute 'iterrows'   

torch is quite a heavy dependency.

Collecting torch==1.12.0
  Downloading torch-1.12.0-cp38-cp38-manylinux1_x86_64.whl (776.3 MB)

I installed them as suggested inside docker and it completed the run.

possible cause is that the images were saved as jpg but the html file is trying to load png files

All the links that should bring to the full documentation appear to be broken.

In particular, i'd like to have more informations on how the feature vectors are created.

Anyway, great work. Thanks a lot for sharing it.

Is there any plans fastdup will support CentOS?

python3.8 -m pip install fastdup

results in

ERROR: Could not find a version that satisfies the requirement fastdup (from versions: none) ERROR: No matching distribution found for fastdup

• Works only if all images are in a flat structure (all images in one directory).
• This is not a feasible layout , especially for millions of images.

img_pil = cv2.cvtColor(img_pil, cv2.COLOR_BGR2RGB)

This unclicensed software version allows up to 1000000 images. You have request to run on 2749565 images.

Hi, I tried passing s3 folder location while running fastdup.run(). It is throwing this error "Failed to find input dir"

Thank you for the amazing tool! I noticed the section "Find wrong labels" in the README, but there was no corresponding function in the RUN.md. How should I find wrong labels, and is it possible for the tool to suggest the correct label?

Happens when image dimensions are ~8px and smaller

Installation fails on M1 pro mac when trying to install the newest version using M1 and conda.

pip install fastdup
Collecting fastdup
  Using cached fastdup-0.177-cp39-cp39-macosx_10_14_x86_64.whl (80.0 MB)
Requirement already satisfied: pillow in /Users/amirm/opt/anaconda3/lib/python3.9/site-packages (from fastdup) (9.2.0)
Requirement already satisfied: certifi in /Users/amirm/opt/anaconda3/lib/python3.9/site-packages (from fastdup) (2022.9.24)
Requirement already satisfied: pyyaml in /Users/amirm/opt/anaconda3/lib/python3.9/site-packages (from fastdup) (6.0)
  Using cached fastdup-0.174-cp39-cp39-macosx_10_14_x86_64.whl (80.0 MB)
  Using cached fastdup-0.173-cp39-cp39-macosx_10_14_x86_64.whl (80.0 MB)
  Downloading fastdup-0.172-cp39-cp39-macosx_10_14_x86_64.whl (80.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.0/80.0 MB 3.4 MB/s eta 0:00:00
  Downloading fastdup-0.171-cp39-cp39-macosx_10_14_x86_64.whl (79.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.9/79.9 MB 3.7 MB/s eta 0:00:00
  Downloading fastdup-0.168-cp39-cp39-macosx_10_14_x86_64.whl (79.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 MB 4.3 MB/s eta 0:00:00
  Downloading fastdup-0.166-cp39-cp39-macosx_10_14_x86_64.whl (79.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 MB 3.6 MB/s eta 0:00:00
  Downloading fastdup-0.162-cp39-cp39-macosx_10_14_x86_64.whl (79.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 MB 3.5 MB/s eta 0:00:00
  Downloading fastdup-0.161-cp39-cp39-macosx_10_14_x86_64.whl (79.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 MB 4.1 MB/s eta 0:00:00
  Downloading fastdup-0.160-cp39-cp39-macosx_10_14_x86_64.whl (79.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 MB 3.7 MB/s eta 0:00:00
  Downloading fastdup-0.158-cp39-cp39-macosx_10_14_x86_64.whl (79.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 MB 4.1 MB/s eta 0:00:00
  Downloading fastdup-0.154-cp39-cp39-macosx_10_14_x86_64.whl (75.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.8/75.8 MB 3.2 MB/s eta 0:00:00
ERROR: Cannot install fastdup==0.154, fastdup==0.158, fastdup==0.160, fastdup==0.161, fastdup==0.162, fastdup==0.166, fastdup==0.168, fastdup==0.171, fastdup==0.172, fastdup==0.173, fastdup==0.174 and fastdup==0.177 because these package versions have conflicting dependencies.

The conflict is caused by:
    fastdup 0.177 depends on opencv-python==4.1.2.30
    fastdup 0.174 depends on opencv-python==4.1.2.30
    fastdup 0.173 depends on opencv-python==4.1.2.30
    fastdup 0.172 depends on opencv-python==4.1.2.30
    fastdup 0.171 depends on opencv-python==4.1.2.30
    fastdup 0.168 depends on opencv-python==4.1.2.30
    fastdup 0.166 depends on opencv-python==4.1.2.30
    fastdup 0.162 depends on opencv-python==4.1.2.30
    fastdup 0.161 depends on opencv-python==4.1.2.30
    fastdup 0.160 depends on opencv-python==4.1.2.30
    fastdup 0.158 depends on opencv-python==4.1.2.30
    fastdup 0.154 depends on opencv-python==4.1.2.30

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Running on GCP, ubuntu 20 and getting the following warning after running "import fastdup":

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (5.1.0) doesn't match a supported version!
warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

When trying to run on a folder of videos I am getting the following error.

I have installed ffmpeg on Ubuntu 20 with: sudo apt-get install ffmpeg

on create_duplicates_gallery()

Install fails on python3.8
i wanted to edit the setuptools if it doesnt include 3.8, then couldnt find it.

Hi,

I was trying to use the fastdup https://visualdatabase.github.io/fastdup/#fastdup.run, as per the instructions mentioned in the link for zipped / tar compressed images.

but for some reason it does not work

Please can anyone suggest.

fastdup == 0.137

Hello! Amazing project, thank you for building it.

I can't find anywhere in the examples or documents the Graph Search function. Could you help me locate it?

I was trying to install fastdup into docker.

Fastdup cant be installed if pip version is not latest, or larger then 22.0. It says it couldn't find the package.
so pip install --upgrade pip was needed.

I think its worth mentioning this fix somewhere, that's the issue. Otherwise people that ll build in docker will spend 3 hours like me to find out whats going on.

Does the program now support EfficientNet weighting as backbone? My model: (batch, channel, width, height )= (1802, 1, 62, 96).output_size=(1803,1280). Here's how I used it:

fastdup.run('/home/datasets', work_dir='out', nearest_neighbors_k=478, model_path='efficientnetb1.ort', d=1280)
When I use the model, I get the following error, what parameters need to be adjusted in the source code to apply, can you give some suggestions?
Found total 479 images to run on Failed assertion false /home/ubuntu/visual_database/cxx/src/image_to_blob.h:222 Failed assertion false /home/ubuntu/visual_database/cxx/src/image_to_blob.h:222 free(): corrupted unsorted chunks Segmentation fault (core dumped)