What is used as feature extractor? about fastdup HOT 2 CLOSED

Hi @levan92 we use a proprietary feature extractor which we found to be both fast and gives accurate similarities for large variety of datasets. It is possible to support external models, for example CLIP is already supported. We are planning to release a demo notebook soon. Let us know if there are any features that are missing for your usecase?

from fastdup.

@levan92, You can get both the model name, layers names and output layer name and dim (~576) from the onnx model which is included as part of the package

For the nearest neighbour search they are probably using FAISS

~/.local/lib/python3.8/site-packages/fastdup $ nm -A libfastdup_shared.so
...
libfastdup_shared.so:                 U _ZN5faiss10read_indexEPKci
libfastdup_shared.so:                 U _ZN5faiss11write_indexEPKNS_5IndexEPKc
libfastdup_shared.so:                 U _ZN5faiss13index_factoryEiPKcNS_10MetricTypeE
libfastdup_shared.so:                 U _ZN5faiss14ParameterSpaceC1Ev
libfastdup_shared.so:                 U _ZN5faiss22NormalizationTransformC1Eif
libfastdup_shared.so:                 U _ZN6cppipc11must_cancelEv
...

from fastdup.