qdrant / qdrant Goto Github PK

Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/

Home Page: https://qdrant.tech

License: Apache License 2.0

Rust 87.92% Shell 1.38% Dockerfile 0.11% JavaScript 0.03% Python 10.55% Mermaid 0.02%

neural-network matching search-engine knn-algorithm hnsw vector-search nearest-neighbor-search image-search embeddings-similarity approximate-nearest-neighbor-search

qdrant's Introduction

Vector Search Engine for the next generation of AI applications

Qdrant (read: quadrant) is a vector similarity search engine and vector database. It provides a production-ready service with a convenient API to store, search, and manage points—vectors with an additional payload Qdrant is tailored to extended filtering support. It makes it useful for all sorts of neural-network or semantic-based matching, faceted search, and other applications.

Qdrant is written in Rust 🦀, which makes it fast and reliable even under high load. See benchmarks.

With Qdrant, embeddings or neural network encoders can be turned into full-fledged applications for matching, searching, recommending, and much more!

Qdrant is also available as a fully managed Qdrant Cloud ⛅ including a free tier.

Quick Start • Client Libraries • Demo Projects • Integrations • Contact

Getting Started

Python

pip install qdrant-client

The python client offers a convenient way to start with Qdrant locally:

from qdrant_client import QdrantClient
qdrant = QdrantClient(":memory:") # Create in-memory Qdrant instance, for testing, CI/CD
# OR
client = QdrantClient(path="path/to/db")  # Persists changes to disk, fast prototyping

Client-Server

To experience the full power of Qdrant locally, run the container with this command:

docker run -p 6333:6333 qdrant/qdrant

Now you can connect to this with any client, including Python:

qdrant = QdrantClient("http://localhost:6333") # Connect to existing Qdrant instance

Before deploying Qdrant to production, be sure to read our installation and security guides.

Clients

Qdrant offers the following client libraries to help you integrate it into your application stack with ease:

Official:
Community:
- Elixir
- PHP
- Ruby
- Java

Where do I go from here?

Quick Start Guide
End to End Colab Notebook demo with SentenceBERT and Qdrant
Detailed Documentation are great starting points
Step-by-Step Tutorial to create your first neural network project with Qdrant

Demo Projects

Discover Semantic Text Search 🔍

Unlock the power of semantic embeddings with Qdrant, transcending keyword-based search to find meaningful connections in short texts. Deploy a neural search in minutes using a pre-trained neural network, and experience the future of text search. Try it online!

Explore Similar Image Search - Food Discovery 🍕

There's more to discovery than text search, especially when it comes to food. People often choose meals based on appearance rather than descriptions and ingredients. Let Qdrant help your users find their next delicious meal using visual search, even if they don't know the dish's name. Check it out!

Master Extreme Classification - E-commerce Product Categorization 📺

Enter the cutting-edge realm of extreme classification, an emerging machine learning field tackling multi-class and multi-label problems with millions of labels. Harness the potential of similarity learning models, and see how a pre-trained transformer model and Qdrant can revolutionize e-commerce product categorization. Play with it online!

API

REST

Online OpenAPI 3.0 documentation is available here. OpenAPI makes it easy to generate a client for virtually any framework or programming language.

You can also download raw OpenAPI definitions.

gRPC

For faster production-tier searches, Qdrant also provides a gRPC interface. You can find gRPC documentation here.

Features

Filtering and Payload

Qdrant can attach any JSON payloads to vectors, allowing for both the storage and filtering of data based on the values in these payloads. Payload supports a wide range of data types and query conditions, including keyword matching, full-text filtering, numerical ranges, geo-locations, and more.

Filtering conditions can be combined in various ways, including should, must, and must_not clauses, ensuring that you can implement any desired business logic on top of similarity matching.

Hybrid Search with Sparse Vectors

To address the limitations of vector embeddings when searching for specific keywords, Qdrant introduces support for sparse vectors in addition to the regular dense ones.

Sparse vectors can be viewed as an generalisation of BM25 or TF-IDF ranking. They enable you to harness the capabilities of transformer-based neural networks to weigh individual tokens effectively.

Vector Quantization and On-Disk Storage

Qdrant provides multiple options to make vector search cheaper and more resource-efficient. Built-in vector quantization reduces RAM usage by up to 97% and dynamically manages the trade-off between search speed and precision.

Distributed Deployment

Qdrant offers comprehensive horizontal scaling support through two key mechanisms:

Size expansion via sharding and throughput enhancement via replication
Zero-downtime rolling updates and seamless dynamic scaling of the collections

Highlighted Features

Query Planning and Payload Indexes - leverages stored payload information to optimize query execution strategy.
SIMD Hardware Acceleration - utilizes modern CPU x86-x64 and Neon architectures to deliver better performance.
Async I/O - uses io_uring to maximize disk throughput utilization even on a network-attached storage.
Write-Ahead Logging - ensures data persistence with update confirmation, even during power outages.

Integrations

Examples and/or documentation of Qdrant integrations:

Cohere (blogpost on building a QA app with Cohere and Qdrant) - Use Cohere embeddings with Qdrant
DocArray - Use Qdrant as a document store in DocArray
Haystack - Use Qdrant as a document store with Haystack (blogpost).
LangChain (blogpost) - Use Qdrant as a memory backend for LangChain.
LlamaIndex - Use Qdrant as a Vector Store with LlamaIndex.
OpenAI - ChatGPT retrieval plugin - Use Qdrant as a memory backend for ChatGPT
Microsoft Semantic Kernel - Use Qdrant as persistent memory with Semantic Kernel

Contacts

Have questions? Join our Discord channel or mention @qdrant_engine on Twitter
Want to stay in touch with latest releases? Subscribe to our Newsletters
Looking for a managed cloud? Check pricing, need something personalised? We're at [email protected]

Contributors ✨

Thanks to the people who contributed to Qdrant:

_{Andrey Vasnetsov} 💻	_{Andre Zayarni} 📖	_{Joan Fontanals} 💻	_trean 💻	_Konstantin 💻	_{Daniil Naumetc} 💻	_{Viacheslav Poturaev} 📖
_{Alexander Galibey} 💻	_HaiCheViet 💻	_{Marcin Puc} 💻	_{Anton V.} 💻	_{Arnaud Gourlay} 💻	_{Egor Ivkov} 💻	_{Ivan Pleshkov} 💻
_Daniil 💻	_{Anton Kaliaev} 💻	_{Andre Julius} 💻	_{Prokudin Alexander} 💻	_{Tim Eggert} 💻	_{Gabriel Velo} 💻	_{Boqin Qin(秦伯钦)} 🐛
_{Russ Cam} 💻	_{erare-humanum} 💻	_{Roman Titov} 💻	_Hozan 💻	_George 💻	_{Kornél Csernai} 💻	_{Luis Cossío} 💻
_{Tim Visée} 💻	_{Timon Vonk} 💻	_{Yiping Deng} 💻	_{Alex Huang} 💻	_{Ibrahim M. Akrab} 💻	_stencillogic 💻	_{Moaz bin Mokhtar} 📖

License

Qdrant is licensed under the Apache License, Version 2.0. View a copy of the License file.

qdrant's People

Contributors

Stargazers

Watchers

Forkers

kiminh riffus zhenwang23 nan-wang hanxiao jina-ai joanfm silky azayarni isgasho xiaoconstantine zeta1999 jonatanraverta anubhabb zergey adbmd trean admariner kgrech gilcu2 kekonen sabari-madhu laplacekorea vearutop iho galibey kokizzu entn-at doytsujin sergey-gusarov logarithmus gklijs lyrictian hercules261188 haicheviet tranzystorekk angrycaptain19 charleschege smarthi e-ivkov gvelo ivanpleshkov prok20 waridley melekes daniilsunyaev notnorom asmmo monatis 1871mg simonsan rieger-jared mamu9 nnneznaika jtprof ansrivas artaxerces ariesdevil sunli829 hoangpq unnuio shreeshjha demonoid81 markussagen cometyang elbart frankfanslc cryptowealth-technology der-ofenmeister hubayirp supersupeng charleneleong-ai raziyeh-fa naveenhooda2000 apetcho roberto-xy teagum lgs russcam dev-protocol sfrias kacperlukawski mrsahabu snayan06 lightyear416 spyrosroum saberboy stjordanis mstyushin jacobjohansen aminembarki duzhanyuan anas-zafar georgepearse angelapytao devopstoday11 jarrodli hajepe dmgolembiowski davidko3

qdrant's Issues

Update tokio to the latest version

Is your feature request related to a problem? Please describe.
Currently used version of tokio library is older than latest major release

Describe the solution you'd like
Update tokio to at least major release (ideally - to the newest version, if it is possible)

Describe alternatives you've considered
Keep as is

Additional context
None

Expose gRPC API

Is your feature request related to a problem? Please describe.
Right now the only way to interact with the system is by using JSON API. It enforces the data to be encoded as a human-readable text format, which is not the most effective approarch. Having the binary encoded protocol like GRPc would be a big win for the project in my option

Describe the solution you'd like

Add 2 features to the project: web and grpc. Enable web by default, disable grpc by default until ready. They way we could merge the code during dev without affecting the production binary. Once ready we could potentially disable/enable the given type of API at the compile time.
Use tonic crate
Refactor the app crate to have 2 submodules under the API modules: web and grpc. Also, add a common submodule.
The common module would contain the functions like this:

fn get_collections(toc: TableOfContent) -> CollectionsResponse {
    let collections = toc
        .all_collections()
        .into_iter()
        .map(|name| CollectionDescription { name })
        .collect_vec();

    CollectionsResponse { collections }
}

Common module is to be used by web and grpc modules

Describe alternatives you've considered

There is also the grpc-rus create, but as per a few blogs I've read it is not production-ready yet
We could use other binary protocols, but I think the grpc is the most popular

Additional context
N/A

On flight HNSW indexing

Is your feature request related to a problem? Please describe.
HNSW Index is read (delete) only, it requires full rebuild to add a new point (which is currently compensated with additional write in-memory segments). It would be nice to implement real-time extendable HNSW Index.

Describe the solution you'd like
The main reason why HNSW segment is read-only is because it is too expensive to serialize full graph on disk.
But with key-value DB it might be possible to save individual link lists in DB.

Describe alternatives you've considered
Alternative - is to sync full graph asynchronously, but reasonable delay might be too short and create too much IO operations

Work on a single core machines

Current Behavior

As reported in #33 - service can not handle the insertion if only a single CPU\worker is available.

Steps to Reproduce

Run Qdrant with docker run --rm -it --cpuset-cpus=0 -p 6333:6333 generall/qdrant
Create collection
Insert vector into the collection with ?wait=true
Service is not responding

Expected Behavior

Should work with a single core same as with multiple cores.

Possible Solution

Do not create less than 2 workers even is a single CPU is available

Add Anton V

Add Viacheslav

Add tranzystorek-io

Add proper API descriptions in OpenAPI

Is your feature request related to a problem? Please describe.
API functionality might not be clear enough

Describe the solution you'd like
Add description field for each OpenAPI endpoint

Add Daniil

qdrant shutting down when performing a search query.

Discussed in #115

^{Originally posted by wammy19 October 27, 2021}
Hi,

I've been using qdrant for a while now in a system, but am running into an issue all of a sudden. When performing a search query (using the python client), qdrant is terminating and the client is spitting out this error message:

qdrant_openapi_client.exceptions.ResponseHandlingException: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE

There's no error coming from the docker container, it's just exiting.

I've not changed any of my search code (which was working just fine yesterday), and I've just updated to the latest version of qdrant.

Any idea's as to why this problem may be occurring? Any help is greatly appreciated.

Add Joan

Add Trean

Add clippy and linter to pre-commit hook

Is your feature request related to a problem? Please describe.
Right now developers might forget to use code linters and static analyzers before submitting changes and only get the feedback after CI runs, which might be too slow.

Describe the solution you'd like
Add pre-commit hook which should be executed before each commit to rust code. This hook should automatically check and (optionally) automatically fix issues related to code formatting.

Describe alternatives you've considered
Do we want to force auto-fix of formatting?

Additional context
No

Payload type consistency is not enforced

The documentation states:

(...) In other words, it is impossible to have one vector associated with payload {"price": 11.99}, and another vector associated with {"price": "cheap"}. To maintain type consistency, Qdrant has a payload schema associated with the collection.

Source: https://qdrant.tech/documentation/payload/

(1) Create a new collection:

curl --request POST \
  --url 'https://***/collections?wait=true' \
  --header 'Content-Type: application/json' \
  --data '{
	"create_collection": {
		"name": "example_collection_2",
		"distance": "Cosine",
		"vector_size": 3
	}
}'

(2) Upsert point with float payload

curl --request POST \
  --url 'https://***/collections/example_collection_2?wait=true' \
  --header 'Content-Type: application/json' \
  --data '{
	"upsert_points": {
		"points": [
			{
				"id": 1,
				"payload": {
					"price": 11.8
				},
				"vector": [
					0.9,
					0.1,
					0.1
				]
			}
		]
	}
}'

(3) Read the collection

curl --request GET \
  --url https://***/collections/example_collection_2

{
  "result": {
    "status": "green",
    "vectors_count": 1,
    "segments_count": 5,
    "disk_data_size": 0,
    "ram_data_size": 0,
    "config": {
      "params": {
        "vector_size": 3,
        "distance": "Cosine"
      },
      "hnsw_config": {
        "m": 64,
        "ef_construct": 500,
        "full_scan_threshold": 10000
      },
      "optimizer_config": {
        "deleted_threshold": 0.2,
        "vacuum_min_vector_number": 1000,
        "max_segment_number": 5,
        "memmap_threshold": 50000,
        "indexing_threshold": 20000,
        "payload_indexing_threshold": 10000,
        "flush_interval_sec": 10
      },
      "wal_config": {
        "wal_capacity_mb": 32,
        "wal_segments_ahead": 0
      }
    },
    "payload_schema": {
      "price": {
        "data_type": {
          "type": "float"
        },
        "indexed": false
      }
    }
  },
  "status": "ok",
  "time": 0.000025104
}

(4) Upsert point with string payload

curl --request POST \
  --url 'https://***/collections/example_collection_2?wait=true' \
  --header 'Content-Type: application/json' \
  --data '{
	"upsert_points": {
		"points": [
			{
				"id": 2,
				"payload": {
					"price": "cheap"
				},
				"vector": [
					0.9,
					0.1,
					0.1
				]
			}
		]
	}
}'

This returns (I would expect an error here):

{
  "result": {
    "operation_id": 3,
    "status": "completed"
  },
  "status": "ok",
  "time": 0.00023371
}

(5) Scroll points:

curl --request POST \
  --url https://***/collections/example_collection_2/points/scroll \
  --header 'Content-Type: application/json' \
  --data '{
  "filter": null,
  "limit": 10,
  "offset": 0,
  "with_payload": true,
  "with_vector": false
}'

{
  "result": {
    "points": [
      {
        "id": 1,
        "payload": {
          "price": {
            "type": "float",
            "value": [
              11.8
            ]
          }
        },
        "vector": null
      },
      {
        "id": 2,
        "payload": {
          "price": {
            "type": "keyword",
            "value": [
              "cheap"
            ]
          }
        },
        "vector": null
      }
    ],
    "next_page_offset": null
  },
  "status": "ok",
  "time": 0.000036864
}

Note the mixed type for the price attribute. I tried versions 0.3.5 and 0.3.6.

Handle corner case of limit = 0 in scroll API

Special case of limit: 0 in not handled properly in scroll API.

Current Behavior

Service panics if limit: 0 in scroll API

Steps to Reproduce

curl -L -X POST "http://$QDRANT_HOST/collections/test_collection/points/scroll" \
  --fail -s \
  -H 'Content-Type: application/json' \
  --data-raw '{ "offset": 0, "limit": 0}'

on any collection

Expected Behavior

400 error

Possible Solution

add check for limit value

Collection status indication

Is your feature request related to a problem? Please describe.
Performance of the search may vary if some segments didn't finish index-building or optimization process.
In some applications it might be useful to wait until the process it complete before enabling production workload.

Describe the solution you'd like
Implement collection status indication in collection info API.
Status should display if some segments of the collection are currently under optimization or not.
Also implement wait-for-status API, which should return result only ones the status is green and halt otherwise.

Describe alternatives you've considered
Implement detailed segment into API

Additional context
None

Remove unimplemented RAM and Disk usage computation for collection statistics

Allow to include payload and vector into search result

Is your feature request related to a problem? Please describe.
Right now search API can only return Id + score. If application requires payload or vector as well, it should make another retrieve call.

Describe the solution you'd like
Introduce with_payload and with_vector flags to search API, which should work similar to retrieve. If with_payload=true, the result should also include payload of the found points. If with_vector=true - vector should also be included.
The flags should be included in either GET or POST parameters ( I would prefer POST) and should be optional for backward compatibility.

Describe alternatives you've considered
Alternative approach for with_payload would be a possibility to specify the exact list of required payload keys. Similar to projection in Mongo, or _source in elastic.

Search query return status 500, error panic

Firstly, thanks a lot for the vector DB really fast and provide what I really need, and still missing in other DB is a filter.

I currently push a lot vector (~100k+) in one collection with a shape array of about 592. When performing a search query, I get the error panic 500.
But I still can get payload and vector in an individual point in this collection and search in another collection.

Current Behavior

Perform search return 500 error and panic error

Expected Behavior

Perform search without return 500 error

Context (Environment)

Docker image latest
Default config

Detailed Description

The logs docker when I performed searching:
thread 'tokio-runtime-worker' panicked at 'Corrupter id_mapper, no external value for 302', lib/segment/src/segment.rs:130:25

Possible Implementation

When pushing a lot vector, I iterate push per batch (1500 vector) and after a certain batch, the API in a specific collection is timeout.
Then I wait about 30 minutes and test push again and succeed, I think the worker is indexing that causes the timeout but after done pushed. I can't perform searching as above

HTTP request semantics: reading a non-existent collection should return 404 Not Found instead of 400 Client error

I think it would be semantically more correct to return a HTTP status code 404 if I'm trying to read a non-existent collection. Now the client gets a 400 client error.

Current behaviour:

> GET /collections/non_existent_collection HTTP/2
> Host: ***
> accept: */*

< HTTP/2 400 
< date: Fri, 24 Sep 2021 11:00:59 GMT
< content-type: application/json
< content-length: 107

According to MDN, I think the 404 code fits perfectly:

The HTTP 404 Not Found client error response code indicates that the server can't find the requested resource.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404

Allow per-collection configuration

Is your feature request related to a problem? Please describe.
Currently most of the collection-related parameters can only be selected with configuration file, it would be more flexible, if it would be possible to override them for each collection individually

Describe the solution you'd like
Use configuration as a source of default values, store copy of config for each collection independently, implement a parameter update API.

Describe alternatives you've considered
Keep as is

Additional context
None

Score filtering request

Is your feature request related to a problem? Please describe.
Allow a filtering condition based on score result

Describe the solution you'd like
Specify minimal (maximal) score in query condition

Describe alternatives you've considered
Search threshold as a separate search condition (do not include into must-should-clauses)

Delete points by filter

Is your feature request related to a problem? Please describe.
E.g. In case of batch updates it might be useful to delete outdated results in batches as well.

Describe the solution you'd like
API which requires a filter and deletes all points which satisfy given filter conditions

Describe alternatives you've considered
None

Additional context
https://github.com/qdrant/qdrant/projects/1#card-51002295

Multi-threaded optimizer

Is your feature request related to a problem? Please describe.
Index building might be slow, it utilizes only a single CPU

Describe the solution you'd like
Allow multi-threaded optimizer

Describe alternatives you've considered
No

Additional context
No

Add HaiCheViet

Iterate records

Is your feature request related to a problem? Please describe.

I want to test the performance of different encoders. To this end, I would like to create a collection containing payloads of a number of documents and then create different collections holding vectors created by different encoders. My life would be easier if there was a way to iterate all available records.

Describe the solution you'd like

Something like collection.keys()

I'm not too familiar with RocksDB, but I know that it provides a way to get all keys, see for example https://python-rocksdb.readthedocs.io/en/latest/tutorial/#iteration

No point with id ... found when upserting points in collection with payload

When the collection reaches a certain size, Qdrant stops accepting new points with payload. All requests end with a
404 Not Found: no point with id xxx found error.

Collection

GET https://***/collections/my_collection

{
  "result": {
    "status": "yellow",
    "vectors_count": 1562043,
    "segments_count": 14,
    "disk_data_size": 0,
    "ram_data_size": 0,
    "config": {
      "params": {
        "vector_size": 1536,
        "distance": "Cosine"
      },
      "hnsw_config": {
        "m": 64,
        "ef_construct": 500,
        "full_scan_threshold": 10000
      },
      "optimizer_config": {
        "deleted_threshold": 0.2,
        "vacuum_min_vector_number": 1000,
        "max_segment_number": 5,
        "memmap_threshold": 50000,
        "indexing_threshold": 20000,
        "payload_indexing_threshold": 10000,
        "flush_interval_sec": 10
      },
      "wal_config": {
        "wal_capacity_mb": 32,
        "wal_segments_ahead": 0
      }
    },
    "payload_schema": {
      "category_id": {
        "data_type": {
          "type": "integer"
        },
        "indexed": false
      },
      "manufacturer_id": {
        "data_type": {
          "type": "integer"
        },
        "indexed": false
      },
      "product_id": {
        "data_type": {
          "type": "integer"
        },
        "indexed": false
      },
      "eshop_id": {
        "data_type": {
          "type": "integer"
        },
        "indexed": false
      },
      "project_id": {
        "data_type": {
          "type": "integer"
        },
        "indexed": false
      },
      "color_hex": {
        "data_type": {
          "type": "keyword"
        },
        "indexed": false
      }
    }
  },
  "status": "ok",
  "time": 0.000066578
}

Example of failing request

POST to https://***/collections/my_collection?wait=true

{
	"upsert_points": {
		"points": [
			{
				"id": 110911491,
				"payload": {
					"product_id": {
						"type": "integer",
						"value": 141999489
					},
					"category_id": {
						"type": "integer",
						"value": 5323
					},
					"eshop_id": {
						"type": "integer",
						"value": 8213
					},
					"project_id": {
						"type": "integer",
						"value": 1
					},
					"manufacturer_id": {
						"type": "integer",
						"value": 0
					},
					"color_hex": {
						"type": "keyword",
						"value": "#999999"
					}
				},
				"vector": [
					-0.143310546875,
					-0.1573486328125,
					-0.09161376953125,
					-0.01230621337890625,
					-0.1424560546875,
					0.07672119140625,
					-0.09765625,
					-0.1265869140625,
					-0.190673828125,
					0.1650390625,
					0.031402587890625,
					-0.09796142578125,
					0.97607421875,
					0.292724609375,
					-0.06005859375,
					0.4375,
					1.251953125,
					0.0276031494140625,
					-0.0281219482421875,
					0.53515625,
					0.2154541015625,
					0.1395263671875,
					1.10546875,
					-0.064453125,
					-0.192626953125,
					0.09423828125,
					-0.1802978515625,
					0.1710205078125,
					-0.020355224609375,
					0.177978515625,
					-0.12548828125,
					-0.10687255859375,
					-0.130126953125,
					-0.1251220703125,
					-0.1019287109375,
					-0.0362548828125,
					0.11517333984375,
					-0.1578369140625,
					-0.171142578125,
					0.71728515625,
					-0.1890869140625,
					-0.1142578125,
					-0.1427001953125,
					-0.0020275115966796875,
					-0.17919921875,
					-0.1905517578125,
					-0.1710205078125,
					-0.022979736328125,
					-0.05413818359375,
					-0.1334228515625,
					-0.1761474609375,
					0.25048828125,
					-0.1644287109375,
					-0.2359619140625,
					0.1669921875,
					0.63427734375,
					0.2476806640625,
					-0.178955078125,
					-0.10809326171875,
					-0.03521728515625,
					0.031768798828125,
					-0.1593017578125,
					0.86572265625,
					-0.1434326171875,
					0.271484375,
					-0.1558837890625,
					-0.176513671875,
					-0.1676025390625,
					-0.179443359375,
					-0.11431884765625,
					0.556640625,
					-0.0860595703125,
					0.068603515625,
					-0.0013456344604492188,
					-0.03472900390625,
					-0.07196044921875,
					-0.09765625,
					-0.06304931640625,
					-0.11492919921875,
					-0.1944580078125,
					-0.2164306640625,
					-0.1649169921875,
					0.47509765625,
					0.157958984375,
					-0.06732177734375,
					0.00608062744140625,
					-0.005573272705078125,
					-0.01503753662109375,
					0.26318359375,
					-0.0985107421875,
					0.1783447265625,
					0.0673828125,
					-0.07763671875,
					-0.09539794921875,
					0.212890625,
					0.318115234375,
					0.042022705078125,
					-0.1375732421875,
					-0.0699462890625,
					-0.12017822265625,
					-0.1414794921875,
					-0.003955841064453125,
					-0.150146484375,
					0.042694091796875,
					-0.16064453125,
					-0.0004341602325439453,
					0.308837890625,
					0.0650634765625,
					-0.1732177734375,
					0.03857421875,
					0.041259765625,
					-0.213134765625,
					-0.20263671875,
					-0.10906982421875,
					-0.0758056640625,
					-0.1431884765625,
					0.1295166015625,
					-0.058563232421875,
					-0.1473388671875,
					-0.08380126953125,
					-0.0986328125,
					-0.0291290283203125,
					-0.185302734375,
					-0.0977783203125,
					0.034088134765625,
					-0.182373046875,
					-0.0986328125,
					1.6748046875,
					0.037445068359375,
					-0.1844482421875,
					-0.1658935546875,
					-0.1593017578125,
					0.30810546875,
					0.82470703125,
					-0.10308837890625,
					-0.006046295166015625,
					0.140625,
					-0.11920166015625,
					0.90185546875,
					-0.054840087890625,
					0.0131378173828125,
					0.11602783203125,
					-0.10089111328125,
					-0.07855224609375,
					-0.043975830078125,
					0.2381591796875,
					0.2484130859375,
					-0.10247802734375,
					0.081787109375,
					-0.192626953125,
					0.383544921875,
					-0.1141357421875,
					-0.1383056640625,
					-0.045989990234375,
					-0.1380615234375,
					-0.1551513671875,
					-0.1942138671875,
					1.4072265625,
					-0.01361846923828125,
					-0.07415771484375,
					-0.1279296875,
					-0.15478515625,
					-0.0697021484375,
					0.293701171875,
					0.335693359375,
					-0.043182373046875,
					0.44287109375,
					-0.12420654296875,
					-0.1046142578125,
					-0.0094757080078125,
					-0.0941162109375,
					-0.08782958984375,
					-0.223876953125,
					-0.206787109375,
					0.001155853271484375,
					-0.0240020751953125,
					-0.165283203125,
					0.1834716796875,
					0.00788116455078125,
					-0.06689453125,
					-0.16162109375,
					-0.12298583984375,
					-0.12890625,
					-0.08038330078125,
					-0.1318359375,
					0.209716796875,
					-0.04541015625,
					-0.017242431640625,
					-0.0416259765625,
					-0.164794921875,
					0.016754150390625,
					-0.08050537109375,
					-0.1839599609375,
					0.476318359375,
					-0.0682373046875,
					-0.192626953125,
					-0.14404296875,
					-0.181396484375,
					-0.1207275390625,
					-0.032135009765625,
					-4.357099533081055e-05,
					-0.01306915283203125,
					0.114013671875,
					-0.11102294921875,
					-0.1954345703125,
					-0.092041015625,
					-0.1475830078125,
					0.030120849609375,
					-0.1549072265625,
					-0.1309814453125,
					-0.099365234375,
					-0.10009765625,
					-0.1287841796875,
					-0.13037109375,
					0.2301025390625,
					-0.06622314453125,
					-0.16650390625,
					-0.09735107421875,
					0.09521484375,
					0.7109375,
					-0.1099853515625,
					0.22119140625,
					-0.15478515625,
					0.1710205078125,
					-0.1676025390625,
					-0.03802490234375,
					-0.18115234375,
					-0.1263427734375,
					-0.110107421875,
					-0.1513671875,
					-0.2342529296875,
					-0.170654296875,
					-0.1063232421875,
					-0.11334228515625,
					-0.139892578125,
					-0.185791015625,
					-0.005462646484375,
					-0.069091796875,
					0.02899169921875,
					0.2119140625,
					-0.127685546875,
					-0.155517578125,
					-0.11102294921875,
					-0.0748291015625,
					-0.162109375,
					-0.09423828125,
					0.2119140625,
					0.00359344482421875,
					-0.147216796875,
					-0.08837890625,
					-0.09442138671875,
					-0.185546875,
					-0.034423828125,
					-0.0643310546875,
					-0.157470703125,
					0.0254058837890625,
					-0.177490234375,
					-0.16259765625,
					0.2183837890625,
					-0.0936279296875,
					-0.1771240234375,
					-0.019805908203125,
					-0.151611328125,
					0.6044921875,
					-0.053192138671875,
					-0.1385498046875,
					-0.163330078125,
					0.1575927734375,
					-0.0792236328125,
					-0.147216796875,
					-0.14404296875,
					0.18310546875,
					0.0133514404296875,
					-0.155029296875,
					-0.17919921875,
					-0.1265869140625,
					-0.1663818359375,
					0.1343994140625,
					-0.0204010009765625,
					-0.031829833984375,
					1.029296875,
					-0.18994140625,
					0.045867919921875,
					-0.0755615234375,
					-0.131591796875,
					0.005329132080078125,
					-0.0894775390625,
					0.11474609375,
					0.12115478515625,
					-0.10540771484375,
					0.052337646484375,
					-0.204345703125,
					0.049407958984375,
					0.0716552734375,
					-0.16943359375,
					-0.17236328125,
					-0.174560546875,
					-0.14794921875,
					-0.17919921875,
					-0.11578369140625,
					-0.130126953125,
					-0.203369140625,
					0.496337890625,
					-0.1328125,
					-0.09356689453125,
					-0.11236572265625,
					-0.048828125,
					-0.058746337890625,
					-0.1348876953125,
					0.002765655517578125,
					-0.2293701171875,
					-0.08837890625,
					-0.086181640625,
					0.387451171875,
					-0.173583984375,
					0.357421875,
					-0.0071258544921875,
					0.1962890625,
					-0.06756591796875,
					-0.1181640625,
					-0.177001953125,
					0.0026340484619140625,
					0.2313232421875,
					0.26806640625,
					0.30859375,
					-0.1541748046875,
					-0.1790771484375,
					-0.1676025390625,
					-0.197509765625,
					-0.18701171875,
					0.034881591796875,
					-0.1868896484375,
					-0.064208984375,
					-0.06317138671875,
					0.2244873046875,
					0.0701904296875,
					-0.09356689453125,
					-0.0311126708984375,
					0.1436767578125,
					-0.17724609375,
					-0.1593017578125,
					-0.1636962890625,
					0.058502197265625,
					-0.056549072265625,
					-0.180419921875,
					0.07427978515625,
					-0.154052734375,
					0.48193359375,
					-0.07196044921875,
					0.58447265625,
					0.07684326171875,
					-0.1492919921875,
					-0.0491943359375,
					-0.11138916015625,
					-0.11492919921875,
					-0.09893798828125,
					-0.2015380859375,
					-0.198974609375,
					-0.14892578125,
					0.73974609375,
					-0.0106964111328125,
					1.009765625,
					-0.1903076171875,
					-0.1826171875,
					-0.0518798828125,
					-0.188720703125,
					-0.153076171875,
					-0.07659912109375,
					-0.1588134765625,
					-0.115234375,
					-0.17236328125,
					0.17333984375,
					0.57177734375,
					-0.054901123046875,
					-0.1522216796875,
					-0.11907958984375,
					-0.1884765625,
					0.332763671875,
					-0.11456298828125,
					-0.15283203125,
					0.2327880859375,
					-0.09356689453125,
					-0.133544921875,
					-0.0380859375,
					-0.087890625,
					-0.032135009765625,
					1.1298828125,
					-0.0243377685546875,
					-0.1363525390625,
					0.44775390625,
					0.367919921875,
					0.88623046875,
					-0.05401611328125,
					-0.1842041015625,
					0.4423828125,
					0.80078125,
					-0.16552734375,
					-0.054962158203125,
					0.64208984375,
					-0.126953125,
					-0.03460693359375,
					-0.2127685546875,
					-0.050048828125,
					-0.123779296875,
					0.77685546875,
					-0.0181884765625,
					0.3876953125,
					0.14794921875,
					-0.017913818359375,
					-0.10467529296875,
					-0.07781982421875,
					-0.078125,
					-0.201416015625,
					-0.130615234375,
					0.001007080078125,
					-0.1146240234375,
					0.07293701171875,
					-0.155517578125,
					0.135986328125,
					0.171142578125,
					-0.146240234375,
					-0.146240234375,
					-0.061126708984375,
					-0.1358642578125,
					0.12890625,
					-0.039276123046875,
					-0.03857421875,
					0.8564453125,
					-0.0513916015625,
					0.2315673828125,
					-0.0823974609375,
					-0.1614990234375,
					-0.1190185546875,
					0.68994140625,
					-0.1307373046875,
					-0.11248779296875,
					-0.06854248046875,
					-0.158447265625,
					0.01517486572265625,
					0.24853515625,
					0.11798095703125,
					0.3193359375,
					-0.05810546875,
					-0.1151123046875,
					0.20849609375,
					0.087158203125,
					-0.1688232421875,
					-0.1639404296875,
					0.161376953125,
					-0.0263824462890625,
					0.1656494140625,
					-0.1522216796875,
					-0.1319580078125,
					-0.16650390625,
					-0.10162353515625,
					-0.1451416015625,
					0.218994140625,
					0.18603515625,
					0.677734375,
					0.0029621124267578125,
					0.76318359375,
					0.81103515625,
					0.348388671875,
					-0.08953857421875,
					-0.01033782958984375,
					0.09320068359375,
					0.7275390625,
					0.39697265625,
					-0.1217041015625,
					0.97900390625,
					-0.1387939453125,
					0.323486328125,
					-0.10791015625,
					0.057159423828125,
					-0.1353759765625,
					-0.08221435546875,
					-0.0462646484375,
					-0.13330078125,
					-0.09881591796875,
					0.580078125,
					-0.144287109375,
					-0.0921630859375,
					-0.114990234375,
					0.2259521484375,
					-0.0308380126953125,
					0.0723876953125,
					0.15673828125,
					-0.12322998046875,
					-0.11077880859375,
					0.307373046875,
					0.5205078125,
					-0.158447265625,
					-0.0419921875,
					-0.1875,
					-0.1787109375,
					0.185302734375,
					0.544921875,
					-0.1961669921875,
					-0.131591796875,
					-0.0146636962890625,
					0.134765625,
					-0.1484375,
					-0.1195068359375,
					0.2626953125,
					-0.102294921875,
					-0.1419677734375,
					-0.060150146484375,
					-0.1915283203125,
					0.29736328125,
					-0.08709716796875,
					-0.060546875,
					-0.1490478515625,
					0.1624755859375,
					-0.1622314453125,
					-0.1610107421875,
					-0.12066650390625,
					-0.196533203125,
					0.177490234375,
					-0.1435546875,
					-0.07586669921875,
					-0.17236328125,
					0.52880859375,
					-0.04949951171875,
					-0.145751953125,
					0.01641845703125,
					-0.05517578125,
					0.0020847320556640625,
					0.0266571044921875,
					0.28759765625,
					-0.1317138671875,
					-0.06622314453125,
					-0.11962890625,
					-0.195556640625,
					-0.1287841796875,
					-0.2080078125,
					0.0038661956787109375,
					0.260498046875,
					-0.1552734375,
					0.0299224853515625,
					0.56640625,
					-0.1307373046875,
					-0.10784912109375,
					0.11480712890625,
					0.9111328125,
					-0.158935546875,
					-0.12060546875,
					-0.1812744140625,
					-0.0562744140625,
					-0.147705078125,
					0.6806640625,
					-0.218505859375,
					-0.120849609375,
					0.1651611328125,
					0.1844482421875,
					-0.1407470703125,
					-0.1282958984375,
					-0.1578369140625,
					0.0943603515625,
					0.08416748046875,
					0.57275390625,
					-0.0987548828125,
					-0.11285400390625,
					1.1240234375,
					-0.1669921875,
					0.11175537109375,
					-0.0026874542236328125,
					0.405029296875,
					-0.1494140625,
					-0.080810546875,
					0.0189971923828125,
					0.36328125,
					-0.1175537109375,
					-0.1429443359375,
					-0.186279296875,
					-0.1602783203125,
					0.343017578125,
					-0.0966796875,
					0.056060791015625,
					0.187744140625,
					0.0193634033203125,
					-0.210693359375,
					0.6298828125,
					-0.008575439453125,
					-0.10528564453125,
					0.485107421875,
					1.6728515625,
					-0.15283203125,
					-0.04583740234375,
					0.1851806640625,
					0.07568359375,
					-0.0308380126953125,
					-0.1666259765625,
					-0.12030029296875,
					-0.140625,
					-0.2001953125,
					-0.09228515625,
					-0.138427734375,
					-0.124267578125,
					0.06536865234375,
					-0.198974609375,
					-0.01727294921875,
					-0.1600341796875,
					-0.0809326171875,
					-0.10791015625,
					-0.1536865234375,
					-0.1954345703125,
					0.06884765625,
					-0.139404296875,
					0.0128173828125,
					-0.118896484375,
					-0.04974365234375,
					-0.11767578125,
					-0.102294921875,
					-0.0273590087890625,
					-0.09033203125,
					-0.189453125,
					-0.175537109375,
					0.234130859375,
					-0.1610107421875,
					-0.1436767578125,
					-0.14599609375,
					-0.1934814453125,
					-0.06396484375,
					-0.163330078125,
					-0.1300048828125,
					-0.224365234375,
					-0.173095703125,
					-0.1318359375,
					0.369873046875,
					0.19287109375,
					-0.066162109375,
					-0.1142578125,
					-0.1142578125,
					0.243408203125,
					-0.031524658203125,
					0.82177734375,
					0.287841796875,
					-0.003437042236328125,
					0.10894775390625,
					-0.18359375,
					0.402587890625,
					-0.130859375,
					-0.11956787109375,
					-0.09088134765625,
					0.28076171875,
					-0.163330078125,
					0.2939453125,
					0.52880859375,
					-0.0880126953125,
					-0.0148162841796875,
					-0.0927734375,
					-0.097412109375,
					0.44384765625,
					0.0234832763671875,
					-0.125732421875,
					-0.060455322265625,
					0.488525390625,
					-0.07977294921875,
					0.041595458984375,
					-0.0916748046875,
					0.04595947265625,
					-0.005565643310546875,
					-0.1805419921875,
					-0.0156707763671875,
					-0.2264404296875,
					0.1229248046875,
					0.051544189453125,
					-0.2320556640625,
					-0.13232421875,
					0.26953125,
					0.72412109375,
					-0.1195068359375,
					-0.046661376953125,
					-0.1328125,
					-0.1923828125,
					-0.039215087890625,
					0.0028743743896484375,
					-0.1253662109375,
					0.10137939453125,
					-0.1414794921875,
					-0.15966796875,
					-0.1578369140625,
					-0.1827392578125,
					-0.1422119140625,
					-0.1087646484375,
					-0.01251983642578125,
					-0.1640625,
					-0.125732421875,
					-0.1497802734375,
					0.90673828125,
					0.279052734375,
					-0.011016845703125,
					0.08099365234375,
					0.034515380859375,
					-0.1585693359375,
					-0.031341552734375,
					-0.1435546875,
					-0.082763671875,
					-0.159423828125,
					1.18359375,
					0.047027587890625,
					0.1944580078125,
					0.298095703125,
					-0.10675048828125,
					-0.1842041015625,
					-0.1937255859375,
					-0.1671142578125,
					-0.06207275390625,
					-0.164794921875,
					0.2001953125,
					-0.1875,
					-0.066650390625,
					0.52001953125,
					-0.09625244140625,
					-0.2271728515625,
					-0.08740234375,
					0.4345703125,
					0.55908203125,
					0.049713134765625,
					0.43310546875,
					0.200927734375,
					-0.1263427734375,
					-0.174560546875,
					-0.11871337890625,
					0.102783203125,
					-0.1585693359375,
					0.28564453125,
					-0.12109375,
					-0.0780029296875,
					0.0250091552734375,
					-0.1300048828125,
					-0.1845703125,
					-0.1282958984375,
					-0.1517333984375,
					0.005573272705078125,
					-0.2261962890625,
					-0.1251220703125,
					-0.1722412109375,
					-0.1435546875,
					-0.021759033203125,
					-0.05072021484375,
					0.247314453125,
					0.609375,
					-0.1407470703125,
					-0.2003173828125,
					-0.04986572265625,
					0.12139892578125,
					-0.0723876953125,
					1.2998046875,
					1.15234375,
					-0.006954193115234375,
					0.1082763671875,
					-0.11181640625,
					0.280517578125,
					0.1761474609375,
					0.321533203125,
					-0.08935546875,
					-0.1219482421875,
					0.0400390625,
					0.1708984375,
					0.194091796875,
					-0.003841400146484375,
					-0.135498046875,
					-0.1854248046875,
					-0.0523681640625,
					-0.12176513671875,
					-0.0771484375,
					-0.15771484375,
					-0.1363525390625,
					-0.18212890625,
					0.1417236328125,
					0.25390625,
					-0.00475311279296875,
					-0.1683349609375,
					0.01922607421875,
					-0.0843505859375,
					0.1961669921875,
					-0.154052734375,
					0.072998046875,
					-0.1685791015625,
					-0.09722900390625,
					-0.146240234375,
					-0.1253662109375,
					-0.145751953125,
					-0.1724853515625,
					-0.0843505859375,
					-0.181884765625,
					0.5751953125,
					-0.162109375,
					0.279052734375,
					-0.12005615234375,
					0.1268310546875,
					0.63134765625,
					0.1478271484375,
					-0.06182861328125,
					-0.048309326171875,
					0.296142578125,
					-0.1064453125,
					0.250732421875,
					0.200439453125,
					0.2025146484375,
					-0.09197998046875,
					-0.102294921875,
					-0.2017822265625,
					-0.14208984375,
					-0.1064453125,
					-0.201416015625,
					-0.154296875,
					-0.0947265625,
					0.078369140625,
					-0.201904296875,
					0.03216552734375,
					-0.11309814453125,
					0.736328125,
					-0.08160400390625,
					-0.1561279296875,
					-0.09820556640625,
					0.11285400390625,
					0.93408203125,
					-0.012481689453125,
					-0.2213134765625,
					0.25927734375,
					-0.08355712890625,
					-0.08795166015625,
					-0.13232421875,
					-0.1583251953125,
					-0.134765625,
					0.326904296875,
					-0.1246337890625,
					0.11810302734375,
					-0.04864501953125,
					0.6025390625,
					-0.210693359375,
					0.1513671875,
					0.196044921875,
					-0.12384033203125,
					0.17724609375,
					-0.008392333984375,
					0.26318359375,
					0.234375,
					-0.1461181640625,
					-0.098388671875,
					-0.1217041015625,
					-0.08447265625,
					0.061859130859375,
					-0.082763671875,
					0.6064453125,
					0.218017578125,
					0.10150146484375,
					-0.1318359375,
					-0.1622314453125,
					0.89697265625,
					0.2459716796875,
					0.37353515625,
					0.0177154541015625,
					-0.1783447265625,
					0.050323486328125,
					-0.0205841064453125,
					-0.1895751953125,
					1.0859375,
					-0.1923828125,
					-0.118896484375,
					-0.049835205078125,
					-0.0289459228515625,
					-0.1728515625,
					-0.186279296875,
					-0.1822509765625,
					0.0692138671875,
					0.0943603515625,
					-0.135498046875,
					-0.043182373046875,
					-0.1346435546875,
					-0.035552978515625,
					-0.175537109375,
					-0.1466064453125,
					-0.12939453125,
					-0.160888671875,
					-0.1646728515625,
					0.12347412109375,
					-0.1700439453125,
					0.591796875,
					-0.1964111328125,
					0.7998046875,
					-0.197265625,
					0.06561279296875,
					-0.068603515625,
					0.1424560546875,
					0.1595458984375,
					-0.15576171875,
					-0.179443359375,
					-0.1229248046875,
					0.291259765625,
					-0.142333984375,
					-0.151611328125,
					0.06878662109375,
					-0.054473876953125,
					-0.1556396484375,
					0.148681640625,
					-0.1185302734375,
					-0.10797119140625,
					-0.11407470703125,
					0.251220703125,
					-0.0101318359375,
					-0.18212890625,
					1.4912109375,
					-0.1773681640625,
					0.291015625,
					0.07958984375,
					-0.1417236328125,
					-0.14013671875,
					0.0933837890625,
					-0.034210205078125,
					0.00684356689453125,
					-0.185302734375,
					-0.00894927978515625,
					-0.1983642578125,
					0.027435302734375,
					0.1673583984375,
					0.273193359375,
					0.180908203125,
					-0.10968017578125,
					-0.052703857421875,
					0.0282440185546875,
					0.322021484375,
					1.294921875,
					-0.064208984375,
					0.2391357421875,
					-0.1484375,
					0.79296875,
					-0.097900390625,
					-0.1490478515625,
					0.002895355224609375,
					-0.152099609375,
					-0.1524658203125,
					-0.1671142578125,
					-0.10101318359375,
					-0.048431396484375,
					-0.1146240234375,
					-0.1778564453125,
					-0.0265655517578125,
					0.771484375,
					-0.1246337890625,
					-0.1580810546875,
					-0.0977783203125,
					-0.224609375,
					-0.0740966796875,
					0.07135009765625,
					-0.1376953125,
					-0.06671142578125,
					0.248046875,
					-0.11968994140625,
					0.381103515625,
					-0.1988525390625,
					0.364990234375,
					-0.1953125,
					-0.12310791015625,
					-0.02960205078125,
					-0.20166015625,
					0.06414794921875,
					-0.09149169921875,
					-0.0010499954223632812,
					-0.086181640625,
					-0.087158203125,
					-0.12841796875,
					-0.1307373046875,
					0.371337890625,
					-0.1513671875,
					-0.05889892578125,
					0.2164306640625,
					-0.0048065185546875,
					-0.138427734375,
					-0.1396484375,
					-0.2071533203125,
					-0.0016345977783203125,
					0.6630859375,
					0.0860595703125,
					-0.187255859375,
					0.052520751953125,
					0.10009765625,
					-0.178466796875,
					-0.1622314453125,
					-0.09356689453125,
					-0.1566162109375,
					0.224365234375,
					-0.2001953125,
					0.07421875,
					-0.173095703125,
					-0.170654296875,
					-0.1014404296875,
					0.10272216796875,
					-0.1549072265625,
					-0.08154296875,
					-0.141845703125,
					-0.08984375,
					-0.1531982421875,
					-0.0296478271484375,
					-0.0220184326171875,
					-0.18017578125,
					-0.1416015625,
					-0.2125244140625,
					-0.06646728515625,
					0.218994140625,
					-0.049591064453125,
					-0.0086517333984375,
					-0.0963134765625,
					0.63671875,
					-0.1129150390625,
					0.0005483627319335938,
					0.2900390625,
					-0.039154052734375,
					0.0168609619140625,
					-0.158203125,
					-0.05169677734375,
					0.2342529296875,
					0.205322265625,
					-0.08880615234375,
					1.2392578125,
					0.18994140625,
					-0.11688232421875,
					-0.09857177734375,
					0.0231475830078125,
					-0.11187744140625,
					0.062164306640625,
					-0.00949859619140625,
					0.03643798828125,
					0.466552734375,
					-0.0312347412109375,
					-0.05926513671875,
					0.222412109375,
					-0.01169586181640625,
					0.0027484893798828125,
					0.292724609375,
					-0.134765625,
					-0.0606689453125,
					0.42138671875,
					-0.1700439453125,
					-0.09814453125,
					0.0394287109375,
					1.0,
					-0.128662109375,
					-0.042388916015625,
					-0.1414794921875,
					-0.085205078125,
					0.517578125,
					0.38232421875,
					-0.1529541015625,
					-0.1976318359375,
					-0.192138671875,
					-0.116455078125,
					0.250732421875,
					-0.1561279296875,
					-0.1376953125,
					-0.035369873046875,
					0.0137176513671875,
					0.73583984375,
					0.046478271484375,
					-0.0028553009033203125,
					0.85986328125,
					-0.1744384765625,
					-0.1865234375,
					-0.0055389404296875,
					0.236083984375,
					0.364990234375,
					0.291015625,
					0.0169219970703125,
					-0.05535888671875,
					-0.1500244140625,
					7.647275924682617e-05,
					-0.036834716796875,
					-0.197509765625,
					-0.0033550262451171875,
					0.50341796875,
					-0.1788330078125,
					-0.0723876953125,
					-0.0601806640625,
					-0.06195068359375,
					0.02398681640625,
					-0.0259246826171875,
					-0.1705322265625,
					-0.078125,
					-0.0714111328125,
					0.1737060546875,
					-0.1435546875,
					0.00396728515625,
					0.11907958984375,
					-0.166748046875,
					0.09393310546875,
					-0.03387451171875,
					-0.0968017578125,
					-0.10760498046875,
					-0.065673828125,
					-0.1011962890625,
					-0.026947021484375,
					-0.1871337890625,
					-0.08404541015625,
					0.4482421875,
					-0.133056640625,
					-0.115478515625,
					-0.1317138671875,
					-0.1805419921875,
					-0.08575439453125,
					0.405517578125,
					-0.12322998046875,
					-0.1671142578125,
					-0.0574951171875,
					-0.07806396484375,
					0.132080078125,
					-0.165771484375,
					-0.1300048828125,
					0.498291015625,
					0.06353759765625,
					0.0019254684448242188,
					-0.01105499267578125,
					-0.0020618438720703125,
					-0.1917724609375,
					0.6552734375,
					-0.1837158203125,
					-0.098388671875,
					0.33642578125,
					-0.07427978515625,
					-0.10723876953125,
					-0.179931640625,
					-0.1805419921875,
					-0.1563720703125,
					-0.146484375,
					-0.138427734375,
					-0.061553955078125,
					0.1903076171875,
					0.376220703125,
					0.2249755859375,
					-0.1446533203125,
					-0.069091796875,
					-0.10498046875,
					-0.061492919921875,
					-0.1197509765625,
					-0.03460693359375,
					-0.0997314453125,
					-0.161376953125,
					0.25146484375,
					-0.1632080078125,
					0.5625,
					0.0697021484375,
					0.193603515625,
					-0.137451171875,
					-0.09698486328125,
					-0.1494140625,
					-0.2060546875,
					-0.189697265625,
					-0.1085205078125,
					-0.1234130859375,
					-0.07330322265625,
					-0.1004638671875,
					0.170166015625,
					-0.1527099609375,
					0.449951171875,
					-0.1387939453125,
					-0.061614990234375,
					-0.1170654296875,
					-0.0275726318359375,
					0.0909423828125,
					-0.1475830078125,
					-0.11273193359375,
					-0.0362548828125,
					-0.177978515625,
					-0.1610107421875,
					0.02496337890625,
					-0.0845947265625,
					0.98583984375,
					-0.10003662109375,
					-0.177001953125,
					0.0193023681640625,
					-0.1461181640625,
					-0.050262451171875,
					-0.149658203125,
					-0.1302490234375,
					-0.1319580078125,
					-0.1168212890625,
					-0.049530029296875,
					-0.1336669921875,
					0.240234375,
					0.08135986328125,
					0.56494140625,
					-0.14794921875,
					-0.1778564453125,
					-0.1832275390625,
					0.08746337890625,
					-0.1983642578125,
					-0.1356201171875,
					0.018280029296875,
					-0.05035400390625,
					-0.10406494140625,
					-0.07537841796875,
					-0.00659942626953125,
					-0.15478515625,
					0.49658203125,
					1.1845703125,
					0.02435302734375,
					-0.10845947265625,
					-0.137451171875,
					-0.140625,
					-0.12939453125,
					0.004802703857421875,
					0.8564453125,
					-0.174560546875,
					0.213134765625,
					-0.11297607421875,
					-0.0288848876953125,
					-0.033203125,
					0.09552001953125,
					0.26171875,
					-0.15771484375,
					0.21630859375,
					0.022857666015625,
					0.2073974609375,
					0.135498046875,
					-0.07562255859375,
					0.4677734375,
					0.09759521484375,
					-0.1470947265625,
					0.49365234375,
					0.1365966796875,
					-0.1507568359375,
					-0.2034912109375,
					0.63037109375,
					-0.07025146484375,
					0.0221405029296875,
					-0.156005859375,
					0.4248046875,
					0.218994140625,
					0.10638427734375,
					0.251708984375,
					0.3212890625,
					-0.07720947265625,
					-0.0928955078125,
					0.20654296875,
					-0.1903076171875,
					0.44677734375,
					-0.1055908203125,
					-0.002864837646484375,
					0.34716796875,
					-0.1273193359375,
					-0.1593017578125,
					-0.1700439453125,
					0.83984375,
					-0.1343994140625,
					-0.16455078125,
					0.142333984375,
					-0.1375732421875,
					-0.1240234375,
					0.01311492919921875,
					-0.1549072265625,
					-0.05792236328125,
					0.12249755859375,
					-0.1453857421875,
					-0.08953857421875,
					-0.1773681640625,
					0.006206512451171875,
					-0.09429931640625,
					-0.1434326171875,
					0.025146484375,
					-0.126220703125,
					0.00406646728515625,
					-0.1251220703125,
					-0.166259765625,
					-0.12445068359375,
					-0.146728515625,
					-0.0831298828125,
					-0.1785888671875,
					0.1884765625,
					0.0055694580078125,
					0.49853515625,
					0.34765625,
					0.1805419921875,
					-0.20751953125,
					-0.0726318359375,
					-0.099365234375,
					-0.0638427734375,
					0.149169921875,
					0.0172882080078125,
					0.05914306640625,
					0.69189453125,
					-0.143310546875,
					-0.08746337890625,
					-0.09783935546875,
					-0.2152099609375,
					0.430419921875,
					0.004150390625,
					-0.137939453125,
					-0.1971435546875,
					-0.1182861328125,
					-0.12274169921875,
					-0.09490966796875,
					-0.167236328125,
					-0.1971435546875,
					-0.1590576171875,
					-0.03558349609375,
					0.11456298828125,
					-0.061614990234375,
					0.12066650390625,
					-0.1649169921875,
					0.108642578125,
					0.59130859375,
					0.9599609375,
					-0.08709716796875,
					0.010345458984375,
					0.0272369384765625,
					0.1544189453125,
					0.360595703125,
					0.223388671875,
					-0.1103515625,
					-0.20703125,
					-0.107177734375,
					0.50341796875,
					-0.0938720703125,
					-0.148681640625,
					-0.08331298828125,
					-0.08673095703125,
					0.341796875,
					-0.1993408203125,
					-0.0389404296875,
					-0.1285400390625,
					-0.177978515625,
					-0.028106689453125,
					-0.11981201171875,
					-0.1585693359375,
					-0.017425537109375,
					-0.047454833984375,
					-0.125244140625,
					0.0164337158203125,
					-0.1697998046875,
					-0.08349609375,
					-0.11669921875,
					0.40869140625,
					0.057861328125,
					0.0119171142578125,
					-0.0258331298828125,
					-0.1627197265625,
					-0.10369873046875,
					-0.0753173828125,
					-0.1983642578125,
					-0.06622314453125,
					-0.10809326171875,
					0.56494140625,
					-0.11566162109375,
					-0.061065673828125,
					0.03485107421875,
					-0.1351318359375,
					-0.1622314453125,
					-0.1005859375,
					-0.1702880859375,
					-0.003543853759765625,
					-0.1617431640625,
					0.07977294921875,
					-0.16552734375,
					-0.16943359375,
					-0.017730712890625,
					-0.154541015625,
					0.5517578125,
					-0.11712646484375,
					-0.126953125,
					-0.2186279296875,
					0.0254058837890625,
					0.75732421875,
					0.54296875,
					-0.120849609375,
					0.2384033203125,
					-0.1474609375,
					-0.1075439453125,
					-0.08526611328125,
					-0.069580078125,
					0.7734375,
					-0.11663818359375,
					0.11102294921875,
					-0.0138092041015625,
					-0.1607666015625,
					0.3798828125,
					-0.09716796875,
					0.490966796875,
					0.1732177734375,
					-0.1915283203125,
					-0.1688232421875,
					0.70751953125,
					-0.1348876953125,
					-0.1669921875,
					0.06121826171875,
					-0.11572265625,
					-0.1328125,
					0.2998046875,
					-0.1383056640625,
					0.06817626953125,
					-0.1934814453125,
					-0.1285400390625,
					0.6201171875,
					-0.1871337890625,
					-0.1588134765625,
					-0.07208251953125,
					-0.00597381591796875,
					-0.137939453125,
					-0.12261962890625,
					-0.1533203125,
					-0.10955810546875,
					-0.014984130859375,
					-0.2254638671875,
					0.04412841796875,
					-0.173583984375,
					0.4853515625,
					-0.112548828125,
					-0.0345458984375,
					-0.1533203125,
					-0.0687255859375,
					0.031585693359375,
					-0.1072998046875,
					-0.127197265625,
					-0.14453125,
					-0.060272216796875,
					0.167236328125,
					0.658203125,
					-0.1484375,
					-0.1376953125,
					0.0309295654296875,
					0.62841796875,
					-0.05389404296875,
					-0.07720947265625,
					-0.02410888671875,
					-0.1258544921875,
					-0.18798828125,
					0.1640625,
					-0.22021484375,
					-0.1817626953125,
					-0.0833740234375,
					0.37060546875,
					0.2271728515625,
					0.34130859375,
					0.158203125,
					-0.11737060546875,
					-0.019256591796875,
					0.54150390625,
					0.26123046875,
					-0.094970703125,
					-0.11224365234375,
					-0.1624755859375,
					-0.0989990234375,
					-0.1761474609375,
					0.0070037841796875,
					-0.1910400390625,
					-0.132080078125,
					-0.1171875,
					-0.09381103515625,
					0.385986328125,
					0.23486328125,
					-0.1883544921875,
					-0.1658935546875,
					-0.0440673828125,
					0.529296875,
					-0.191650390625,
					-0.023681640625,
					-0.1025390625,
					-0.146240234375,
					0.07135009765625,
					-0.12158203125,
					0.5517578125,
					-0.174560546875,
					0.2083740234375,
					-0.138916015625,
					-0.16943359375,
					-0.0218505859375,
					-0.1044921875,
					0.07196044921875,
					-0.1514892578125,
					-0.12457275390625,
					0.0030193328857421875,
					0.1578369140625,
					-0.126708984375,
					-0.0274505615234375,
					-0.04461669921875,
					-0.07489013671875,
					0.0003311634063720703,
					-0.024688720703125,
					0.1971435546875,
					-0.217041015625,
					-0.16064453125,
					0.50146484375,
					-0.005096435546875,
					-0.19091796875,
					-0.2177734375,
					-0.1741943359375,
					-0.053924560546875,
					0.67578125,
					-0.1396484375,
					0.2178955078125,
					-0.006885528564453125,
					0.869140625,
					-0.1490478515625,
					-0.12091064453125,
					-0.11700439453125,
					0.88232421875,
					0.03839111328125,
					-0.016876220703125,
					0.51416015625,
					-0.19140625,
					0.50244140625,
					-0.184326171875,
					0.23828125,
					-0.051727294921875,
					-0.124755859375,
					0.69775390625,
					-0.05853271484375,
					-0.048126220703125,
					-0.05682373046875,
					-0.1649169921875,
					-0.0322265625,
					0.0157318115234375,
					-0.0267791748046875,
					-0.032928466796875,
					-0.1058349609375,
					0.2115478515625,
					-0.1546630859375,
					-0.182373046875,
					-0.1644287109375,
					-0.0716552734375,
					-0.182373046875,
					-0.1180419921875,
					0.4111328125,
					-0.04925537109375,
					-0.07476806640625,
					0.1942138671875,
					-0.157470703125,
					-0.1541748046875,
					0.0797119140625,
					-0.11004638671875,
					0.209228515625,
					-0.212890625,
					0.10919189453125,
					0.0177001953125,
					-0.2138671875,
					1.0830078125
				]
			}
		]
	}
}

Response (404):

{
  "result": null,
  "status": {
    "error": "No point with id 110911491 found"
  },
  "time": 0.000274568
}

Log record:

[2021-09-22T08:59:07Z INFO  actix_web::middleware::logger] 192.168.57.180:64938 "POST /collections/my_collection?wait=true HTTP/1.1" 404 88 "-" "insomnia/2021.5.3" 0.036706

If I send the same data without payload, the response is this (200):

{
  "result": {
    "operation_id": 68036,
    "status": "completed"
  },
  "status": "ok",
  "time": 0.000252208
}

Versions

I use this Docker image: generall/qdrant:v0.3.5.

Make euclidian metric positive

Right now Euclid metric is inverted (for some internal ordering reason). Make it positive again

Clean the project using clippy

fix issues suggested by https://github.com/rust-lang/rust-clippy
setup linter pipeline executed on project test

Closed WAL segments are not deleted

Issue reported by user https://github.com/iho via Telegram group

Current Behavior

WAL segments are stored along with collection data even after all changes are applied

Steps to Reproduce

Run a stress test.
Check the size of WAL folder

Expected Behavior

WAL folder size should be about the size of 2 WAL segments

Possible Solution

Check the WAL remove policy

Context (Environment)

None

Read points with optional filter without vector

Is your feature request related to a problem? Please describe.
In some cases it might be useful to be able to iterate over all points or points given some filter.

Describe the solution you'd like
API to retrieve paginated results of a given filter

Describe alternatives you've considered
None

Additional context
https://github.com/qdrant/qdrant/projects/1#card-51002295
#35

Collection remove times-out while optimization is in process

Current Behavior

Collection update request awaits for optimizer finishes even if it's a remove request

Steps to Reproduce

Crete collection
Enable indexing or other long optimization process
Try to re-create collection while optimization is still running
Request is not finished until optimization runs

Expected Behavior

Collection removed immediately

Possible Solution

Process finished handles
- Implement a JoinHandle wrapper, which could be checked if it is finished
- Implement removing of finished handles before each new optimization scheduting
Handle optimization shutdown
- Implement a stop-flag ( as AtomicBool) which could be flipped if it is required to stop optimization
- Optimization thread should check for the flag value periodically, and if it is flipped - exit the optimization and restore segment states (unwrap proxy, release locks, e.t.c.)
- Flip the flag on collection drop & on config update, gracefully wait for optimizations to finish

Collections return 0 results even after inserting vectors

I tried to follow the instructions given in the blog post as close as possible, but the neural search always shows 0 results. At no point did I get any errors.

When querying the collection itself, curl -s http://localhost:6333/collections/startups | jq .result.vectors_count returns 0.

The ./qdrant_storage folder keeps growing as I insert records though.

Here's the full output of the collections endpoint:

{
  "result": {
    "status": "green",
    "vectors_count": 0,
    "segments_count": 5,
    "disk_data_size": 0,
    "ram_data_size": 0,
    "config": {
      "params": {
        "vector_size": 768,
        "distance": "Dot"
      },
      "hnsw_config": {
        "m": 16,
        "ef_construct": 100,
        "full_scan_threshold": 10000
      },
      "optimizer_config": {
        "deleted_threshold": 0.2,
        "vacuum_min_vector_number": 1000,
        "max_segment_number": 5,
        "memmap_threshold": 50000,
        "indexing_threshold": 20000,
        "payload_indexing_threshold": 10000,
        "flush_interval_sec": 10
      },
      "wal_config": {
        "wal_capacity_mb": 32,
        "wal_segments_ahead": 0
      }
    },
    "payload_schema": {}
  },
  "status": "ok",
  "time": 3.8746e-05
}

Kind regards,
Alex

Search query crushes container (depends on host)

On one host system, a search query crushes the container. On another with the same data set it works fine.

Current Behavior

After a search query, docker container crushed.

Docker logs:
docker run -p 6333:6333 -l debug -v ${pwd}/qdrant/storage:/qdrant/storage generall/qdrant:

[2021-07-27T16:09:49Z INFO  wal::segment] Segment { path: "./storage/collections/test_collection/wal/open-5", entries: 31, space: (2424040/33554432)
}: opened
[2021-07-27T16:09:49Z INFO  wal::segment] Segment { path: "./storage/collections/test_collection/wal/open-6", entries: 0, space: (8/33554432) }: opened
[2021-07-27T16:09:49Z INFO  wal::segment] Segment { path: "./storage/collections/test_collection/wal/closed-1374", entries: 418, space: (33474360/33554432) }: opened
[2021-07-27T16:09:49Z INFO  wal] Wal { path: "./storage/collections/test_collection/wal", segment-count: 2, entries: [1374, 1823)  }: opened
[2021-07-27T16:09:56Z INFO  qdrant] loaded collection: test_collection
[2021-07-27T16:09:56Z INFO  actix_server::builder] Starting 12 workers
[2021-07-27T16:09:56Z INFO  actix_server::builder] Starting "actix-web-service-0.0.0.0:6333" service on 0.0.0.0:6333

Docker logs with strace:
strace docker run -p 6333:6333 -l debug -v ${pwd}/qdrant/storage:/qdrant/storage generall/qdrant:

futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL[2021-07-27T16:13:27Z INFO  wal::segment] Segment { path: "./storage/collections/test_collection/wal/open-5", entries: 31, space: (2424040/33554432) }: opened
[2021-07-27T16:13:27Z INFO  wal::segment] Segment { path: "./storage/collections/test_collection/wal/open-6", entries: 0, space: (8/33554432) }: opened
) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL[2021-07-27T16:13:27Z INFO  wal::segment] Segment { path: "./storage/collections/test_collection/wal/closed-1374", entries: 418, space: (33474360/33554432) }: opened
[2021-07-27T16:13:27Z INFO  wal] Wal { path: "./storage/collections/test_collection/wal", segment-count: 2, entries: [1374, 1823)  }: opened
) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL[2021-07-27T16:13:35Z INFO  qdrant] loaded collection: test_collection
[2021-07-27T16:13:35Z INFO  actix_server::builder] Starting 12 workers
[2021-07-27T16:13:35Z INFO  actix_server::builder] Starting "actix-web-service-0.0.0.0:6333" service on 0.0.0.0:6333
) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
futex(0x55d8f6e79c40, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn({mask=[]})                 = 202
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
futex(0x55d8f6e5cce8, FUTEX_WAIT_PRIVATE, 0, NULL) = 0
epoll_pwait(4, [], 128, 0, NULL, 0)     = 0
epoll_pwait(4,  <unfinished ...>)       = ?
+++ exited with 132 +++

Steps to Reproduce

Collection like:

{
    "create_collection": {
        "name": "test_collection",
        "vector_size": 256,
        "distance": "Cosine"
    }
}

Number of vectors does not affect the error.

Search request like:
POST /collections/test_collection/points/search

{
    'vector': [...],
    'top': 10
}

Expected Behavior

Stable container work and return search result. And сhecking resource availability.

Possible Solution

Maybe affected by processor instructions set or problems with free memory? What is the size of the free memory block for searching?

Context (Environment)

Host system based on Intel(R) Core(TM) i7-3930
Memory free: ~1.8-2G
Ubuntu 18.04
Docker version 20.10.2
latest pre-built image from DockerHub

The server should not allow you to delete a non-existent collection, instead an HTTP error should be returned

I think the server should be stricter to the client and an attempt to delete a non-existent collection should return an error (preferably error 404).

This is related to #99

> POST /collections HTTP/2
> Host: ***
> content-type: application/json
> accept: */*
> content-length: 54
| {
|     "delete_collection": "non_existent_collection"
| }


< HTTP/2 200 
< date: Fri, 24 Sep 2021 11:05:15 GMT
< content-type: application/json
< content-length: 46

Individual endpoints for collection & point operations

Is your feature request related to a problem? Please describe.

REST API documentation, and hence python client generated out of it, suffers from poor structure of update API.
OpenAPI supports Enums with structures poorly, so the readability of the documentation suffers.
Collection and point update endpoints are not clear enough.
It might be hard to find required operation in the OpenAPI Schema.

It would be useful to create additional REST endpoints for all point update method variants

Describe the solution you'd like

Create an additional endpoints for each operation individually
- For collection update
- For points update
Update python client with new APIs

Describe alternatives you've considered
Write better documentation

Additional context
Based on discussion in ODS.ai

API "routing" fails - 404 error

From time a time API returns a 404 error on trying to upsert points

Current Behavior

On system - process qdrant loads one processor core 100%

On server side:

...
[2021-07-30T05:27:55Z INFO  actix_web::middleware::logger] 172.17.0.1:43706 "POST /collections/test_collection?wait=true HTTP/1.1" 200 86 "-" "Python
/3.6 aiohttp/3.7.4.post0" 0.024770
[2021-07-30T05:28:01Z INFO  actix_web::middleware::logger] 172.17.0.1:43710 "POST /collections/test_collection?wait=true HTTP/1.1" 404 87 "-" "Python
/3.6 aiohttp/3.7.4.post0" 0.014020
[2021-07-30T05:28:09Z INFO  actix_web::middleware::logger] 172.17.0.1:43714 "POST /collections/test_collection?wait=true HTTP/1.1" 404 86 "-" "Python
/3.6 aiohttp/3.7.4.post0" 0.015836
[2021-07-30T05:28:17Z INFO  actix_web::middleware::logger] 172.17.0.1:43718 "POST /collections/test_collection?wait=true HTTP/1.1" 404 88 "-" "Python
/3.6 aiohttp/3.7.4.post0" 0.012977
...

On client side:

...
{'result': None, 'status': {'error': 'No point with id 100911063 found'}, 'time': 0.003710724}
{'result': None, 'status': {'error': 'No point with id 100890079 found'}, 'time': 0.009171441}
...

Steps to Reproduce

Create collection like:

{
    "create_collection": {
        "name": "test_collection",
        "vector_size": 256,
        "distance": "Cosine"
    }
}

Insert enough points making many POST requests. From 80 to 150 point on request, one request every 5-10 seconds.
After about 300-500 requests Qdrant returning 404 for each request

Expected Behavior

Message in server logs about cause of the error
tips to avoid this

Possible Solution

Wait for qdrant to restart web server or restart qdrant manually

Context (Environment)

Host system based on i7-3930
Ubuntu 18.04
Memory free: ~1G
Disk free
Docker version 20.10.2
latest docker image builted on host

Allow partial updated of serialized index

Is your feature request related to a problem? Please describe.
Right now it requires segment rebuild in order to create index for payload. This way minimizes disk writes and selected because right now index serializes fully into a single file.

Describe the solution you'd like
It could be useful to enable partial disk updates for payload index (using RocksDB or other approach) and thereby enable partial. updates of payload index. And therefor, enable online payload indexes without rebuild requirement

Describe alternatives you've considered
Instead of RocksDB enable additional layer of buffering which will only update index if some significant changes happened or flush forced

Additional context
Add any other context or screenshots about the feature request here.

Replace BLAS with packed_simd

https://github.com/rust-lang/packed_simd

Is your feature request related to a problem? Please describe.
BLAS is currently used for a single purpose - to make dot production faster. It looks like overkill.
Compile time is pretty high and it complicates the development

Describe the solution you'd like
Replace BLAS with packed_simd library, which appear to be much lighter.

Describe alternatives you've considered
Use some other lightweight library for SIMD (idk which one)

Additional context
None

Request: Support explicit schema changes

Is your feature request related to a problem? Please describe.

As collections in Qdrant grow, it is sometimes necessary to add a new column/field to an existing collection's payload.

Right now this can be done by manually creating a new record with the new column added, including the type flag.

This works OK but it would be good if there was a way to change the schema without upserting a new record.

This feature is especially important when using a connector library that features migrations.

Describe the solution you'd like

Ideally:

GET collections/mycoll/schema
..would return the current schema (like we get from the payload_schema fields from the POST collections/mycoll endpoint), and..

POST collections/mycoll/schema
..would allow us to add new fields.

On a secondary level, it would be killer if we could fill in a default for that new field in all existing point payloads.

Additional context

(Discussed 8/19/2021 on Qdrant Telegram group)

Add Andre

@all-contributors please add @azayarni for doc

Go Driver needed

extract nested payload as original

With the introduction of #15 , nested payload is unwrapped to have filtrable payload.

For instance:

{ "metadata": {"hello": "world"}}

is expanded to a flat key:

"metadata__hello": "world"

and now if I try to check the payload of this specific point I will get this flattened point.

It would be nice to get the original input back.

I think there could be 2 ways.

Convert it at extracting type, doing some sort of grouping by common prefixes and so on (like with a Trie)
What I think could be more mantainable, but at space cost, to store original Payload along with the filtrrable fields.
Some intermediate, keep duplicated payloads but JUST for the nested fields

What is your opinion on this? I could try to help a bit on this.

Prefer tokio::sync::RwLock and tokio::sync::Mutex over parking_lot ones in async functions

Is your feature request related to a problem? Please describe.
Tokio async runtime provides async version of RwLock and Mutex. When we are inside the async method, we should prefer using them instead of using sync implementations.
There are 2 reasons for it:

If you try to lock synchronous lock or mutex, you block the entire thread. There is no way for the reactor to perform a context switch and start to perform another async function while the required resource is occupied.
It inlines with the general rule of async rust: no blocking calls in async functions.
In certain cases the async API would accept not just Future trait, but Future + Send trait. This is quite often, the example could be tokio::spawn API. In a situation like below

let mut update_handler = removed.update_handler.lock(); // sync lock here, block the whole thread
update_handler.wait_worker_stops().await?;

the resulting feature (the async function calling these 2 lines) would NOT implement Send trait as synchronous MutexGuard obtained by lock method does not implement Send trait (the async one does). This would limit us on usage of the async method in such cases. Coming to practice now: the tonic grpc framework I use to implement its services with async functions which are required to implement Send trait and usage of the above lock is a blocker for it.

Describe the solution you'd like
Use async locks and mutex the locks which are only used in async methods

pub update_handler: Arc<tokio::sync::Mutex<UpdateHandler>>,
..
let mut update_handler = removed.update_handler.lock().await; //If resource is busy, reactor can run another future while current async function waits
update_handler.wait_worker_stops().await?;

Describe alternatives you've considered
We could keep existing implementation, however, it is likely less efficient and would block us with grpc additions.

Additional context

Add Konstantin

When I set the alias, qdrant cannot accesse

Hello:
When I set the alias, Qdrant cannot be accessed,But Qdrant's service is still alive.
Create alias & Remove alias & Switch collection all have this problem.
Even the newly created collection cannot create alias

qdrant version：0.3.5

thanks

expected u64 at line 1 column 2540

When doing a Recommendation search, I got this error in the request.

Current Behavior

Instead of doing the Recommendation search, it returns the error on the request.

Steps to Reproduce

$ curl -XPOST -H "Content-type: application/json" -d '{"filter": null, "negative": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 12, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], "params": null, "positive": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 13, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], "top": 5}' 'http://localhost:6333/collections/reddit_videos/points/recommend'

Note that these positive and negative vectors are embeddings returned from a deep learning model. The output I get is

Json deserialize error: invalid value: integer `-1`, expected u64 at line 1 column 3471(base)

Expected Behavior

Possible Solution

Context (Environment)

Detailed Description

Possible Implementation

Keep vectors in memeory while building a HNSW Index

Is your feature request related to a problem? Please describe.
It might be slow to build a HNSW index while vectors are stored in mmap storage.

Describe the solution you'd like
So during the training it might be helpful to load vectors in memory during the build.

Describe alternatives you've considered
Keep as is.

Additional context
Downside is that it will require a RAM size of at least one segment.


Chat Bots	Matching Engines	Anomaly Detection

qdrant / qdrant Goto Github PK

qdrant's Introduction

Getting Started

Python

Client-Server

Clients

Where do I go from here?

Demo Projects

Discover Semantic Text Search 🔍

Explore Similar Image Search - Food Discovery 🍕

Master Extreme Classification - E-commerce Product Categorization 📺

API

REST

gRPC

Features

Filtering and Payload

Hybrid Search with Sparse Vectors

Vector Quantization and On-Disk Storage

Distributed Deployment

Highlighted Features

Integrations

Contacts

Contributors ✨

License

qdrant's People

Contributors

Stargazers

Watchers

Forkers

qdrant's Issues

Current Behavior

Steps to Reproduce

Expected Behavior

Possible Solution

Discussed in #115

Current Behavior

Steps to Reproduce

Expected Behavior

Possible Solution

Current Behavior

Expected Behavior

Context (Environment)

Detailed Description

Possible Implementation

Collection

Example of failing request

Versions

Current Behavior

Steps to Reproduce

Expected Behavior

Possible Solution

Context (Environment)

Current Behavior

Steps to Reproduce

Expected Behavior

Possible Solution

Current Behavior

Steps to Reproduce

Expected Behavior

Possible Solution

Context (Environment)

Current Behavior

Steps to Reproduce

Expected Behavior

Possible Solution

Context (Environment)

Current Behavior

Steps to Reproduce

Expected Behavior

Possible Solution

Context (Environment)

Detailed Description

Possible Implementation

Recommend Projects

Recommend Topics

Recommend Org