Giter VIP home page Giter VIP logo

qdrant-dotnet's Introduction

Qdrant .NET SDK

NuGet Release Build

📥 Installation

dotnet add package Qdrant.Client

📖 Documentation

Usage examples are available throughout the Qdrant documentation.

🔌 Getting started

Creating a client

A client can be instantiated with

var client = new QdrantClient("localhost");

which creates a client that will connect to Qdrant on http://localhost:6334.

Internally, the high level client uses a low level gRPC client to interact with Qdrant. Additional constructor overloads provide more control over how the gRPC client is configured. The following example configures a client to use TLS, validating the certificate using its thumbprint, and also configures API key authentication:

var channel = QdrantChannel.ForAddress("https://localhost:6334", new ClientConfiguration
{
    ApiKey = "<api key>",
    CertificateThumbprint = "<certificate thumbprint>"
});
var grpcClient = new QdrantGrpcClient(channel);
var client = new QdrantClient(grpcClient);

Important

IMPORTANT NOTICE for .NET Framework

.NET Framework has limited supported for gRPC over HTTP/2, but it can be enabled by

  • Configuring qdrant to use TLS, and you must use HTTPS, so you will need to set up server certificate validation
  • Referencing System.Net.Http.WinHttpHandler 6.0.1 or later, and configuring WinHttpHandler as the inner handler for GrpcChannelOptions

The following example configures a client for .NET Framework to use TLS, validating the certificate using its thumbprint, and also configures API key authentication:

var channel = GrpcChannel.ForAddress($"https://localhost:6334", new GrpcChannelOptions
{
  HttpHandler = new WinHttpHandler
  {
    ServerCertificateValidationCallback =
      CertificateValidation.Thumbprint("<certificate thumbprint>")
  }
});
var callInvoker = channel.Intercept(metadata =>
{
  metadata.Add("api-key", "<api key>");
  return metadata;
});

var grpcClient = new QdrantGrpcClient(callInvoker);
var client = new QdrantClient(grpcClient);

Working with collections

Once a client has been created, create a new collection

await client.CreateCollectionAsync("my_collection", 
    new VectorParams { Size = 100, Distance = Distance.Cosine });

Insert vectors into a collection

// generate some vectors
var random = new Random();
var points = Enumerable.Range(1, 100).Select(i => new PointStruct
{
  Id = (ulong)i,
  Vectors = Enumerable.Range(1, 100).Select(_ => (float)random.NextDouble()).ToArray(),
  Payload = 
  { 
    ["color"] = "red", 
    ["rand_number"] = i % 10 
  }
}).ToList();

var updateResult = await client.UpsertAsync("my_collection", points);

Search for similar vectors

var queryVector = Enumerable.Range(1, 100).Select(_ => (float)random.NextDouble()).ToArray();

// return the 5 closest points
var points = await client.SearchAsync(
  "my_collection",
  queryVector,
  limit: 5);

Search for similar vectors with filtering condition

// static import Conditions to easily build filtering
using static Qdrant.Client.Grpc.Conditions;

// return the 5 closest points where rand_number >= 3
var points = await _client.SearchAsync(
  "my_collection",
  queryVector,
  filter: Range("rand_number", new Range { Gte = 3 }),
  limit: 5);

qdrant-dotnet's People

Contributors

russcam avatar ivanpleshkov avatar anush008 avatar roji avatar ffuugoo avatar generall avatar azayarni avatar lordhansolo avatar timvisee avatar vaishalid2 avatar

Stargazers

Harald Lapp avatar  avatar Ferry To avatar token avatar 玖亖伍 avatar Dominik Antal avatar André Falk avatar wangchunlong avatar  avatar Luka Radovanovic avatar Wes Doyle avatar Michel Schep avatar Thorsten Sommer avatar  avatar Mike R. avatar Daniel Noel avatar Yizhe Wang avatar LinkinWu avatar  avatar  avatar  avatar Simone Festa avatar Sam Fang avatar  avatar Michał Jagusiak avatar Hubert Wójtowicz avatar Marcin Jakubas avatar  avatar Victor Lee avatar BA2Ops avatar Theodor Dimache avatar tbr09 avatar  avatar Pavel Vařenka avatar  avatar Milkey Tan avatar Konstantin S. avatar Jin avatar Ricardo Groß avatar Ulisses Cappato avatar Emrah Tokalak avatar  avatar Monrage avatar  avatar Vasilev Pyotr avatar  avatar Simon  Birrer avatar  avatar H avatar  avatar Jesper Hansen avatar AOKI Takashi avatar Mohsen Rajabi avatar  avatar Laurent Kempé avatar Pawel Kasperek avatar David Noreña Perez avatar Todd avatar Daniel Blendea avatar Lee Seung Hu avatar Buddhi Adhikari avatar Heron Barreto Peixoto avatar Clay avatar iwaitu avatar Artem Snurnikov avatar Adam Trepka avatar  avatar Marcus Bowyer avatar YC_Lin avatar Michael John Peña avatar  avatar Behzad Khosravifar avatar Marc Mojica avatar Caleb Jenkins avatar Ernesto Herrera avatar  avatar Luis Cossío avatar Luis Merino avatar  avatar Alyona Kavyerina avatar  avatar  avatar  avatar Arjun Shetty avatar Ranjan Dailata avatar  avatar  avatar

Watchers

Fabrizio Schmidt avatar  avatar  avatar  avatar Tara W avatar  avatar

qdrant-dotnet's Issues

Add HTTP API support

While integrating Qdrant in our workflows I've built the Qdrant's HTTP API client.
For the sake of completeness I think it would be great to add an HTTP implementation to this client as well.

My Qdrant HTTP client repo is located here -> https://github.com/aliexpressru/qdrant-client

I will be happy to connect and contribute parts that you find interesting (if any 😊).

Implement trace provider

For cloud native observability, it would be nice to have a tracing story that we could leverage to add to OpenTelemetry.

Switch to class-scoped collection names in the tests

All tests currently use the same Qdrant collection name, and the initialization code (which runs before each test) removes all collections from Qdrant. This makes it impossible to run test classes in parallel (the default xunit behavior).

We can have class-scoped collections to allow for more parallelism.

Discussed in #11 (comment)

Introduce pause in list snapshot tests

It appears that there is eventual consistency between creation of a snapshot, and that snapshot returned in the list of snapshots. This can be seen in intermittent failing list snapshot tests that assert the count of snapshots

[xUnit.net 00:00:19.56]     Qdrant.Client.SnapshotTests.ListSnapshots [FAIL]
  Failed Qdrant.Client.SnapshotTests.ListSnapshots [1 s]
  Error Message:
   Expected snapshotDescriptions to contain 2 item(s), but found 1: {{ "name": "collection_1-3537752039972922-2023-10-28-00-33-41.snapshot", "creationTime": "2023-10-28T00:33:41Z", "size": "67214848" }}.
  Stack Trace:
     at FluentAssertions.Execution.XUnit2TestFramework.Throw(String message)
   at FluentAssertions.Execution.TestFrameworkProvider.Throw(String message)
   at FluentAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(Func`1 failReasonFunc)
   at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args)
   at FluentAssertions.Collections.GenericCollectionAssertions`3.HaveCount(Int[32](https://github.com/qdrant/qdrant-dotnet/actions/runs/6673499640/job/18139357383#step:5:33) expected, String because, Object[] becauseArgs)
   at Qdrant.Client.SnapshotTests.ListSnapshots() in /home/runner/work/qdrant-dotnet/qdrant-dotnet/tests/Qdrant.Client.Tests/SnapshotTests.cs:line 44

Snapshot creation does not have any properties to control whether the call should wait for snapshot creation, so it may be prudent to introduce a small delay before listing snapshots, or to retry the call with delay

public-private keypair in build/keys

The Qdrant.Grpc assembly is signed (strong named) with the public-private keypair in build/keys.

For information on strong named assemblies, see

Publicly published .NET libraries should still be strong named, but the public-private keypair should be checked in, in case users need to build their own versions.

The currently released Qdrant.Grpc package on nuget has been signed with the public-private keypair in build/keys. If Qdrant assumes ownership of this package, the same keypair ought to be used to continue signing the assembly, but if a new package will be released, the public-private keypair should be replaced with a pair of Qdrant's choosing.

RpcException: Error connecting to subchannel.

I am writing a dotnet core app that I run on an AWS linux EC2 instance. (I develop it locally on a Mac.)

I had it working, allowing me to add vectors to a local instance of Qdrant, and then later querying it. I just setup my Qdrant client using this:

public static QdrantClient VectorClient = new QdrantClient("localhost");

But now I'm trying to get this running remotely on AWS linux. So first I setup a new Qdrant instance using Qdrant Cloud, hosted at AWS. I now setup my client like this:

public static QdrantClient VectorClient = new QdrantClient(new Uri("https://my-unique-qdrant-identifier.us-east-1-0.aws.cloud.qdrant.io:6334")), apiKey: "my-api-key");

Now, my querying of the Qdrant database works fine, but when I try to use VectorClient to INSERT new points, I get this error:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException: Connection refused")
 ---> System.Net.Sockets.SocketException (111): Connection refused
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context)
   --- End of inner exception stack trace ---
   at Grpc.Net.Client.Balancer.Internal.ConnectionManager.PickAsync(PickContext context, Boolean waitForReady, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   at Qdrant.Client.QdrantClient.UpsertAsync(String collectionName, IReadOnlyList`1 points, Boolean wait, Nullable`1 ordering, ShardKeySelector shardKeySelector, CancellationToken cancellationToken)
   at DN.MachineLearning.Embeddings.Save(Single[] vectors, String destination, Nullable`1 id, Nullable`1 guid, IDictionary`2 payload) in /home/ec2-user/code/DN/MachineLearning/Embeddings.cs:line 216
   at DN.Indexer.FinishIndexing(String source, DateTimeOffset startTime) in /home/ec2-user/tools/Vectorizer.cs:line 122

I call it using:

await VectorClient.UpsertAsync(destination, new List<PointStruct> { point })

Any ideas what I'm doing wrong?

Can't seem to filter down to results with matching ints

I've got a Qdrant vector database filled vectors and payloads, where the payload contains a "Field1" of long values.

If I run this search without the filter, I get some results, and I can see the payloads do contain the 4686 value as one of the values within Field1's list of values.

var result = await VectorClient.SearchAsync
(
    "mydb", 
    embeddings,
    //filter: Conditions.Match("Field1", new long[] 
    //{
    //    4686
    //}),
    limit: 10
);

But if I uncomment that filter, I get only 1 (really low scored) match, none of the top matches I normally get.

When I define the payload, it looks like this:

var myIntList = new List<int>()
{
    ...
    4686
    ...
};
var payload = new Dictionary<string, Value> 
{ 
    ["Field1"] = new Value 
    { 
        ListValue = new ListValue 
        { 
            Values = { myIntList.Select(s => new Value { IntegerValue = (long) s })
        }
    }
};

Any idea what I'm doing wrong?

Grpc.Core.RpcException Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)

Hi,

I am running dotnet8 on macOS and I have not been able to connect to my Qdrant running on a Docker container over network.

I get this exception on any calls including the following:

var collectionExists = await _client.CollectionExistsAsync("my_collection");
        
        
        await _client.CreateCollectionAsync("my_collection",
            new VectorParams { Size = 100, Distance = Distance.Cosine });

This is the exception:


Grpc.Core.RpcException
Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)")
   at Grpc.Net.Client.Internal.GrpcCall`2.GetResponseHeadersCoreAsync()
   at Qdrant.Client.QdrantClient.CollectionExistsAsync(String collectionName, CancellationToken cancellationToken)

I am certainly sure this only happens in this library as I can access it via a python client.

Can't assign to Payload because it's read-only?

Just starting to experiment with Qdrant, running into something weird.

In my demo, I'm trying to take a dynamic payload in my method params:

public void InsertPayload(float[] vectors, Google.Protobuf.Collections.MapField<string, Value> payload)
{
    ... 

    var point = new PointStruct
    {
        Vectors = vectors,
        Payload = payload
    };

    ...
}

This won't compile, Payload shows this error:

Property or indexer 'PointStruct.Payload' cannot be assigned to -- it is read only

The only way I seem to be able to set the payload when creating a PointStruct is to either do so explicitly, hard-coded:

var point = new PointStruct
{
    Vectors = vectors,
    Payload = 
    {
        { "TestKey", "TestValue" }
    }
};

or by adding string/value pairs manually:

var point = new PointStruct
{
    Vectors = vectors
};

point.Payload.Add("TestKey", "TestValue");

I assume I'm just using the library wrong or misunderstanding how this all works?

Recommend methods does not provide parameter for RecommendStrategy

Below is how RecommedAsync method looks like-

public async Task<IReadOnlyList<ScoredPoint>> RecommendAsync(
		string collectionName,
		IReadOnlyList<PointId> positive,
		IReadOnlyList<PointId>? negative = null,
		ReadOnlyMemory<Vector>? positiveVectors = null,
		ReadOnlyMemory<Vector>? negativeVectors = null,
		Filter? filter = null,
		SearchParams? searchParams = null,
		ulong limit = 10,
		ulong offset = 0,
		WithPayloadSelector? payloadSelector = null,
		WithVectorsSelector? vectorsSelector = null,
		float? scoreThreshold = null,
		string? usingVector = null,
		LookupLocation? lookupFrom = null,
		ReadConsistency? readConsistency = null,
		ShardKeySelector? shardKeySelector = null,
		TimeSpan? timeout = null,
		CancellationToken cancellationToken = default)

There is no option to specify RecommendStrategy in any of the RecommendMethods available in this QdrantClient class.

Consider publishing proto files via a separate git repo

The proto files used to generate gRPC types in SDKs are currently in the main Qdrant repo. This means that SDKs need to have some scripting machinery for download new proto file versions; the protos also need to get checked into the SDK's repo, at which point it's not clear which version they represent etc.

One way to possibly improve management around this is to separate out the proto files into their own repo, which could then be referenced as a git submodule by SDKs (and qdrant itself).

/cc @generall @russcam

Consider reducing the number of TFMs targeted

Qdrant.Grpc currently targets the following TFMs: net462;net471;netstandard2.0;net6.0;net7.0; that's quite a lot, here are some thoughts on possibly reducing that. tl;dr we may want to simply target net462 and net6.0 drop net471 and net7.0`.

  • net471 and net7.0 don't appear to be necessary; that is, I don't cases where specific logic exists for them using new APIs only available for these TFMs. Since later versions of .NET Framework can use net462 and later versions of .NET can use net6.0, it seems we could remove these without affecting any actual use-case.
  • Given that net462 is targeted, AFAICT the only cases where netstandard2.0 would be used would be for users using out-of-support versions of .NET Core (the minimum supported version of .NET Core is net6.0). Users of modern .NET Core generally upgrade to newer versions relatively quickly (as opposed to people still stuck on .NET Framework, where the upgrade to Core can be much more difficult).
  • As a result, we should be able to drop netstandard2.0 and simply target net462 and net6.0, which should simplify things.

@russcam what do you think? Were there any special consideration when choosing the TFM list/

/cc @stephentoub

Geo filter not working

I am using the following code to filter and it does not filter by the GeoRadius. If I filter by key everything goes well

var expected1 = new Filter { Must = { new Condition { Field = new FieldCondition { Key = "Location", GeoRadius = new GeoRadius { Radius = 10000, Center = new GeoPoint { Lon = (double)userPromtInfo.Longitud, Lat = (double)userPromtInfo.Latitud } } } }, } };

In the quadrant documentation the filter is like this

{ "key": "location", "geo_radius": { "center": { "lon": 13.403683, "lat": 52.520711 }, "radius": 1000.0 } }

When you run the code what you get is

"{ ""key"": ""Location"", ""geoRadius"": { ""center"": { ""lon"": -4.4216, ""lat"": 36.7213 }, ""radius"": 10000 } }"

There may be a problem in which one key is geoRadius and the other geo_radius?

do the fucking homeworks: Documentation

Write some decent documentation with code examples and don't leave people to guess your half baked work.

Example:
image

Does not fucking work.
And if you try using it with SearchAsync it's even worse and throws more errors.
Provide working examples of how to use this.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.