Fast Elasticsearch Vector Scoring

This Plugin allows you to score Elasticsearch documents based on embedding-vectors, using dot-product or cosine-similarity.

General

Updated version for ES 6.1 of this plugin
Cosine support removed.
This plugin was inspired from This elasticsearch vector scoring plugin and this discussion to achieve 10 times faster processing over the original.
lior-k gained this substantial speed improvement by using the lucene index directly
lior-k developed it for their workplace which needs to pick KNN from a set of ~4M vectors. Their current ES setup is able to answer this in ~80ms

Elasticsearch version

Currently designed for Elasticsearch 6.1.2.

Setup

In order to install this plugin, you need to create a zip distribution first by running

gradle clean assemble

This will produce a zip file in build/distributions.

After building the zip file, you can install it like this

elasticsearch-plugin install file:///path/to/iplugin/build/distribution/FILENAME.zip

Debugging

Place this into an elasticsearch checkout, add the plugin to the projects list in /settings.gradle and run

gradle :plugins:vector-scoring:run --debug-jvm

Usage

Documents

Each document you score should have a field containing the base64 representation of your vector. for example:

   {
   	"id": 1,
   	....
   	"content_vector": "v7l48eAAAAA/s4VHwAAAAD+R7I5AAAAAv8MBMAAAAAA/yEI3AAAAAL/IWkeAAAAAv7s480AAAAC/v6DUgAAAAL+wJi0gAAAAP76VqUAAAAC/sL1ZYAAAAL/dyq/gAAAAP62FVcAAAAC/tQRvYAAAAL+j6ycAAAAAP6v1KcAAAAC/bN5hQAAAAL+u9ItAAAAAP4ckTsAAAAC/pmkjYAAAAD+cYpwAAAAAP5renEAAAAC/qY0HQAAAAD+wyYGgAAAAP5WrCcAAAAA/qzjTQAAAAD++LBzAAAAAP49wNKAAAAC/vu/aIAAAAD+hqXfAAAAAP4FfNCAAAAA/pjC64AAAAL+qwT2gAAAAv6S3OGAAAAC/gfMtgAAAAD/If5ZAAAAAP5mcXOAAAAC/xYAU4AAAAL+2nlfAAAAAP7sCXOAAAAA/petBIAAAAD9soYnAAAAAv5R7X+AAAAC/pgM/IAAAAL+ojI/gAAAAP2gPz2AAAAA/3FonoAAAAL/IHg1AAAAAv6p1SmAAAAA/tvKlQAAAAD/I2OMAAAAAP3FBiCAAAAA/wEd8IAAAAL94wI9AAAAAP2Y1IIAAAAA/rnS4wAAAAL9vriVgAAAAv1QxoCAAAAC/1/qu4AAAAL+inZFAAAAAv7aGA+AAAAA/lqYVYAAAAD+kNP0AAAAAP730BiAAAAA="
   }

Use this field mapping:

PUT my_index
{
  "mappings": {
    "doc": {
      "properties": {
        "embedding_vector": {
          "type": "binary",
          "doc_values": true
        }
      }
    }
  }
}

The vector can be of any dimension

Querying

For querying the 100 KNN documents use this POST message on your ES index:

POST /_search
{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "name": "Doe"
        }
      },
      "functions": [
        {
          "script_score": {
            "script": {
              "source": "vector_scoring",
              "lang": "binary_vector_score",
              "params": {
                "vector_field": "content_vector",
                "vector": [
                  -0.09217305481433868,
                  0.010635560378432274,
                  -0.02878434956073761,
                  0.06988169997930527,
                  0.1273992955684662,
                  -0.023723633959889412,
                  0.05490724742412567,
                  -0.12124507874250412,
                  -0.023694118484854698,
                  0.014595639891922474,
                  0.1471538096666336,
                  0.044936809688806534,
                  -0.02795785665512085,
                  -0.05665992572903633,
                  -0.2441125512123108,
                  0.2755320072174072,
                  0.11451690644025803,
                  0.20242854952812195,
                  -0.1387604922056198,
                  0.05219579488039017,
                  0.1145530641078949,
                  0.09967200458049774,
                  0.2161576747894287,
                  0.06157230958342552,
                  0.10350126028060913,
                  0.20387393236160278,
                  0.1367097795009613,
                  0.02070528082549572,
                  0.19238869845867157,
                  0.059613026678562164,
                  0.014012521132826805,
                  0.16701748967170715,
                  0.04985826835036278,
                  -0.10990987718105316,
                  -0.12032567709684372,
                  -0.1450948715209961,
                  0.13585780560970306,
                  0.037511035799980164,
                  0.04251480475068092,
                  0.10693439096212387,
                  -0.08861573040485382,
                  -0.07457160204648972,
                  0.0549330934882164,
                  0.19136285781860352,
                  0.03346432000398636,
                  -0.03652812913060188,
                  -0.1902569830417633,
                  0.03250952064990997,
                  -0.3061246871948242,
                  0.05219300463795662,
                  -0.07879918068647385,
                  0.1403723508119583,
                  -0.08893408626317978,
                  -0.24330253899097443,
                  -0.07105310261249542,
                  -0.18161986768245697,
                  0.15501035749912262,
                  -0.216160386800766,
                  -0.06377710402011871,
                  -0.07671763002872467,
                  0.05360138416290283,
                  -0.052845533937215805,
                  -0.02905619889497757,
                  0.08279753476381302
                ]
              }
            }
          }
        }
      ]
    }
  }
}

The example above shows a vector of 64 dimensions
Parameters:
1. field_vector: The field containing the base64 vector.
2. vector: The vector (comma separated) to compare to.

muhleder / elasticsearch-vector-scoring Goto Github PK

elasticsearch-vector-scoring's Introduction

Fast Elasticsearch Vector Scoring

General

Elasticsearch version

Setup

Debugging

Usage

Documents

Querying

elasticsearch-vector-scoring's People

Stargazers

Watchers

Forkers

elasticsearch-vector-scoring's Issues

Recommend Projects

Recommend Topics

Recommend Org