The Elasticsearch vector retrieval plug-in is developed by Baidu Elasticsearch team, and can quickly realize vector retrieval, vector computation and other requirements.

Background

In recent years, the vector retrievals based on Text (Document) Embedding, eigenvector, etc. are widely applied in the similarity retrieval of recommended systems and pictures. The user can use Word2vec and other tools to map the image, audio, natural language and other complex data information to eigenvectors, then retrieve the eigenvectors by the vector retrieval algorithm to realize the handling of complex data information. To process the vector data, Baidu Elasticsearch vector retrieval plug-in provides two vector retrieval algorithms: Linear algorithm and hnsw algorithm.

Cluster Preparation

Method of Application

Before writing data, the user should configure the knn parameters according to the vector dimension information and performance requirements of business, select the distance computation algorithm, and create the required knn index. You can write data after building the index. After building the index, you can conduct the vector retrieval query by the query mode provided below.

Create the knn index

We should create the knn index in advance by the following method:

As shown in the following example below, we create an index with the name of test-index, including field1 and field2. You can also customize the index name and field name according to your own requirements.

PUT /test-index 
{ 
    "settings": { 
      "index": { 
         "codec": "bpack_knn_hnsw", 
         "bpack.knn.hnsw.space": "cosine", 
         "bpack.knn.hnsw.m": 16,
         "bpack.knn.hnsw.ef_construction": 512 
      } 
   }, 
   "mappings": { 
      "properties": { 
         "field1": { 
            "type": "bpack_vector", 
            "dims": 2 
         }, 
         "field2": { 
            "type": "bpack_knn_vector", 
            "dims": 2 
         } 
      } 
   } 
} 

The bpack.knn.hnsw parameter meaning in settings is as shown in the index level parameter optimization below.

Write and query data

Write data

We write data in the _doc of index test-index we created just now, and the example of writing data is as below:

POST /test-index/_doc/ 
{ 
    "field1" : [6.5, 2.5], 
    "field2" : [6.5, 2.5], 
    "price" : 10 
} 

And field1 is the field of bpack_vector type we just set. field2 is the field of bpack_knn_vector type we just set. price stands for other common fields.

After building an index, we can query the data as below:

linear query

The linear algorithm can query the field of bpack_knn_vector as well as that of bpack_vector type. In the following example, we query the field field1 of bpack_ vector.

POST /test-index/_search 
{ 
   "query": { 
      "script_score": { 
         "query": { 
            "match_all": {} 
         }, 
         "script": { 
            "source": "bpack_knn_script", 
            "lang": "knn", 
            "params": { 
               "space": "cosine", 
               "field": "field1", 
               "vector": [3.5, 2.5] 
            } 
         } 
      } 
   }, 
   "size": 100 
} 
Or 
POST /test-index/_search 
{ 
  "query": { 
    "function_score": { 
      "boost_mode": "replace", 
      "script_score": { 
        "script": { 
          "source": "bpack_knn_script", 
          "lang": "knn", 
          "params": { 
            "space": "cosine", 
            "field": "field1", 
            "vector": [3.5, 2.5] 
          } 
        } 
      } 
    } 
  }, 
  "size": 100 
} 

And the query parameter means:

hnsw query

When we use hnsw for query. The index must specify index.codec as bpack_knn_hnsw. Meanwhile, the type specified by the vector field mapping to be queried must be bpack_ knn_ hnsw. In the following example, we query the field field2 of bpack_knn_ vector.

POST /test-index/_search 
{ 
    "size" : 10, 
    "query": { 
        "knn": { 
            "field2": { 
                "vector": [3, 4], 
                "k": 2, 
                "ef": 512 
            } 
        } 
    } 
} 

And the query parameter means:

Parameter Optimization

Index level parameter

The index settings parameter must be provided when an index is created. The default value is used when these settings are not provided. These settings are static. This means that you can't modify the created indexes. The specific parameter analysis is as below:

Cluster level parameters

Common parameters

Cache settings

Settings of cache parameters of linear algorithm

Settings of cache parameters of hnsw algorithm

Settings of Circuit Breaker

The hnsw algorithm consumes a lot of out-of-core memory. If the consumed memory is too much, the pagecache which can be used by Elasticsearch/Lucene is insufficient, and the cluster performance declines. To avoid this situation, we can configure Circuit Breaker to limit the excessive consumption of out-of-core memory. Currently, when the memory reaches the breaker limit we configure, the eviction mechanism is triggered to trigger the cache items which are uncommonly used.

Example

PUT /_cluster/settings 
{ 
    "persistent" : { 
        "bpack.knn.hnsw.index_thread_qty" : 1, 
        "bpack.knn.cache.item.expiry.time": "15m", 
        "bpack.knn.memory.cache.limit": "1g", 
        "bpack.knn.memory.cache.expiry.time":"10m", 
        "bpack.knn.memory.circuit_breaker.limit" : "55%", 
        "bpack.knn.circuit_breaker.unset.percentage": 23 
    } 
} 

View the related statistical information of hnsw algorithm

The method to query the status is as below:

GET /_bpack/_knn/stats 
GET /_bpack/_knn/nodeId1,nodeId2/stats/statName1,statName2 

The result example is as below:

{ 
   "_nodes": { 
      "total": 1, 
      "successful": 1, 
      "failed": 0 
   }, 
   "cluster_name": "my-application",
   "circuit_breaker_triggered": false, 
   "nodes": { 
      "HYMrXXsBSamUkcAjhjeN0w: { 
         "eviction_count" : 0,
         "miss_count" : 1,
         "graph_memory_usage_kb" : 1,
         "cache_capacity_reached" : false, 
         "load_exception_count" : 0,
         "hit_count" : 0,
         "load_success_count" : 1,
         "total_load_time_nanos" : 2878745
      } 
   } 
} 

Cluster status parameter:

Node status parameter:

Performance Comparison

• Memory configuration: 30G
• cpu Configuration: Number of logical cores: 56, 2 physical cpus, each cpu cores: 14
• Elasticsearch Node: Single node

The performance comparison results are as below:

Algorithm Summary

• Applicable scenarios of linear algorithm:

  • Small data volume (the single segmentation is usually below 100w);
  • First execute the normal search filter condition, and then conduct vector retrieval computation on the filtered result set.
  • The recall rate is 100%, and the query performance is slower compared with hnsw.

• Applicable scenarios of hnsw algorithm:

  • The data volume is relatively large (cluster data volume is at the level of tens of millions)
  • The vector retrieval computation and other filterings are conducted simultaneously. It is recommended to appropriately increase the query parameter k of hnsw to guarantee that the data satisfying filtering conditions can be involved in the computation;
  • The query performance requirement is high, and the recall rate is 90% above.

Best Practices

• It is recommend to conduct regularly forceMerge in the business low peak period after writing to reduce the query delay.
• When using the linear algorithm for query, you should define the "bpack.knn.memory.cache.limit" parameter according to the data volume. For example, if the node data volume is 10G, and the default value of "bpack.knn.memory.cache.limit" is used (the default value of computation 2 type is 30G*10%=3G), the cache is unavailable. The bulk query may trigger the fusing operation of Elasticsearch, and the error circuitBreakingException is reported.

• When a vector index is built for a larger data volume, the building may be slow. You can appropriately adjust the "bpack.knn.hnsw.index_thread_qty" before writing data according to the partitions and node cpu cores. For example, 1kw data, 1 node and 2 partitions, and 16-core cpu for the node. We can set "bpack.knn.hnsw.index_thread_qty" as 4-6 (if we set it as 8, the cpu can be fully loaded, and the production environment may be in risk) to improve the building efficiency.

  It should be noted that a higher "bpack.knn.hnsw.index_thread_qty" parameter set may cause excessive start threads in the building. In a cluster with a higher load, it is not recommend to adjust this parameter to avoid full load of the cluster. If it is slow to write and build the vector, you can quicken building by temporarily reducing the cluster loads (reduce other writes and queries) and enlarging "bpack.knn.hnsw.index_thread_qty", and then adjust "bpack.knn.hnsw.index_thread_qty" to 1 after building.

• When the written data volume is 1kw (for example about 10G), 1 node 1 partition, and computation 2 type node (16-core cpu and 64G memory), it is recommend to set the parameter as:

  PUT /_cluster/settings 
  { 
      "persistent" : { 
          "bpack.knn.hnsw.index_thread_qty" : 1, 
          "bpack.knn.cache.item.expiry.time": "1h", 
          "bpack.knn.memory.cache.limit": "12g", 
          "bpack.knn.memory.cache.expiry.time":"1h", 
          "bpack.knn.memory.circuit_breaker.limit" : "70%" 
      } 
  } 

Analysis:

1. "bpack.knn.hnsw.index_thread_qty" : 1： Generally, it is recommended to set it as 1; when the index building is too slow, you may appropriately adjust this parameter by reference to the recommendations above.
2. "bpack.knn.cache.item.expiry.time": "1h"： You can set the timeout according to your own business.
3. "bpack.knn.memory.cache.limit": "12g"： The data volume is about 10G. The cache should accommodate all the data.
4. "bpack.knn.memory.cache.expiry.time":"1h"： You can set the timeout according to your own business.
5. "bpack.knn.memory.circuit_breaker.limit" : "70%"： The default jvm memory of computation 2 type Elasticsearch is 30G. The "bpack.knn.memory.circuit_breaker.limit" is 70%*(64-30)=23.8G, it can accommodate the out-of-core memory occupied by data.

FAQ

• Q： How is the recall rate defined?

  A: Use the same vector to query two query mode. Compare the recalled documents, and get the ratio of two identical documents and recall documents. Now we can get the recall rate of the vector to be measured. We use the recall rate to characterize the accuracy rate of query.

• Q： Why don't the indexed documents increase or completely reach the writing volume, and may the query fail when the writing has been successful?

  A: The vector index is built in the refresh or flush period. Although the writing is completed, the vector index building tasks at the background may still continue.

• Q： How to install a vector retrieval plug-in?

  A: The newly applied 7.4.2 cluster has its own vector retrieval plug-in; if you already install the vector retrieval plug-in, you can contact the customer service personnel to assist installation.