Milvus stores the vectors produced by machine learning models in one or more Milvus collections.
For each collection, the following parameters need to be set:
Collection Name- A name for the Milvus collection you are creating. Milvus requires that collection names only include alphanumeric characters and the underscore
Dimension- The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the
Smart Answers Pre-trained Coldstartmodels output vectors of 512 dimension size. Dimensionality of encoders trained by
Smart Answers Supervised Trainingjob depends on the provided parameters and printed in the training job logs. See Smart Answers Supervised model training for more details.
Index file size- Vectors are added to the raw data files and then indexed. Files larger than this will trigger index building for raw data files.
Metric- The type of metric used to determine how close the search vector is to the vectors in the collection. It is used to calculate the vector similarity scores. In most cases we suggest to use Cosine Similarity which can be obtained by using the
Inner Productmetric over normalized vectors. It produces values between -1 and 1, where a higher value means higher similarity.
The Metric options for Milvus collections are: Euclidean, Inner Product, Hamming, Jaccard, Tanimoto, Substructure, and Superstructure. The following metrics all require that the Dimension is a multiple of 8: Hamming, Jaccard, Tanimoto, Substructure, and Superstructure. Otherwise the Create Milvus Collection job will fail. See Milvus metrics documentation for more details.
Milvus can be configured to pre-load one or more collections via the Milvus parameter
preload_collection. Fusion does not currently ship with a method to manage this parameter, but the Milvus API can be used directly (after first setting up port forwarding via
kubectl port-forward milvus-writeable-<hash>-<random> 19121:19121).
For an example of how to create and utilize a Milvus collection, see Configure The Smart Answers Pipelines.
When a Milvus collection is large, creating an index can speed up the similarities search. If you do not create an index on a collection then a FLAT index is used. See Milvus index documentation for more details.
When you create an index on a collection, in addition to the
collection name you will need to specify:
Index Type- The index type must be one of [
RNSG]. We recommend you use the Fusion job default value of
Index Parameters- Parameters used to create an index in Milvus. These required parameters are specific to the chosen
Index Type. For example,
nlist=4096for IVF indexes or
efConstruction=500for HNSW indexes.
|If you create an index for the collection different from FLAT, index specific query parameters have to be added as Search Parameters in the Milvus Query stage. Refer to the Milvus supported indexes for query parameters for each index type.|
Milvus management jobs
For more information about using these stages see Smart Answers Detailed Pipeline Setup.
The Encode into Milvus index pipeline stage invokes a machine learning model to encode a field and store it into Milvus.
Query pipeline stages: