Milvus is a scalable open source vector database. What it used for?

Searching data by easily defined criteria, such as querying a movie database by actor, director, genre, or release date, is very simple. A relational database is well suited for these kinds of basic searches using a query language such as SQL. But when the search involves complex objects and more abstract queries, such as searching a streaming video library using natural language or a video clip, simple similarity metrics such as matching words in the title or description are no longer sufficient.

Artificial intelligence (AI) has greatly improved the ability of computers to understand the semantics of language, and has helped humans make sense of vast, hard-to-analyze unstructured data sets (e.g., audio, video, documents, and social media data). AI is enabling Netflix to create sophisticated content recommendation systems, Google users to search the Internet for images, and pharmaceutical companies to discover new drugs.

The problem of searching in large unstructured data sets

These technological advances are achieved by using artificial intelligence algorithms to convert dense unstructured data into vectors, a numerical data format that is easily readable by machines. Additional algorithms are then used to compute the similarity between vectors for a given search. Due to the large volume of unstructured datasets, their complete search is too time consuming for most machine learning applications. To solve this problem, Approximate Nearest Neighbors (ANN) algorithms are used to combine similar vectors into clusters and then search only the part of the dataset that is most likely to contain vectors similar to the target search vector.

This allows for much faster (albeit slightly less accurate) similarity searches and is key to creating useful AI tools. Thanks to vast government resources, it is easier and cheaper than ever to build machine learning applications. However, AI-based vector similarity search often requires the pairing of various tools, the number and complexity of which depend on the specific requirements of the project. Milvus is an open source AI search engine that aims to simplify the process of building machine learning applications by providing robust functionality on a single platform.

What's Milvus?

Milvus is an open source data management platform built specifically to handle massive vector data and optimize machine learning operations (MLOps). Powered by Facebook AI Similarity Search (Faiss), Non-Metric Space Library (NMSLIB) and Annoy, Milvus combines many powerful tools in one place while extending their standalone functionality. The system was specifically designed to store, process, and analyze large vector datasets and can be used to build AI applications spanning computer vision, recommender systems, and more.

Milvus was created to enhance vector similarity search

Milvus is flexible, allowing developers to optimize the platform for specific tasks. CPU/GPU-only and heterogeneous computing support enables faster data processing and optimized resource requirements for any scenario. Data is stored in Milvus on a distributed architecture, making it easy to scale data volumes. With support for different AI models, programming languages (e.g. C++, Java and Python) and processor types (e.g. x86, ARM, GPU, TPU and FPGA) Milvus provides high compatibility with a wide range of hardware and software.