Milvus in 2023: A year in review of open source vector databases

Last year marked a turning point in artificial intelligence (AI). Large Language Models (LLMs) took center stage, becoming widely recognized for their exceptional natural language processing capabilities. This surge in popularity has greatly expanded the possibilities of machine learning applications, allowing developers to create more intelligent and interactive applications.

Against the backdrop of this revolution, vector databases have become a critical component that serves as a long-term memory for LLMs. The emergence of search-enhanced generation (RAG) models, intelligent agents, and multimodal search applications has demonstrated the enormous potential of vector databases in improving the efficiency of multimodal data retrieval, reducing hallucinations in LLM, and augmenting domain knowledge.

The evolution of LLM has also catalyzed significant progress in embedding technologies. According to the Massive Text Embedding Benchmark (MTEB) leaderboard on the Hugging Face website, leading embedding models such as UAE, VoyageAI, CohereV3, and Bge were released in 2023. These advances have improved the performance of vector search in various vector search technologies such as Milvus, providing more accurate and efficient processing capabilities for AI applications.

However, with the growing popularity of vector databases, there was a debate about the need for specialized solutions. New startups have entered the vector database arena. Many traditional relational and NoSQL databases have begun to consider vectors as an important data type, and many claim to be able to replace specialized vector databases in any situation.

Launching for the first time in 2019, Milvus

No downtime during rolling updates

Many application developers also pay less attention to the stability of vector databases than transactional databases because their applications are often in the early stages of development. However, stability becomes indispensable if you intend to deploy your AIGC application in a production environment and achieve the best user experience.

Milvus prioritizes not only functionality but also stability by adding rolling updates to Milvus starting with version 2.2.3. After constant tweaking, this feature ensures that there is no downtime during updates without interrupting business processes.

3x Improved performance in production environments

Improving vector search performance should be a major challenge for vector databases. Many vector search solutions have chosen to rely on adapting the Hierarchical Navigable Small Worlds (HNSW) algorithm in order to get to market quickly. Unfortunately, this means that they face serious challenges in real-world production environments, especially for searches with high filtering rates (over 90%) and frequent data deletion.

Milvus focused on performance optimization throughout all phases of development, especially in production environments, achieving a three-fold improvement in search performance, especially in filtered and streaming insertion/search situations.

We also introduced VectorDBBench, an open-source benchmarking tool to make it easier for developers to evaluate vector databases in various environments. Unlike traditional evaluation methods, VectorDBBench evaluates databases using real data, including ultra-large datasets or data that closely resembles data from real-world embedding models, providing users with deeper information to make informed decisions.

Improved memorization rate by 5% on the Beir dataset

While dense embeddings have proven their effectiveness in vector search, they have to catch up with us when searching for names, objects, acronyms, and short query contexts. In response to their shortcomings, Milvus introduced a hybrid query approach that combines dense embeddings with sparse embeddings to improve the quality of search results. This hybrid solution with a ranking model resulted in a 5% improvement in recall on the Beir dataset, which was validated by our tests.

Milvus also introduced a graph-based search solution adapted for sparse embeddings, outperforming conventional search algorithms such as WAND.

At the NeurIPS BigANN 2023 competition, Zilliz engineer Zihao Wang presented Pyanns

10-fold memory saving when working with large data sets

In 2023, the most popular use case for vector databases will be search with extension (RAG). However, the increasing volume of vector data in RAG applications creates a data storage problem for these applications. This problem is particularly acute when the volume of transformed vectors exceeds the size of the original document chunks, potentially increasing memory utilization costs. For example, after splitting documents into chunks, the size of a 1536-dimensional float32 vector (about 3 kb) converted from a 500-token chunk (about 1 kb) exceeds the size of a 500-token chunk.

Milvus is the first open source vector database to support on-disk indexing, providing 5x memory savings. By the end of 2023, Milvus 2.3.4 will include the ability to load scalar and vector data/indexes to disk using memory-mapped (MMap) files. This advancement provides more than a 10x reduction in memory usage compared to traditional in-memory indexing.

20 Milvus releases

We have 20 releases in 2023, which is a testament to the dedication of over 300 community developers and our commitment to a user-centered development approach.

For example, Milvus 2.2.9 introduced dynamic schema, marking a shift from prioritizing performance to improving usability. Milvus 2.3 introduces important features such as upsert, range searching

A million tenants in one Custer

Realization of multi-user relationships

Conversely, enterprises working with tens of thousands of tenants may benefit from a more fine-grained strategy that involves isolating physical resources. The latest version of Milvus 2.3.4 improves memory management, coroutine handling, and CPU optimization, making it easier to create tens of thousands of tables within a single cluster. This enhancement also addresses the needs of B2B companies by increasing efficiency and control.

10 million downloads of Docker images

By the end of 2023, Milvus has reached 10 million Docker pull downloads. This achievement demonstrates the increased developer interest in Milvus and emphasizes its growing importance in the vector database field. As a cloud vector database, Milvus integrates seamlessly with Kubernetes and the broader container ecosystem.

Beyond Numbers: A new look at vector databases

Beyond numerical milestones, 2023 has enriched us with valuable insights into the subtle nuances and dynamics of vector search technology.

Vector databases are on the road to diversification

Similar to the evolution of databases into categories such as online transaction processing (OLTP), online analytical processing (OLAP), and NoSQL, vector databases show a clear trend toward diversification. In contrast to the traditional focus on online services, offline analytics has developed significantly. Another prominent example of this shift is the emergence of GPTCache, an open source semantic cache released in 2023. It improves the efficiency and speed of GPT-based applications by storing and retrieving responses generated by language models.

Elastic scalability is essential for native AI applications.

The exponential growth of artificial intelligence applications, exemplified by ChatGPT, which has amassed over 100 million monthly active users in two months, is likely to surpass all previous business trajectories. Rapid scaling from 1 million to 1 billion data points becomes paramount once companies reach their peak growth. AI application developers benefit from the pay-as-you-go service model of LLM vendors, resulting in significant reductions in operational costs. Similarly, data warehousing consistent with this pricing model is proving to be beneficial for developers, allowing them to focus more on their core business.

Unlike language models (LLMs) and various other technological systems, vector databases operate in stateful mode, requiring persistent data storage. Consequently, elasticity and scalability should be prioritized in the selection of vector databases to meet the dynamic requirements of evolving AI applications.

Machine learning in vector databases can produce unusual results

In 2023, our significant investment in AI4DB (Artificial Intelligence for Databases) projects brought notable success. As part of our efforts, we implemented two critical capabilities in Zilliz Cloud, a fully managed Milvus solution: 1) AutoIndex, an auto-tuned index based on machine learning, and 2) a data partitioning strategy based on data clustering. Both solutions have played a crucial role in significantly improving search performance in Zilliz Cloud.

Open Source vs. Closed Source

However, in vector databases, users will ultimately favor open source. There are many benefits to choosing open source, including more diverse use cases, faster iteration, and a more robust ecosystem.

In addition, database systems are so complex that they cannot afford the opacity often associated with LLM. Users must thoroughly understand the database before choosing the most sensible approach to using it. Moreover, the transparency inherent in open source allows users to customize the database to suit their needs.

An epilogue and a new beginning

It's exciting to see the innovation of many AI startups founded in 2023. It reminds me of why I got into VectorDB development in the first place. In 2024, all these innovative applications will gain real popularity, attract not only funding but also real paying customers who will make different demands on developers.

We look forward to seeing even more diverse applications and system developments in vector databases in the coming year.