Vector databases you can choose from

Vector databases provide efficiency and scalability and are changing the way we utilize the potential of data embedding in the digital age. There are quite a few open source vector databases that have their own advantages. We will review them briefly.

In Natural Language Processing (NLP), embedding is the representation of text as vectors. The purpose of embedding is to convey the semantic meaning of words or documents in a way that can be understood by a machine learning model.

A vector database (or embedding database) in NLP is a specialized database designed to efficiently store, retrieve, and perform operations on high-dimensional vector data (such as the embeddings mentioned above). Vector databases are optimized to efficiently perform nearest neighbor search operations, which is a common requirement in NLP applications. They provide a way to organize and search large amounts of embedding data, which can be useful in various tasks such as information retrieval, document similarity, clustering, and others.

As an example, let's say you have nested a large number of documents using the Doc2Vec model. Now, having received a new document, you want to find the most similar documents in your database. To do this, you need to:

1. First, nest the new document in the same high-dimensional space.
2. Next, in the vector database, we search for vectors that are closest to the vector of the new document. This is the nearest neighbors search.

Due to the high dimensionality of the data, such searches can be computationally intensive. However, vector databases use specialized indexing and query algorithms (e.g., k-d trees, ball trees, or hashing methods) to speed up these operations. Examples of such databases are FAISS, developed by Facebook AI, and Annoy, developed by Spotify.

Open source vector databases

Weaviate :

This open source vector database allows storing and retrieving data objects based on their semantic properties using vector indexing.

• It supports various media types including text, images, etc., and offers features such as semantic search, question and answer extraction, classification, and customizable models.
• It provides GraphQL API for easy data access and is optimized for high performance, as proven by open source benchmarks.
• Key features include fast queries, support for multiple media types via modules, combined vector and scalar search, real-time and persistent data access, horizontal scalability, and graph-like relationships between objects.
• Weaviate is recommended for improving search quality, performing similarity search with machine learning (ML) models, efficiently combining vector and scalar search, scaling ML models for production, and performing fast classification tasks.
• It finds applications in semantic search, image search, anomaly detection, recommender systems, e-commerce search, data classification, cyber threat analysis, etc.

Pgvector :

It is an open-source PostgreSQL extension that enables vector similarity search in a database. It allows efficient storage and querying of high-dimensional vectors, making it suitable for applications such as recommender systems, natural language processing, and image analysis.

• pgvector offers functions and operators for vector similarity search, such as finding nearest neighbors based on distances between vectors or performing similarity joins between vectors. The search process is optimized using index structures such as K-d trees or Annoy.
• With pgvector, vector data can be stored directly in PostgreSQL tables, making it easy to integrate vector similarity search into existing database workflows. It also provides indexing and query support for multiple vector fields in a single table.
• The extension is implemented in C and supports many vector data types such as float4, float8 and integer. It offers a simple SQL interface for vector operations and can be easily integrated with other PostgreSQL features and extensions.
• pgvector extends PostgreSQL by adding vector similarity search functionality, allowing developers to perform efficient and scalable similarity searches on high-dimensional vector data directly in the database.

Chroma DB :

Chroma is an open source database for embedding. It facilitates the development of LLM applications by allowing knowledge, facts and skills to be embedded in LLMs.

Chroma provides the means to:

• Store embeddings and associated metadata
• Inserting documents and requests
• Search for embedded content

Milvus

Milvus was developed in 2019 to store, index, and manage massive embedding vectors generated by deep neural networks and other ML models.

• It is designed to handle queries on vector data and is capable of indexing vectors on a trillion-dollar scale. Milvus can handle embedding vectors derived from unstructured data, unlike relational databases that work with structured data.
• Unstructured data such as emails, documents, IoT sensor data, photos, and protein structures are increasingly common on the Internet. Milvus stores and indexes these vectors, allowing computers to interpret unstructured data.
• Milvus is able to analyze the correlation between two vectors by calculating their similarity distance, which indicates the similarity of the original data if the vectors are extremely similar.

QDrant :

This open source vector database is designed for fast and scalable storage and retrieval of high-dimensional data.

• It uses advanced indexing techniques, including approximate nearest neighbors (ANN) and product quantization, for efficient data search and retrieval.
• QDrant supports CPU and GPU-based computing, providing flexibility and adaptation to different hardware configurations.
• The database is highly scalable and can handle large data volumes and high user parallelism.
• A unique feature of QDrant is its ability to store and search geospatial data, making it well suited for location-based applications.

Each of these vector databases has its own strengths and applications. Weaviate stands out for its semantic search capabilities and support for various media types. Pgvector integrates seamlessly with PostgreSQL and provides efficient vector similarity search. Chroma DB focuses on storing and searching embedded data for LLM applications. Milvus specializes in working with massive embedding vectors and unstructured data. QDrant is characterized by fast and scalable storage and retrieval of high-dimensional data, as well as support for geospatial data. The choice of database depends on the specific requirements and usage scenarios of the application.