3 important AI/ML tools that can be deployed on Kubernetes

Organizations know how important it is to move their entire application stack to Kubernetes. Artificial intelligence is next in line.

In the world of infrastructure technology, things happen fast. It wasn't that long ago that running a database on Kubernetes was considered too complicated to be worth doing. But that was a problem of yesterday. Cloud application builders have learned how to deal with government workloads because Kubernetes is a powerful way to build virtual data centers quickly and efficiently.

Last time I wrote about this, I expanded the aperture a bit and looked at other parts of the application stack in a virtual data center - specifically streaming workloads and analytics.

With the introduction of these two tools to Kubernetes, the discussion of use cases becomes more and more interesting. What will we do with these fundamental data tools if we get access to them?

Fortunately, we don't need to explore this issue too deeply because the industry has already chosen a direction: AI/ML workloads. At the heart of this is the need for faster and more flexible MLOps to support online predictive analytics, otherwise known as real-time artificial intelligence (AI). Companies like Uber and Netflix have been early adopters of this direction, but there are plenty of great projects to help you get up to speed with Kubernetes.

The specialty of serving with a feast

Creating and maintaining machine learning (ML) models is increasingly moving out of the back office and closer to users in production. The feature store acts as a bridge between data and machine learning models, providing consistent access for models to data in both offline and interactive modes. It manages the data processing requirements during model training and provides real-time, low-latency access to models in the online phase. This ensures data consistency in both phases and satisfies both online and offline requirements. Feast is an example of a feature store running in Kubernetes.

It is open source and allows organizations to consistently store and provide attributes for offline training and online inference. Feast goes beyond traditional databases by providing specialized capabilities such as point-in-time correctness.

Model service with KServe

KServe is an API endpoint for deploying machine learning models to Kubernetes that manages the fetching and loading of models and determines which processor or GPU is required. It integrates with KNative eventing for scaling and offers observability features such as metrics and logging.

The best part? It's easy to use. Just point KServe to a model file and it creates an API and does the rest. The explanation function provides insight into why a particular decision was made for each forecast, suggesting the meaning of the features and highlighting the factors in the model that led to a particular outcome.

This can be used to detect model drift and bias, which are some of the "important but difficult" parts of machine learning. These capabilities can reduce the labor-intensive nature of MLOps and increase confidence in the application. KServe recently spun off from Google's KubeFlow project and was highlighted by Bloomberg as part of an effort to build a platform for ML inference.

Vector similarity search

Complementing traditional data mining, vector similarity search (VSS) is a machine learning tool that uses vector math to determine how "close" two objects are to each other. It does this by using the K-nearest neighbor (KNN) algorithm, which expresses the data as a vector.

The data is then vectorized using the processor-intensive KNN algorithm and indexed for searching with less CPU overhead. End users can specify a vector and find nearby objects using the query mechanism provided by VSS servers. Open source VSS servers that can be deployed in Kubernetes include Weaviate and Milvus. Both provide everything you need to add similarity search to your application stack.

Team building

Combine my previous article with this one and you have a recipe for a full stack deployed in Kubernetes. The outcome every organization should strive for is increased productivity and reduced costs. Recent studies show that data leaders are finding both when deploying data infrastructure in Kubernetes.

AI/ML workloads are probably something you're just starting to learn, so now is a good time to start off on the right foot. The three areas mentioned - feature maintenance, model maintenance, and vector similarity search - are covered in a book I co-authored with Jeff Carpenter, Managing Cloud Data with Kubernetes. The big picture of applying AI/ML to the application stack: Real-time requirements will soon dominate most AI applications. Using Kubernetes for fast and reliable builds is no longer an AI hallucination.