What is unstructured data?

Examining the intricacies of unstructured data and how to process, analyze, and form queries.

Our world is constantly evolving digitally, and the amount of data is growing exponentially every second. The development of artificial intelligence technologies has only accelerated this process. However, not all data is the same. Shockingly, 80% of the new data generated is unstructured. This proportion is expected to increase as industries and technologies evolve. Most importantly, unstructured data is abundant and is a valuable source of rich information that can help inform business decisions.

So, what is unstructured data and how does it differ from structured and semi-structured data? How to effectively process, analyze and search for information in unstructured data? In this blog, we will explore the intricacies of unstructured data and discuss methods for processing, analyzing and searching it.

Structured data vs. Unstructured data vs. Semi-structured data

First, let's familiarize ourselves with the different types of data - structured, semi-structured, and unstructured.

Structured data

Structured data has a defined format that makes it easier to store and analyze using traditional data management tools such as SQL. Examples of structured data are customer information, transaction records, and inventory lists.

Semi-structured data

Semi-structured or partially structured data is a mixture of structured and unstructured data. They contain some level of organization, such as metadata or tags, but are not fully structured. Semi-structured data is typically found in XML files, JSON documents, and other types of data that obey a particular schema. Such data is usually stored in NoSQL databases, such as wide-column stores or object-document databases, because it cannot be directly stored in a relational database.

Unstructured data

Unstructured data is data that does not have a specific format or structure. This type of data is often created by humans in the form of text, images, videos, emails, and social media posts. However, unstructured data can also include less common examples such as protein structures, executable file hashes, human-readable code, etc. - the possibilities are endless.

The following are specific examples of unstructured data of both machine and human origin.

• Sensor data: Data collected from various sensors, including temperature, humidity, GPS, and motion sensors.
• Machine Log Data: Data generated by machines, devices, or applications, including system logs, application logs, and event logs.
• Internet of Things (IoT) data: Data collected from smart devices, including smart thermostats, home assistants, and wearable devices.
• Computer vision data: Data generated by computer vision technologies such as image recognition, object detection and video analysis.
• Natural Language Processing (NLP) data: Data generated by NLP technologies such as speech recognition, language translation, and sentiment analysis.
• Web server and application data: Data generated by web servers, web applications, and mobile applications, including user behavior data, error logs, and application performance data.
• Emails: Email messages typically contain unstructured text, images, and attachments.
• Text messages: Text messages can be informal, unstructured, and contain abbreviations or emoji.
• Social media posts: Social media posts may vary in structure and content, including text, images, videos and hashtags.
• Audio recordings: Human-generated audio recordings can include phone calls, voicemails, audio files and audio notes. They are considered unstructured data.
• Handwritten notes: Handwritten notes can be unstructured and contain pictures, diagrams, and other visual elements.
• Meeting notes: Meeting notes can contain unstructured text, diagrams and action items.
• Transcripts: Transcripts of speeches, interviews and meetings may contain unstructured text with varying degrees of accuracy.
• User-generated content: User-generated content on websites and forums can be unstructured data, including free-form text, images and video files.

Analyzing unstructured data is challenging

Working with unstructured data can be difficult because it lacks a standardized format. In addition, queries and data analysis become more complex, especially when compared to structured and semi-structured data.

When working with structured or semi-structured data, searching or filtering certain items in a database is easy. For example, to get the first book of a certain author in MongoDB, you can use the following code fragment (using pymongo).

   >>> document = collection.find_one({'Author':'Bill Bryson'})

This query methodology is similar to traditional relational databases that filter and retrieve data using SQL statements. The basic idea is the same: databases built for structured or semi-structured data perform filtering and querying using mathematical (such as <=, row spacing) or logical (EQUALS, NOT) operators over numeric values and strings. For traditional relational databases, this is called relational algebra. This is why they always return exact matches for a given set of filters.

However, traditional relational databases and data management tools cannot handle the complexities of analyzing unstructured data. For example, if a user wants to find similar shoes based on a collection of shoe photos taken from different angles, a relational database will not be able to capture the nuances of style, size, color, etc. of the shoes based only on the raw pixel values of those images. This poses a serious problem for industries and companies that utilize unstructured data: How to transform, store, and similarly search structured/semi-structured data in unstructured data?

How to search and analyze unstructured data

Specialized software tools and techniques, such as machine learning or, more specifically, deep learning, are used to solve the problem of analyzing and searching unstructured data. Machine learning is an artificial intelligence technique that allows computers to learn from unstructured data without explicit programming. Most machine learning models transform a single piece of unstructured data into a list of floating point values, more commonly referred to as embeddings or embedding vectors, before searching and analyzing the data.

For example, the well-known convolutional neural network ResNet-50 can represent the image below as a vector of length 2048. The first three and last three elements of this vector are of the form: [0.1392, 0.3572, 0.1988, ..., 0.2888, 0.6611, 0.2909].

Embeddings generated by a properly trained neural network have mathematical properties that facilitate their retrieval and analysis. For example, embedding vectors for semantically similar objects are close to each other in terms of distance. As a result, unstructured data can be understood, searched and analyzed using vector arithmetic.

Why work with unstructured data?

While working with unstructured data can be challenging, it is still valuable to developers and enterprises. Unstructured data makes up a whopping 80% of both existing and newly created data, especially in the age of artificial intelligence. They contain a wealth of information that can provide valuable insights into customer behavior, market trends, and other important business metrics to make better decisions. Thanks to technological advances such as natural language processing and deep learning, managing unstructured data will become increasingly easier over time.

In addition, working with unstructured data can help uncover hidden patterns and relationships that would be difficult to identify using traditional methods. Working with unstructured data will also lead to innovation and product development. We are already seeing the emergence of revolutionary applications, services, and products that use large language models (LLMs), such as ChatGPT from OpenAI, to extract value from unstructured data. There will be more of these in the future.

Summary

In this article, we have looked at the meaning and features of unstructured data. We have also looked at the challenges and methods of working with and analyzing unstructured data to make informed business decisions.

In my next articles, I'll go into more detail about vector databases, a simple but effective solution for storing, indexing, and searching unstructured data using the embedding capabilities generated by machine learning models. I will also introduce Milvus, a highly scalable and efficient open source vector database, and detail how Milvus can improve the efficiency of your AI-driven applications. Stay tuned for more information.