What's the Magic Behind AI?

There’s a lot of exciting things that AI can do. What’s behind it that helps make the magic happen?

We know that computers can process text, images, audio, videos and more. But how did they gain the ability to truly understand? In the context of AI and training Large Language Models, an important factor is that machines must reliably and accurately extract meaning out of different kinds of data. For example, Netflix and Spotify can understand your taste and display content that you may like, or when typing a search query on Google, it shows you the most relevant results, even though your query isn’t fully formed or complete. How does this happen? The answer lies in a process called feature extraction. Once the important features are identified, they’re stored in vector embeddings which are numerical representations of this information. 

For example, when a vector embedding captures the relationship between [tree, leaf] or [root, soil], it can understand that [tree, soil] are probably more closely connected than [tree, purple]. The numbers in an embedding represent features of the data that help capture these relationships. While it cannot exactly capture meaning the way humans do, it can get impressively close.

Details Matter! Embeddings become more powerful when we can capture more dimensions from a single token. For example, some data points that we could extract out of a token [leaf] could be: noun, color, singular, nature, natural, and so on. Vectorization algorithms can create hundreds, thousands and even millions of these data points. The more dimensions an embedding has, the more accurately our algorithm can capture meaning and nuances in these tokens.

In addition to text, embeddings can also represent images and videos. This allows us to significantly reduce the complexity of the data that we’re dealing with and enable machine learning algorithms work more efficiently.

Training is Key Embeddings are created by training on massive amounts of text, images, or audio. At a very high level, they learn the meaning of a word by predicting what token must come next and then correcting itself. The following visual from Taipaulus may shed light on how mathematical calculations could be done from vector embeddings.

Figure 1: An example of a two-dimensional vector

So Now What? Once embeddings are created, what can we do with them? Think of them as building blocks—each embedding is a compact representation that captures meaning and relationships between tokens. This is valuable information.

What About Confidential Data?

Trade secrets, expertise and confidential data are certainly areas that we don’t want AI to train on and make available to the general public. Can we still train models on it and keep it be safe? The answer is yes. It’s entirely possible to build ML systems that can store this data, answer questions, and be able to search this data for specific information… all while making sure that your valuable data stays yours!

It must be said that the role of embeddings for training Neural Networks – for deep learning – changed the landscape significantly. It helped to scale down the volume of data that we’re dealing with through feature extraction, thereby allowing us to perform even more complex calculations for better results by training only on the features that represent the data rather than the data itself.

So don’t worry… AI won’t replace human thinking, rather, when harnessed appropriately, it allows us to “work smarter.”

References:  Attention is all you need arXiv:1706.03762

A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge arXiv:2310.11703

Toni Taipalus, Vector database management systems: Fundamental concepts, use-cases, and current challenges, Cognitive Systems Research, Volume 85, 2024, 101216, ISSN 1389-0417, https://doi.org/10.1016/j.cogsys.2024.101216. (https://www.sciencedirect.com/science/article/pii/S1389041724000093)