|
A CIO on the data foundation AI agents need (Sponsored)
 |
The CIO of GlobalFoundries had a straightforward view of it: you don’t get AI agents until the data underneath them is real-time and governed.
So he rebuilt that layer first — one platform carrying data across fabs on three continents, with identity, permissions, and audit trails handled once instead of per project. The agents followed, across IT, procurement, and other business functions.
Join us on September 10 to hear how he did it, and get your questions answered live.
RAG, or Retrieval-Augmented Generation, is helping a lot of companies build chatbots specific to their requirements. However, the success of any such RAG system depends on the quality of the embedding model (the model that helps translate words into numbers) that is used by the RAG system.
Consider the example where a particular product’s documentation specifies that annual subscriptions can be refunded only within 30 days. A chatbot built using RAG is supposed to answer customer queries based on this support documentation. However, when a customer asks whether a subscription bought 45 days ago can be refunded, the chatbot confidently answers “yes”.
Why did the chatbot make a mess of this seemingly simple question?
The answer lies in the behaviour of the embedding model, which is kind of a translator for AI. This model controls the answer-searching process and is different from the language model that generates the actual answer. No matter how good the language model might be, it cannot give correct answers if the embedding model doesn’t do its job properly.
In this article, we’re going to look at how this embedding model works in an RAG setup and what makes it such a critical part of the system. Here’s what we will cover:
Why a RAG system searches before it answers
How embeddings make it possible to search by meaning
Why related information is not always the right information
Why a better language model cannot repair bad retrieval
What makes an embedding model suitable for a RAG system
How to compare embedding models without trusting benchmark scores blindly
Choosing between commercial APIs and locally run models
Why changing the embedding model later becomes expensive
How Matryoshka embeddings offer more control over vector size
 |
Why a RAG System Searches Before Answering
A generic language model’s knowledge depends on what it was taught during training. You can’t expect such a model to know about a company’s private documents, policies, or internal source code. It won’t even know about the company’s latest information that might have been published after the model was trained. Technically, it’s quite costly to retrain a language model every time there’s a change in the information.
This is where RAG helps. It separates the ability to generate language from finding relevant details from a pool of stored knowledge. While the language model handles the understanding and writing parts, the external knowledge collection contains the actual information that should be used for writing the answers.
 |
RAG works in two different phases: indexing and retrieval.
During the indexing phase, the system prepares the documents:
It collects documents from various sources (files, websites, databases, etc.).
It extracts their text.
It divides the text into smaller passages known as chunks.
It sends each chunk into an embedding model.
It stores the resulting vector with the original text and metadata.