What is the need for RAG?
LLMs are taught on a vast amount of data. They can answer queries based on the trained data set. If you ask any query about things not covered in the dataset, LLMs may hallucinate and provide incorrect answers.
LLMs are very powerful, but are also costly to train. Only a handful of companies have the required capability and resources for training LLMs.
LLMs currently train on data sets with a cutoff date. So any recent public data is not part of their training data set. Organizations have their own proprietary and private data which is not used for LLM training. Some datasets change frequently and are very dynamic.
There are few ways to incorporate the recent or private company data. The costliest option is to retrain the LLM using new data. RAG is a simpler and cheaper alternative.
Understanding RAG
Retrieval Augmented Generation name is self explanatory. While querying the LLMs we add additional data specific to our requirement. The LLM uses this augmented information to generate responses that are more relevant to the given context.
If the additional data is small, we can simply provide all data to LLMs as part of our prompts. But LLMs have a limited context window. LLMs with larger context windows are generally more expensive. Also too much data in context adversely affects the quality of results.
In many cases the additional data to consider is huge. In those cases, RAG is useful. RAG helps us to specify only the relevant data to the context instead of all additional data.
There are various ways to specify the relevant data. It is essentially a search problem. One can just grep a list of files which will be slower. One can also use OpenSearch or Elastic Search to do textual search to find relevant data.
However most RAG solutions prefer embedding-based vector stores. All the additional data sets are converted into a vector representation and stored in a vector database. During querying, the same embedding model is used to identify the most relevant information.
How is RAG done?
RAG is done in two stages.
Document/Data Ingestion Step
In this step, the documents i..e datasets are ingested into the storage. As specified above, people generally prefer vector databases. This can be done once or in regular intervals to maintain freshness of content.
The documents are passed through an embedding model. The embedding model converts the data into vector representations. The vector representations are stored in a vector database.
Querying Step
This happens in real time when the user submits a prompt or query to the system.
The specific query is passed through the same embedding model. The resulting vector representation of the query or prompt is used to pull relevant document or document chunks from the vector database.
The original user prompt is then augmented with the retrieved information to create an augmented prompt for the LLM. LLM processes this augmented prompt and generates the response.