This is a summary based on the recent Google Cloud tech talk with Generali Italia, which discusses how their machine learning team created a RAG pipeline for building a retrieval/querying system for their large enterprise documents.
This article serves as a summary of the key points of the talk, targetted for those who would like the gist without having to watch the full video. The pipeline may be similar to other articles, but they define their improvement stages as well as explain in detail how their system works in a lower level, which is quite a good reference for teams who want to build a RAG pipeline but are not sure where to start.
Table of Contents
- Introduction
- Initial Architecture
- Experiments
- Final Architecture
- Learnings
- Q&A
Initial Architecture
- Vertex AI
- LangChain
- Gemini and PaLM models
- Qdrant
As a GCP shop, they used mostly Google components, even for the LLM layer, but with Qdrant as their vector storage. Though towards the end of the talk they mentioned about attempting to use Vertex AI Search to save their embeddings.
Evaluation
As with most RAG pipelines, the team divided their evaluation into retrieval and response. They mostly focused on different parameters for chunking in the experiments, but overall used pretty standard evaluation procedures to determine how well the pipeline returns the correct documents, as well as how it actually answers the users’ queries.
- Document Retrieval
- Response (Q&A)
Experiments
The team explained how they iterated on different experiments to improve the pipeline, created the synthetic dataset, as well as the evaluation process.
Experiment #1 (base case)
Status
- Default value for chunk splitting
- Fixed chunk size
- Base embedding model
- Default value for retriever
Results
- Chunks Generated: 45K
- No evaluation metrics (qualitative only)
Experiment #2
From experiment #2 onwards, they created synthetic data for evaluation as well as use the diagram in [Initial Architecture] section
Improvements
- Split by paragraph
- Tune values for chunk splitting
- Multilingual embedding (embedding-multilingual)
- Tune values for retriever
Results
- Best chunk size: 1000
- Chunks generated: 13,000
Metrics:
- Recall: 80.0% (15 documents)
- Q&A Accuracy: 73.1%
Experiment #3
Improvements
- Added chunks for definitions
Results
- Best chunk size: 1000
- Chunks generated: 17,000
Metrics:
- Recall: 78.0% (15 documents)
- Q&A Accuracy: 72.5%
Experiment #4
From experiment #4, they observed that there is direct correlation between the performance of retrieval and the Q&A accuracy
Improvements
> Hybrid context retriever
- Multilingual embedding
- BM25 (more details here)
Results
- Best chunk size: 1000
- Chunks generated: 17,000
Metrics:
- Recall: 84.0% (15 documents)
- Q&A Accuracy: 76.0%
- Q&A Accuracy: 76.0%
Experiment #5
In this experiment, the team referenced Lost-in-the-middle (Liu et al.) where RAG tends to have lower accuracy from long documents when the context is in the middle of the document.
Improvements
- Context re-ranking using reranking LLM
Results
- By ordering document-chunks in importance ascending order, LLM reaches better performances
Metrics:
- Recall: 84.0% (15 documents)
- Q&A Accuracy: 77.9%
Experiment #6
Improvements
- Larger document collection
- Prompt engineering
- PaLM v1 → PaLM v2
Results
- Increase tool knowledge
Metrics:
- Recall: 69.0% (15 documents)
- PaLM v1 Q&A Accuracy: 72.0%
- PaLM v2 Q&A Accuracy: 78.0%
Final Architecture
Learnings & Next Steps
Summary
- increased accessibility of internal knowledge for the company
- opportunity to experiment cutting-edge AI technologies
- scalability & reliability of Google’s infra
- sharing knowledge with Google Cloud engineers
Next Items to Consider
- Try new Gemini Pro 1.5 models
- Vertex AI Auto SxS for evaluation and benchmarking solutions
- Trying Vertex AI Vector Search instead of using Qdrant
Q&A Takeaways
1. How do you build a consistent test framework especially when you don’t have resources to collect a custom Q&A dataset?
- Ingesting chunks of paragraphs into LLM and creating a synthetic Q&A dataset with the LLM
- Note that a synthetic dataset needs to be checked for quality
2. What do you do when the documents get updated — do you rebuild the index?
- You can just rebuild the index with the text embedding model every time if the document count is still small
- The rule of thumb they found is once the document count gets big enough (which still depends on your use case), you can start to introduce pipelines for updating the index or the parameters (ex. — for chunking)
3. What was your approach to chunking?
- Split by paragraph then sub-split using LangChain’s IterativeSplitter, with the goal that a single chunk is semantically-different to another one
4. Where do you store chunks of text in a gap Cloud Storage or BigQuery?
- Using Vertex AI Pipelines, it saves all the artifacts on Cloud Storage
5. Best practices for chunking large table data such as complex spreadsheets with many sheets?
- You can use libraries like Pandas to preprocess the cells of each sheet
- The way you feed the LLM is also important, so you may need to instruct with a prompt
- Can integrate sentences, separators, etc. to create something more semantically-valid for the LLM
6. How large was your validation set? How many Q&A pairs?
- 2000 questions split into test/validation sets
- Chunk lengths depend on the questions
7. How do you store the chunks with metadata other than the source file name?
- Vector database (Qdrant) has a feature that can store and search through metadata for each chunk
- You can run semantic search as well as filter based on metadata
8. How do you handle scenarios where a use asks a follow-up question that lacks context. (ex. — Question 1: What color is Ivan’s shirt? Question 2: What about Dominico’s?)
- You can create a summary of the previous conversation or,
- You can embed the entire previous conversation into the prompt
- Depends on the length input that the LLM can accept, so it may be limited
- This is still a work in progress for the team