diff --git a/docs/rag_guide.md b/docs/rag_guide.md new file mode 100644 index 000000000..810cd2f34 --- /dev/null +++ b/docs/rag_guide.md @@ -0,0 +1,236 @@ +# RAG Configuration Guide + +This document explains how to configure and customize your RAG pipeline using the `llama-stack` configuration YAML file. You will: + +* Initialize a vector store +* Download and point to a local embedding model +* Configure an inference provider (LLM) +* Enable Agent-based RAG querying + +--- + +## Table of Contents + +* [Introduction](#introduction) +* [Prerequisites](#prerequisites) + * [Set Up the Vector Database](#set-up-the-vector-database) + * [Download an Embedding Model](#download-an-embedding-model) +* [Configure Vector Store and Embedding Model](#configure-vector-store-and-embedding-model) +* [Add an Inference Model (LLM)](#add-an-inference-model-llm) +* [Complete Configuration Reference](#complete-configuration-reference) +* [References](#references) + + +--- + +# Introduction + +RAG in Lightspeed Core Stack (LCS) is yet only supported via the Agents API. The agent is responsible for planning and deciding when to query the vector index. + +The system operates a chain of command. The **Agent** is the orchestrator, using the LLM as its reasoning engine. When a plan requires external information, the Agent queries the **Vector Store**. This is your database of indexed knowledge, which you are responsible for creating before running the stack. The **Embedding Model** is used to convert the queries to vectors. + +> [!NOTE] +> The same Embedding Model should be used to both create the store and to query it. + +--- + +# Prerequisites + +## Set Up the Vector Database + +Use the [`rag-content`](https://github.com/lightspeed-core/rag-content) repository to build a compatible vector database. + +> [!IMPORTANT] +> The resulting DB must be compatible with Llama Stack (e.g., FAISS with SQLite metadata, SQLite-vec). This can be configured when using the tool to generate the index. + +--- + +## Download an Embedding Model + +Download a local embedding model such as `sentence-transformers/all-mpnet-base-v2` by using the script in [`rag-content`](https://github.com/lightspeed-core/rag-content) or manually download and place in your desired path. + +> [!NOTE] +> Llama Stack can also download a model for you, which will make the first start-up slower. In the YAML configuration file `run.yaml` specify a supported model name as `provider_model_id` instead of a path. LLama Stack will then download the model to the `~/.cache/huggingface/hub` folder. + +--- + +## Configure Vector Store and Embedding Model + +Update the `run.yaml` file used by Llama Stack to point to: + +* Your downloaded **embedding model** +* Your generated **vector database** + +Example: + +```yaml +models: +- model_id: # e.g. sentence-transformers/all-mpnet-base-v2 + metadata: + embedding_dimension: # e.g. 768 + model_type: embedding + provider_id: sentence-transformers + provider_model_id: # e.g. /home/USER/embedding_model + +providers: + inference: + - provider_id: sentence-transformers + provider_type: inline::sentence-transformers + config: {} + + # FAISS vector store + vector_io: + - provider_id: custom-index + provider_type: inline::faiss + config: + kvstore: + type: sqlite + db_path: # e.g. /home/USER/vector_db/faiss_store.db + namespace: null + +vector_dbs: +- embedding_dimension: # e.g. 768 + embedding_model: # e.g. sentence-transformers/all-mpnet-base-v2 + provider_id: custom-index + vector_db_id: +``` + +Where: +- `provider_model_id` is the path to the folder of the embedding model (or alternatively, the supported embedding model to download) +- `db_path` is the path to the vector index (.db file in this case) +- `vector_db_id` is the index ID used to generate the db + +--- + +## Add an Inference Model (LLM) + +Add a provider for your language model (e.g., OpenAI): + +```yaml +models: +[...] +- model_id: my-model + provider_id: openai + model_type: llm + provider_model_id: # e.g. gpt-4o-mini + +providers: +[...] + inference: + - provider_id: openai + provider_type: remote::openai + config: + api_key: ${env.OPENAI_API_KEY} +``` + +Make sure to export your API key: + +```bash +export OPENAI_API_KEY= +``` + +> [!NOTE] +> When experimenting with different `models`, `providers` and `vector_dbs`, you might need to manually unregister the old ones with the Llama Stack client CLI (e.g. `llama-stack-client vector_dbs list`) + + +--- + +# Complete Configuration Reference + +To enable RAG functionality, make sure the `agents`, `tool_runtime`, and `safety` APIs are included and properly configured in your YAML. + +Below is a real example of a working config, with: + +* A local `all-mpnet-base-v2` embedding model +* A `FAISS`-based vector store +* `OpenAI` as the inference provider +* Agent-based RAG setup + +> [!TIP] +> We recommend starting with a minimal working configuration (one is automatically generated by the `rag-content` tool when generating the database) and extending it as needed by adding more APIs and providers. + +```yaml +version: 2 +image_name: rag-configuration + +apis: +- agents +- inference +- vector_io +- tool_runtime +- safety + +models: +- model_id: gpt-test + provider_id: openai # This ID is a reference to 'providers.inference' + model_type: llm + provider_model_id: gpt-4o-mini + +- model_id: sentence-transformers/all-mpnet-base-v2 + metadata: + embedding_dimension: 768 + model_type: embedding + provider_id: sentence-transformers # This ID is a reference to 'providers.inference' + provider_model_id: /home/USER/lightspeed-stack/embedding_models/all-mpnet-base-v2 + +providers: + inference: + - provider_id: sentence-transformers + provider_type: inline::sentence-transformers + config: {} + + - provider_id: openai + provider_type: remote::openai + config: + api_key: ${env.OPENAI_API_KEY} + + agents: + - provider_id: meta-reference + provider_type: inline::meta-reference + config: + persistence_store: + type: sqlite + db_path: .llama/distributions/ollama/agents_store.db + responses_store: + type: sqlite + db_path: .llama/distributions/ollama/responses_store.db + + safety: + - provider_id: llama-guard + provider_type: inline::llama-guard + config: + excluded_categories: [] + + vector_io: + - provider_id: ocp-docs + provider_type: inline::faiss + config: + kvstore: + type: sqlite + db_path: /home/USER/lightspeed-stack/vector_dbs/ocp_docs/faiss_store.db + namespace: null + + tool_runtime: + - provider_id: rag-runtime + provider_type: inline::rag-runtime + config: {} + +# Enable the RAG tool +tool_groups: +- provider_id: rag-runtime + toolgroup_id: builtin::rag + args: null + mcp_endpoint: null + +vector_dbs: +- embedding_dimension: 768 + embedding_model: sentence-transformers/all-mpnet-base-v2 + provider_id: ocp-docs # This ID is a reference to 'providers.vector_io' + vector_db_id: openshift-index # This ID was defined during index generation +``` + +# References + +* [Llama Stack - RAG](https://llama-stack.readthedocs.io/en/latest/building_applications/rag.html) +* [Llama Stack - Configuring a “Stack"](https://llama-stack.readthedocs.io/en/latest/distributions/configuration.html) +* [Llama Stack - Sample configurations](https://github.com/meta-llama/llama-stack/tree/main/llama_stack/distributions)