RAG-MCP: Mitigating Prompt Bloat and Enhancing Tool Selection for LLM

Large Language Models (LLMs) face significant challenges in effectively using a growing number of external tools, such as those defined by the Model Context Protocol (MCP). These challenges include prompt bloat and selection complexity. As the number of available tools increases, providing definitions for every tool in the LLM's context consumes an enormous number of tokens, risking overwhelming and confusing the model, which can lead to errors like selecting suboptimal tools or hallucinating non-existent ones.

To address these issues, the RAG-MCP framework is introduced. This approach leverages Retrieval-Augmented Generation (RAG) principles applied to tool selection. Instead of presenting all available tool descriptions to the LLM at once, RAG-MCP uses semantic retrieval to dynamically identify and select only the most relevant tools from an external index based on the user's query. Only the descriptions of these selected tools (or MCPs) are then passed to the LLM.

This process significantly reduces the prompt size and simplifies the decision-making required from the LLM. The framework's pipeline involves encoding the user's task input, submitting it to a retriever that searches a vector index of MCP schemas, ranking candidates, and optionally validating them, before the LLM executes the task using only the selected MCP's information.

Key benefits demonstrated by RAG-MCP include a drastic reduction in prompt tokens (cutting usage by over 50% compared to including all tools) and a significant boost in tool selection accuracy (tripling the success rate of baseline methods, achieving 43.13% compared to 13.62% for Blank Conditioning). The approach also leads to lower cognitive load for the LLM, resource efficiency by only activating selected MCPs, and multi-turn robustness. RAG-MCP enables scalable and accurate tool integration and remains extensible, as new tools can be added simply by indexing their metadata without needing to retrain the LLM.