I Engineered Copilot for 3.5 Million Pages: The Epstein Files Challenge

Three and a half million pages. Two thousand videos. One hundred and eighty thousand images. Most people assume that once you connect Microsoft Copilot to a massive dataset, the answers simply appear. The reality is very different.In this episode of the M365 FM Podcast, we go deep into the engineering challenges behind building a retrieval architecture capable of handling one of the largest and most complex information collections imaginable. Using the Epstein Files challenge as a case study, we explore what happens when traditional search and standard Retrieval-Augmented Generation (RAG) approaches collide with millions of documents, transcripts, images, and videos.This is not a discussion about AI marketing. It is a technical deep dive into the infrastructure, orchestration, governance, chunking strategies, retrieval systems, and performance engineering required to make Copilot work at extreme scale.

THE DATA BLINDNESS PROBLEM

Organizations often think Copilot is simply a smarter search engine. In reality, Copilot is an orchestration layer that relies entirely on the quality of the retrieval architecture beneath it.At massive scale, information overload becomes the primary challenge. Questions that should have straightforward answers become buried beneath millions of irrelevant documents. Standard keyword search floods large language models with noise, making it increasingly difficult to identify meaningful signals. The result is what we call data blindness: the information exists, but it becomes practically invisible because of the overwhelming volume of competing content.We explore how retrieval systems fail when legal documents, emails, transcripts, photographs, scanned PDFs, and multimedia assets all compete within the same search environment.

WHY STANDARD RAG COLLAPSES AT SCALE

Retrieval-Augmented Generation works well in controlled environments with relatively small knowledge bases. The assumptions behind standard RAG begin to break down once the dataset reaches millions of pages.In this segment, we analyze why semantic chunking often underperforms at enterprise scale despite sounding attractive in theory. We discuss the hidden costs of sentence-level embeddings, similarity calculations, and preprocessing pipelines that dramatically increase infrastructure costs while sometimes reducing retrieval accuracy.You will learn why more data does not automatically lead to better answers and how poorly designed retrieval architectures can actually increase hallucinations rather than reduce them.

THE SELECTIVE ACTIVATION MODEL

Not every document deserves the same investment.One of the most important concepts discussed in this episode is Selective Activation, a three-tier architecture designed to prioritize the content that delivers the highest business value.Rather than embedding every document equally, the system intelligently separates content into active, supporting, and archival tiers. This dramatically reduces infrastructure costs while improving retrieval performance and maintaining governance requirements.The discussion covers:

• Tier 1 high-value evidence and core documents
• Tier 2 supporting records and operational content
• Tier 3 cold storage and archival retrieval

This model allows organizations to focus resources where they generate the greatest return.

RECURSIVE STRUCTURE-AWARE CHUNKING

Chunking is one of the most overlooked components of enterprise AI architecture.Legal documents, contracts, investigations, and regulatory records contain natural structures that traditional token-based chunking frequently destroys. In this section, we explore recursive structure-aware chunking and how respecting document hierarchy significantly improves retrieval quality.Instead of splitting content at arbitrary token limits, this approach preserves articles, sections, clauses, and narrative context. The result is better grounding, higher retrieval precision, and more accurate answers.We also discuss overlap strategies, metadata preservation, and benchmark results showing why recursive chunking consistently outperforms many expensive alternatives.

BUILDING A MULTIMODAL INGESTION PIPELINE

Modern knowledge repositories are no longer text-only environments.Organizations must process images, scanned documents, video recordings, transcripts, handwritten notes, and multimedia evidence. Making this information searchable requires a sophisticated ingestion pipeline that performs OCR, transcription, image analysis, metadata extraction, and enrichment before users ever submit a query.This episode explores how multimodal ingestion transforms unsearchable content into structured knowledge that Copilot can retrieve and reason over.

ENTITY EXTRACTION AND KNOWLEDGE GRAPHS

Raw text is information. Relationships create understanding.We examine how entity extraction transforms millions of disconnected references into a structured knowledge graph capable of identifying people, organizations, locations, events, and relationships.Rather than forcing the AI model to discover relationships during generation, the system extracts and organizes these connections during ingestion. This reduces hallucinations, improves retrieval accuracy, and enables advanced relationship-based questioning across large datasets.

THE AGENTIC ROUTER

Not all questions require the same retrieval strategy.The Agentic Router serves as the intelligence layer that determines what a user is actually asking and routes requests to the most appropriate retrieval systems.Whether a query requires structured databases, knowledge graphs, keyword indexes, vector search, or document retrieval, the router decomposes complex requests into specialized tasks and orchestrates the response process.This section provides a practical look at query decomposition, intent classification, fallback mechanisms, and confidence scoring.

HYBRID RETRIEVAL AND RERANKING

Modern enterprise retrieval requires more than vector search alone.We explore why combining BM25 keyword retrieval, vector search, Reciprocal Rank Fusion, metadata filtering, and transformer-based reranking delivers superior results compared to any individual approach.Hybrid retrieval balances precision and recall while reducing retrieval noise before information ever reaches the large language model.The conversation includes practical implementation considerations, latency tradeoffs, and the impact of reranking on answer quality.

PERMISSION-AWARE RETRIEVAL

Security cannot be an afterthought.When dealing with millions of pages, access control becomes a foundational architectural requirement rather than a feature.We discuss chunk-level permissions, Azure Active Directory integration, sensitivity labels, compliance boundaries, audit trails, and governance models that ensure users only receive information they are authorized to access.This section highlights why permission-aware retrieval is one of the most critical components of enterprise AI deployment.

LATENCY, PERFORMANCE, AND TIME-TO-FIRST-TOKEN

Users judge AI systems by speed.Even the most accurate answer loses value if it arrives too slowly.This episode examines Time-to-First-Token (TTFT), retrieval latency, reranking overhead, permission filtering costs, caching strategies, and parallel processing techniques that enable sub-second experiences at enterprise scale.You will learn where latency accumulates inside the retrieval pipeline and how architectural decisions directly influence user adoption.

GOVERNANCE, COMPLIANCE, AND ENTERPRISE READINESS

Enterprise AI is not simply about retrieval performance.Governance frameworks, retention policies, legal holds, audit logging, data residency requirements, and compliance controls determine whether a system can safely operate in production environments.We explore how governance becomes increasingly important as datasets grow and why organizations must design compliance directly into their architecture rather than adding it later.

THE ORCHESTRATION LAYER

Every component discussed in this episode ultimately converges inside the orchestration layer.The orchestration layer coordinates ingestion, chunking, enrichment, indexing, retrieval, reranking, permission filtering, answer generation, feedback loops, monitoring, and scaling.Without orchestration, organizations are left with disconnected technologies. With orchestration, those technologies become a coherent AI system capable of turning millions of pages into actionable knowledge.

KEY TAKEAWAYS

• Copilot is an orchestration engine, not a search engine.
• Retrieval architecture determines answer quality.
• Recursive chunking often outperforms expensive semantic approaches.
• Metadata enrichment dramatically improves retrieval accuracy.
• Hybrid retrieval provides the best balance of precision and recall.
• Governance and security must be built into the architecture from day one.

CONNECT WITH M365 FM

If you enjoyed this episode, subscribe to M365 FM for deep technical conversations covering Microsoft 365, Microsoft Copilot, Azure AI, enterprise search, knowledge management, governance, security, and the future of intelligent workplaces.New episodes explore real-world architectures, implementation strategies, lessons learned from large-scale deployments, and the technologies shaping the next generation of work.Subscribe, leave a review, and share the episode with anyone building AI-powered solutions at enterprise scale.

Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365--6704921/support.