When you ask an AI tool about your document and it responds with a specific, cited answer — "according to section 4.2 on page 18" — something more sophisticated than "read the whole document" is happening. That process has a name: Retrieval-Augmented Generation, or RAG.

Understanding RAG helps you choose better AI tools, interpret their answers more critically, and understand why some tools are dramatically more reliable than others.

The Problem RAG Solves

Large language models have a context window — a hard limit on how much text they can process in one request. Mainstream cloud AI models handle anywhere from 128,000 tokens (roughly 100,000 words) to several hundred thousand tokens in more capable tiers.

This sounds large until you consider:

  • A 500-page PDF contains ~150,000 words — often too long for most context windows
  • Even when it fits, processing the entire document for every query is slow and expensive
  • Models hallucinate more with very long contexts — a known failure mode called "lost in the middle"
  • You rarely need to search across one document. You need to search across dozens or hundreds

RAG solves all of these problems by finding the relevant sections first, then generating an answer using only those sections as context.

How RAG Works: The Four-Step Pipeline

Step 1 — Chunking

Your document is divided into small, overlapping pieces called chunks — typically 300–600 words each. The overlap between adjacent chunks ensures that ideas that span a boundary are not split apart. Each chunk is a retrievable unit of information with a known location (document, page, position).

Step 2 — Embedding

Each chunk is converted into a numerical vector — a list of numbers, typically 384 or 768 dimensions — using a specialized embedding model. Think of this as translating the meaning of text into a point in mathematical space. Similar ideas end up geometrically close together. "Attorney-client privilege" and "lawyer confidentiality" land near each other. "Baking sourdough" lands elsewhere entirely.

Step 3 — Storage

The original chunk text and its embedding vector are stored together — the text in a database, the vector in a vector store. This is your knowledge base. For tens of thousands of chunks, a data structure called HNSW (Hierarchical Navigable Small World) allows sub-millisecond nearest-neighbour search.

Step 4 — Retrieval + Generation

When you ask a question, the RAG pipeline:

  1. Embeds your question using the same model
  2. Finds the top-K chunks whose embeddings are closest to your question's embedding (cosine similarity)
  3. Retrieves those chunks and their source locations
  4. Passes the chunks as context to the language model, along with your question
  5. The model generates an answer based on those chunks — and can cite exactly which chunk (and which page) each piece of information came from

The key insight: The AI never needs to read your whole document for every question. It finds the relevant sections first, then answers using only those sections. This is faster, more accurate, and more auditable than context-stuffing.

Why Citations Are a Feature of Good RAG, Not a Coincidence

When a RAG system tells you "this comes from page 47, section 3.2", that is not a bonus — it is evidence the pipeline is working correctly. The system retrieved a specific chunk, used it to generate an answer, and can trace that answer back to its source.

Tools that give you AI answers without citations are either not using RAG (they are relying on the model's general knowledge, which may be wrong or outdated) or using a poor RAG implementation that does not track provenance.

For professional use — legal research, medical literature, financial due diligence — citations are not optional. They are the difference between an AI that helps you verify information and one that invites you to trust it blindly.

On-Device RAG: The Privacy Breakthrough

Until recently, RAG required server infrastructure — cloud-hosted vector databases, cloud API calls to embedding models, cloud language models for generation. Every document had to leave your device.

Modern on-device hardware — is now fast enough to run the entire RAG pipeline locally:

  • Embedding models run in real time on the Neural Engine
  • HNSW vector search across 10,000+ chunks completes in milliseconds
  • 7B–14B language models generate cited answers locally using MLX and llama.cpp

This is what makes genuinely private document AI possible. Not "private by policy" — private by design. The data never moves.

Kynora is built around this on-device RAG architecture, designed specifically Macs, iPads, and iPhones.