The RAG Pipeline is the end-to-end story of what happens to your knowledge — from the moment you add a source to the moment an agent cites it in an answer.

Overview

Every source goes through the same pipeline before an agent can use it:

1. Fetch — the content is brought in: uploaded files are read from storage, websites are crawled, and application sources are downloaded through their connector.

2. Parse — each document is converted to text by a parser chosen for its file type.

3. Chunk — the text is split into small, retrievable pieces using your chunking settings (Token Text Splitter with a max chunk size of 512 by default).

4. Embed — each chunk is converted into a vector using the configured embedding model.

5. Store — chunks are written to the knowledge store (Open Search) together with their metadata: which knowledge source and set they belong to, source details for citations (file name, page, URL, author), and — for application sources — the list of users allowed to see them.

While this runs, the source shows Syncing; when every document is processed it becomes Completed, and per-document progress (total vs. indexed) drives the counters you see on the source's detail page.

Which Parser Reads My File?

The pipeline first detects the file's type, then routes it:

The default document parser is the platform's built-in reader (Docling). In Indexing settings you can switch a strategy to another parsing application (such as Unstructured or LlamaParse), to Agentic parsing (a vision LLM reads each page — useful for scanned or visually complex documents), or to a custom Automation. If a chosen parser is misconfigured, the platform falls back to the built-in reader.

How Is Text Actually Chunked?

Your indexing settings pick the algorithm — Token Text Splitter (default, max chunk size 512), Recursive Character, Character, Language-aware, Code, or Semantic. Two behaviors worth knowing:

• Chunk overlap — when you set an overlap greater than 0, the splitter shares that many characters between adjacent chunks so sentences aren't cut off at boundaries.

• Semantic Text Splitter — embeds sentences and starts a new chunk where meaning shifts, using your chosen threshold type (Percentile, Standard Deviation, Inter Quartile, or Gradient).

Chunks that come out of a parser larger than the max chunk size are re-split before embedding, so nothing oversized reaches the store. Enrichment strategies (manual values, LLM-generated fields, or automations) run on each chunk during indexing and attach extra metadata that can later be used for filtering and citations. If contextual retrieval is enabled for the source, an LLM also prepends a short document-level context to each chunk before it's embedded, which improves matching for ambiguous snippets.

Which Embedding Model Is Used, and Where Do Vectors Live?

The embedding model is resolved by precedence: a model set on the knowledge source wins, then the knowledge set, then the agent, and finally the workspace's default embedding model. Chunks are embedded in batches and written to an Open Search vector index that matches the model's embedding dimension — models with different dimensions (for example 768, 1024, 1536, 3072) use separate indexes, which is why changing the embedding model triggers a full re-index. Alongside the vector, each chunk stores its text, knowledge source and set IDs, permitted users, and chunk metadata.

What Happens at Question Time — End to End?

When an agent needs knowledge for a query:

1. Rephrase — the query rephrasing step rewrites the user's message into a self-contained question using the conversation history.

2. Scope — the platform works out which knowledge sources apply: the agent's attached sets and sources, minus anything disabled, deleted, or archived.

3. Filter — retrieval filters are assembled: your permissions (for permission-aware application sources), plus any custom metadata filters defined on the agent or knowledge set.

4. Search — the query is embedded and matched against the vector store (semantic similarity, optionally combined with keyword matching — hybrid search). This stage casts a wide net, pulling a few hundred candidate chunks.

5. Rerank — the ranking model re-scores candidates against the query and keeps the top chunks — the Chunk Retrieval number from Pre-processing settings (default 50).

6. Answer — the kept chunks, numbered for citation, go into the answer model's context; the response cites the chunks it used.

How Does Query Rephrasing Work Internally?

The rephraser is an LLM step that sees the conversation history, the current message, and your user context (email, department, and similar profile fields). It decides one of three things:

• Chit-chat — greetings or questions answerable from the conversation itself get a direct reply without any retrieval.

• Rephrase — the message is rewritten into one complete, standalone question (e.g., "for smoking?" after a policy question becomes "what is the company policy for smoking?").

• Retry with context — if the question is incomplete, it is enriched from your user context (e.g., "give me the leave policy" becomes "what is the leave policy of the sales department?").

Rephrasing is skipped — and the raw query used as-is — when the agent is running in full agentic (task-executor) mode, where the agent plans its own tool calls instead. You can also override the default rephrasing behavior by attaching a custom rephraser prompt to the agent.

How Does Reranking Work Internally?

The wide vector search returns far more candidates than the answer needs, in embedding-similarity order. The reranker re-scores them against the query in batches and returns them ranked, keeping the top Chunk Retrieval count (default 50). The ranking model can be a dedicated reranker (hosted rerank models and rerank APIs are supported), a custom reranker automation you provide, or — if the selected model is a text-generation model — an LLM used as a judge. If a reranking call fails, the pipeline falls back to the vector order rather than failing the answer.

How Do Citations Get Attached to Answers?

During retrieval every kept chunk is numbered and recorded as a citation record — its text, source metadata (file name, page, URL, author), relevance score, and the conversation it belongs to. The answer model receives the numbered chunks, and when the response references them the platform publishes a citations block with the message: each citation resolves back to its source document or page so users can click through. Citation usage is counted, which feeds source-usage reporting. When web search is used, web sources are attached the same way.

How Does Permission-Aware Retrieval (RBAC) Work?

For application sources, the connector fetches each item's permissions during sync — every indexed chunk stores the list of users/groups allowed to see the source item. At question time the platform resolves your identity (by email) to those principals and filters the vector search so you only match chunks you're allowed to see. One setting on the knowledge source changes this:

• The knowledge source's Skip User-Level Source Permission Checks toggle ("Allow users to access all application knowledge regardless of their permissions on the source") indexes/serves content without per-user filtering. Choose this when you create the source — it can't be changed afterwards, so to change it you have to recreate the source.

Permission changes at the source are picked up on sync — the pipeline re-checks each item's permissions and updates chunks whose access lists changed without re-embedding unchanged content.

How Does Website Crawling Actually Work?

For All webpages of a URL, the crawler starts at your URL, renders and scrapes the page, converts it to Markdown, indexes it, then extracts the page's links and repeats — skipping already-visited pages — until it reaches your Webpage fetch limit (default 100) or Crawl depth (default 2). For Specific webpages, only the listed pages are indexed (crawl depth is treated as zero). If Include documents is on, files linked from pages (PDFs and similar) are downloaded and indexed through the normal file pipeline, up to a download limit. Every fetched and failed URL is tracked per sync, which is what the Discovered/Synced/Failed counters report. Scraping can optionally use an agentic model to clean complex pages into Markdown.

How Do Application Sources Sync — and What Is the Downloader/Indexer Split?

Application sources run a staged pipeline: a downloader pulls items from the app (files, messages, pages, tickets — whatever the connector supports) into a raw staging store, and an indexer then processes staged items into chunks. Progress cursors track both stages per source, so large sources sync incrementally and resume where they left off. Each connector's import operation defines what gets downloaded, how permissions are fetched, which metadata lands on chunks, and the default incremental-sync cadence. Two sync directions run:

• Backward — the initial (historical) sync of existing content.

• Forward — scheduled incremental grabs of new and changed items (typically hourly or daily by default, adjustable per source).

What's the Difference Between Scheduled and Real-Time Sync?

• Scheduled — the source re-syncs at its Schedule Knowledge Sync interval (plus the forward incremental grabs for app sources). Scheduled syncs are concurrency-limited: if too many sources sync at once, later ones wait as Queued until a slot frees up.

• Real-time — for sources that support it, the platform deploys a webhook listener from the connector's template: the app pushes create/update/delete events, and only the affected items are re-indexed, so changes appear in minutes rather than at the next schedule. The Update Frequency column shows Realtime vs. Scheduled per source.

What Does Indexing a Data Catalog Source Do?

Data catalog sources don't produce text chunks. Instead the platform indexes the catalog's table metadata — schemas, columns, relationships, plus any curated few-shot examples and instructions — into a metadata store. At question time the agent uses that metadata to generate SQL, runs the query against the underlying platform, and answers from the results (with the executed query and result table available as a citation). This is also the machinery behind the Tables to SQL option for spreadsheets.

What Happens When a Document Fails to Index?

Failures are tracked per document, not per source: each staged item carries its own status, error details, and retry count, and websites track failed URLs per sync. A failed document doesn't stop the rest of the source — the source can complete with a Failed count, and the detail page's Error Details lists exactly which items failed and why. The platform retries transient failures, runs background jobs that detect and restart stuck syncs, and can raise alerts when a source's indexing fails or a sync runs unusually long. A full re-sync via Sync Knowledge reprocesses failed items; content removed at the origin is cleaned from the index, and superseded versions of chunks are removed after a successful re-index (each sync writes a new knowledge version, and retrieval uses the latest).