Chunk Text splits long text into smaller, retrievable segments — the foundational step for embedding, search, and RAG pipelines. Choose from multiple splitting strategies to match your content type and retrieval goals.
Overview
The Chunk Text operation divides input text into segments that downstream embedding and retrieval systems can process effectively. You select a strategy and, optionally, a chunk size and overlap. Most pipelines start with recursive character, which tries to break on natural boundaries — paragraphs first, then lines, then words — making it the sensible default for prose. The semantic strategy uses an embedding model to group text by meaning; it produces more coherent chunks but is slower and depends on that model being available. Code-aware and syntax-tree strategies handle source code and respect language grammar. Fixed character and token splitting use exact counts. Passthrough disables splitting entirely and returns the text unchanged.
By default chunks have no overlap. Setting an overlap value causes adjacent chunks to share a configurable number of characters or tokens, which reduces missed context at segment boundaries.
Input
Field | Type | Required | Description |
|---|---|---|---|
Text | string | Yes | The text to split into chunks. |
Chunking Strategy | enum | Yes | How to split the text. Options: recursive character (default for prose), fixed character, token, semantic, code-aware, syntax-tree, passthrough. |
Chunk Size | integer | No | Maximum characters or tokens per chunk. Applies to all strategies except passthrough. |
Overlap | integer | No | Characters or tokens shared between adjacent chunks. Defaults to 0 (no overlap). |
Embedding Model | string | Conditional | Required when Chunking Strategy is semantic. The model used to group text by meaning. |
Output
Returns an array of text chunks. Each element is a segment of the original text within the configured size limit. When Chunking Strategy is passthrough, the array contains a single element equal to the full input.
Notes
Start with recursive character for general prose; switch to semantic when retrieval quality matters more than speed.
The semantic strategy calls an embedding model and is slower than the others. Ensure that model is available before using this strategy.
The syntax-tree code splitter targets a set of common programming languages and caps each chunk to the configured size.
Passthrough returns the full input as a single chunk — useful when you want a consistent pipeline shape without actually splitting.
Overlap defaults to 0; increase it if retrieval results are missing context at segment boundaries.
Use Chunk Text before an embedding step; chunked text produces vectors that retrieval engines can compare efficiently.