Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

ChonkieRecursiveDocumentSplitter

Split documents into chunks using recursive splitting rules with the Chonkie chunking library. This splitter tries progressively finer separators (paragraphs, then sentences, then words) to create chunks that stay within the token limit while preserving as much semantic structure as possible.

Key Features​

  • Recursively tries different splitting separators from coarser to finer granularity.
  • Supports custom recursive splitting rules via RecursiveRules.
  • Configurable minimum characters per chunk to prevent very small fragments.
  • Preserves page break information and source document metadata.

Configuration​

Add Workspace-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Workspace>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Organization>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.
  1. Drag the ChonkieRecursiveDocumentSplitter component onto the canvas from the Component Library.
  2. Configure the chunk_size to control the maximum number of tokens per chunk.
  3. Optionally provide custom rules to control the splitting hierarchy.

Connections​

ChonkieRecursiveDocumentSplitter receives a list of Document objects as input. It outputs a list of smaller Document objects (chunks) you can connect to a document embedder.

Source Code​

To check this component's source code, open recursive_splitter.py in the Haystack Core Integrations repository.

Usage Examples​

Basic Configuration​

  ChonkieRecursiveDocumentSplitter:
    type: haystack_integrations.components.preprocessors.chonkie.recursive_splitter.ChonkieRecursiveDocumentSplitter
    init_parameters:
      tokenizer: character
      chunk_size: 512
      min_characters_per_chunk: 24

Parameters​

Inputs​

ParameterTypeDescription
documentsList[Document]A list of documents to split recursively.

Outputs​

ParameterTypeDescription
documentsList[Document]A list of document chunks with updated metadata.

Init Parameters​

These are the parameters you can configure in Pipeline Builder:

ParameterTypeDefaultDescription
tokenizerstrcharacterThe tokenizer to use for counting tokens. Use character for character-based counting, or a model name (for example, gpt2) for model-specific tokenization.
chunk_sizeint2048The maximum number of tokens per chunk.
min_characters_per_chunkint24The minimum number of characters required for a chunk to be kept.
rulesOptional[RecursiveRules or Dict]NoneCustom recursive splitting rules. If None, uses default rules (paragraphs → sentences → words).
skip_empty_documentsboolTrueWhether to skip documents with no content.
page_break_characterstr\fThe character used to represent page breaks in document text.

Run Method Parameters​

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

ParameterTypeDefaultDescription
documentsList[Document]The documents to split.