Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

ChonkieTokenDocumentSplitter

Split documents into fixed-size token chunks using the Chonkie chunking library. Use this component in indexing pipelines when you need predictable, token-aligned chunks rather than sentence-boundary splitting.

Key Features​

  • Splits text into chunks of a fixed maximum token size.
  • Supports multiple tokenizer backends including character-based and model-specific tokenizers.
  • Configurable overlap between adjacent chunks.
  • Preserves page break information and source document metadata.

Configuration​

Add Workspace-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Workspace>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Organization>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.
  1. Drag the ChonkieTokenDocumentSplitter component onto the canvas from the Component Library.
  2. Configure the chunk_size to control the maximum number of tokens per chunk.
  3. Optionally set chunk_overlap to include overlapping content between adjacent chunks.

Connections​

ChonkieTokenDocumentSplitter receives a list of Document objects as input from a document converter or file router. It outputs a list of smaller Document objects (chunks) you can connect to a document embedder.

Source Code​

To check this component's source code, open token_splitter.py in the Haystack Core Integrations repository.

Usage Examples​

Basic Configuration​

  ChonkieTokenDocumentSplitter:
    type: haystack_integrations.components.preprocessors.chonkie.token_splitter.ChonkieTokenDocumentSplitter
    init_parameters:
      tokenizer: character
      chunk_size: 512
      chunk_overlap: 50

Parameters​

Inputs​

ParameterTypeDescription
documentsList[Document]A list of documents to split into token-based chunks.

Outputs​

ParameterTypeDescription
documentsList[Document]A list of document chunks with updated metadata.

Init Parameters​

These are the parameters you can configure in Pipeline Builder:

ParameterTypeDefaultDescription
tokenizerstrcharacterThe tokenizer to use for counting tokens. Use character for character-based counting, or a model name (for example, gpt2) for model-specific tokenization.
chunk_sizeint2048The maximum number of tokens per chunk.
chunk_overlapint0The number of tokens to overlap between adjacent chunks.
skip_empty_documentsboolTrueWhether to skip documents with no content.
page_break_characterstr\fThe character used to represent page breaks in document text.

Run Method Parameters​

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

ParameterTypeDefaultDescription
documentsList[Document]The documents to split.