Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

ChonkieSentenceDocumentSplitter

Split documents into chunks at sentence boundaries using the Chonkie chunking library. Use this component in indexing pipelines to prepare documents for embedding and retrieval.

Key Features​

  • Splits text at sentence boundaries to preserve semantic coherence within chunks.
  • Supports multiple tokenizer backends including character-based, word-based, and model-specific tokenizers.
  • Configurable minimum sentences per chunk and minimum characters per sentence.
  • Preserves page break information and source document metadata.
  • Adds split metadata to each chunk: source_id, page_number, split_id, split_idx_start, split_idx_end, and token_count.

Configuration​

Add Workspace-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Workspace>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Organization>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.
  1. Drag the ChonkieSentenceDocumentSplitter component onto the canvas from the Component Library.
  2. Configure the chunk_size to control the maximum number of tokens per chunk.
  3. Optionally set chunk_overlap to include overlapping content between adjacent chunks for better retrieval continuity.

Connections​

ChonkieSentenceDocumentSplitter receives a list of Document objects as input from a document converter or file router. It outputs a list of smaller Document objects (chunks) you can connect to a document embedder.

Source Code​

To check this component's source code, open sentence_splitter.py in the Haystack Core Integrations repository.

Usage Examples​

Basic Configuration​

  ChonkieSentenceDocumentSplitter:
    type: haystack_integrations.components.preprocessors.chonkie.sentence_splitter.ChonkieSentenceDocumentSplitter
    init_parameters:
      tokenizer: character
      chunk_size: 512
      chunk_overlap: 50

Parameters​

Inputs​

ParameterTypeDescription
documentsList[Document]A list of documents to split into sentence-based chunks.

Outputs​

ParameterTypeDescription
documentsList[Document]A list of document chunks with updated metadata.

Init Parameters​

These are the parameters you can configure in Pipeline Builder:

ParameterTypeDefaultDescription
tokenizerstrcharacterThe tokenizer to use for counting tokens. Use character for character-based counting, or a model name (for example, gpt2) for model-specific tokenization.
chunk_sizeint2048The maximum number of tokens per chunk.
chunk_overlapint0The number of tokens to overlap between adjacent chunks.
min_sentences_per_chunkint1The minimum number of sentences to include in each chunk.
min_characters_per_sentenceint12The minimum number of characters a sentence must have to be included.
approximateboolFalseWhether to use approximate token counting for faster processing.
delimAnyNoneCustom sentence delimiters. If None, uses default sentence-ending punctuation.
include_delimstrprevWhere to include the delimiter in the chunk. One of prev or next.
skip_empty_documentsboolTrueWhether to skip documents with no content.
page_break_characterstr\fThe character used to represent page breaks in document text.

Run Method Parameters​

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

ParameterTypeDefaultDescription
documentsList[Document]The documents to split.