TransformersNamedEntityExtractor

Extract named entities from documents using Hugging Face token classification models and store the annotations in each document's metadata.

Key Features

• Works with any token classification model from the Hugging Face model hub.
• Adds named entity annotations to the named_entities metadata field on each document.
• Supports configurable batch size for processing multiple documents.
• Supports GPU inference with configurable device placement.
• Includes entity scores from the model when available.

Configuration

1. Drag the TransformersNamedEntityExtractor component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model to a Hugging Face NER model (for example, dslim/bert-base-NER).
   2. If the model requires authentication, create a secret with your Hugging Face API token. Use HF_API_TOKEN or HF_TOKEN as the environment variable name. For instructions, see Create Secrets.
4. Go to the Advanced tab to configure pipeline_kwargs and device.

Connections

TransformersNamedEntityExtractor receives a list of documents and outputs the same documents with a named_entities metadata field. Connect it to downstream components that read document metadata.

Source Code

To check this component's source code, open named_entity_extractor.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  TransformersNamedEntityExtractor:
    type: haystack_integrations.components.extractors.transformers.named_entity_extractor.TransformersNamedEntityExtractor
    init_parameters:
      model: dslim/bert-base-NER

Using the Component in a Pipeline

# haystack-pipeline
components:
  TransformersNamedEntityExtractor:
    type: haystack_integrations.components.extractors.transformers.named_entity_extractor.TransformersNamedEntityExtractor
    init_parameters:
      model: dslim/bert-base-NER
      token:
        type: env_var
        env_vars:
        - HF_API_TOKEN
        - HF_TOKEN
        strict: false

  document_writer:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
        init_parameters: {}

connections:
- sender: TransformersNamedEntityExtractor.documents
  receiver: document_writer.documents

max_runs_per_component: 100

metadata: {}

Parameters

Inputs

Parameter	Type	Description
documents	List[Document]	Documents to annotate with named entities.
batch_size	int	Batch size used for processing the documents.

Outputs

Parameter	Type	Description
documents	List[Document]	The input documents with meta["named_entities"] set to a list of entity annotations. Each annotation includes entity, start, end, and optionally score.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
model	str		Name of the Hugging Face model or a path to the model on the local disk.
pipeline_kwargs	Optional[Dict[str, Any]]	None	Keyword arguments passed to the Hugging Face pipeline.
device	Optional[ComponentDevice]	None	The device on which the model is loaded. If None, the default device is automatically selected.
token	Optional[Secret]	Secret.from_env_var(["HF_API_TOKEN", "HF_TOKEN"], strict=False)	The API token to download private models from Hugging Face.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
documents	List[Document]		Documents to annotate with named entities.
batch_size	int	1	Batch size used for processing the documents.

Related Information

• SpacyNamedEntityExtractor
• Pipeline Components