SentenceTransformersDocumentImageEmbedder

Compute document embeddings from image files using Sentence Transformers multimodal models. Use this component in indexing pipelines when document content references image or PDF files stored in metadata.

Embedding Models in Query Pipelines and Indexes

The embedding model you use to embed documents in your indexing pipeline must be the same as the embedding model you use to embed the query in your query pipeline.

This means the embedders for your indexing and query pipelines must match. For example, if you use CohereDocumentEmbedder to embed your documents, you should use CohereTextEmbedder with the same model to embed your queries.

Key Features

• Embeds images referenced by a metadata field such as file_path.
• Supports image files and PDF pages through the configured metadata field.
• Works with multimodal models that embed images and text into the same vector space.
• Stores embeddings in each document's embedding field.
• Supports multiple embedding precisions and backends (PyTorch, ONNX, OpenVINO).

Configuration

1. Drag the SentenceTransformersDocumentImageEmbedder component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Set the model to a multimodal Sentence Transformers model (for example, sentence-transformers/clip-ViT-B-32).
   2. Set file_path_meta_field to the metadata key that contains the image or PDF file path.
4. Go to the Advanced tab to configure root_path, batch_size, normalize_embeddings, precision, and backend.

note

Compatible models include CLIP variants and Jina CLIP models. The model must support image embedding.

Connections

SentenceTransformersDocumentImageEmbedder receives a list of documents whose metadata contains file paths to images or PDFs. It outputs the same documents with embeddings added to their embedding field, ready to be sent to a DocumentWriter.

Source Code

To check this component's source code, open sentence_transformers_doc_image_embedder.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  SentenceTransformersDocumentImageEmbedder:
    type: haystack_integrations.components.embedders.sentence_transformers.SentenceTransformersDocumentImageEmbedder
    init_parameters:
      model: sentence-transformers/clip-ViT-B-32
      file_path_meta_field: file_path
      normalize_embeddings: true

Using the Component in a Pipeline

# haystack-pipeline
components:
  SentenceTransformersDocumentImageEmbedder:
    type: haystack_integrations.components.embedders.sentence_transformers.SentenceTransformersDocumentImageEmbedder
    init_parameters:
      model: sentence-transformers/clip-ViT-B-32
      file_path_meta_field: file_path
      token:
        type: env_var
        env_vars:
        - HF_API_TOKEN
        - HF_TOKEN
        strict: false

  document_writer:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack.document_stores.in_memory.document_store.InMemoryDocumentStore
        init_parameters: {}

connections:
- sender: SentenceTransformersDocumentImageEmbedder.documents
  receiver: document_writer.documents

max_runs_per_component: 100

metadata: {}

Parameters

Inputs

Parameter	Type	Description
documents	List[Document]	Documents whose metadata contains paths to image or PDF files.

Outputs

Parameter	Type	Description
documents	List[Document]	Documents with embedding populated and an embedding_source entry added to metadata.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
file_path_meta_field	str	file_path	The metadata field that contains the file path to the image or PDF.
root_path	Optional[str]	None	Root directory for resolving relative file paths in document metadata.
model	str	sentence-transformers/clip-ViT-B-32	The Sentence Transformers model to use. Must support image embedding.
device	Optional[ComponentDevice]	None	The device to load the model on.
token	Optional[Secret]	Secret.from_env_var(["HF_API_TOKEN", "HF_TOKEN"], strict=False)	The API token to download private models from Hugging Face.
batch_size	int	32	Number of documents to embed at once.
progress_bar	bool	True	Whether to show a progress bar while embedding.
normalize_embeddings	bool	False	Whether to normalize embedding vectors to unit length.
trust_remote_code	bool	False	Whether to allow custom models and scripts from Hugging Face.
local_files_only	bool	False	Whether to use only local files without downloading from Hugging Face.
model_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for the model constructor.
tokenizer_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for the tokenizer.
config_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for the model configuration.
precision	Literal["float32", "int8", "uint8", "binary", "ubinary"]	float32	The precision of the output embeddings.
encode_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for SentenceTransformer.encode.
backend	Literal["torch", "onnx", "openvino"]	torch	The backend to use for the Sentence Transformers model.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
documents	List[Document]		Documents to embed from image or PDF file paths in metadata.

Related Information

• SentenceTransformersDocumentEmbedder
• Pipeline Components