DeepsetAmazonBedrockVisionGenerator

Generate text using prompts containing text and images with large language models hosted on Amazon Bedrock.

Deprecation Notice

This component is deprecated. It will continue to work in your existing pipelines for now. You can replace it with the AmazonBedrockChatGenerator component.

DeepsetAmazonBedrockVisionGenerator makes it possible to use models in Amazon Bedrock through Haystack Platform's Amazon Bedrock account. This component only works with models that support multimodal inputs.

For a full list of models, see Amazon Bedrock documentation.

Key Features

• Accepts both text prompts and images (as Base64Image objects) for multimodal generation.
• Accesses Amazon Bedrock models through Haystack Platform's account — no separate AWS account required.
• Supports streaming responses token by token.
• Any text generation parameters valid for the underlying model can be passed via generation_kwargs.

Configuration

1. Drag the DeepsetAmazonBedrockVisionGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Enter the model name in the Model field (for example, anthropic.claude-3-5-sonnet-20241022-v2:0).
   2. Optionally, enter a system prompt to define the model's behavior.
4. Go to the Advanced tab to configure streaming_callback and generation_kwargs.

Connections

DeepsetAmazonBedrockVisionGenerator receives a text prompt from PromptBuilder through its prompt input and a list of Base64Image objects (typically from DeepsetPDFDocumentToBase64Image) through its images input. It outputs generated text as a list of strings through its replies output, which you connect to DeepsetAnswerBuilder.

Usage Examples

Basic Configuration

  DeepsetAmazonBedrockVisionGenerator:
    type: deepset_cloud_custom_nodes.generators.deepset_amazon_bedrock_vision_generator.DeepsetAmazonBedrockVisionGenerator
    init_parameters:
      model: anthropic.claude-3-5-sonnet-20241022-v2:0

This example uses the Sonnet 3.5 model hosted on Amazon Bedrock to generate answers. It gets the prompt with documents from PromptBuilder and then sends the generated replies to AnswerBuilder:

# haystack-pipeline
components:
    bm25_retriever: # Selects the most similar documents from the document store
      type: haystack_integrations.components.retrievers.opensearch.bm25_retriever.OpenSearchBM25Retriever
      init_parameters:
        document_store:
          type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
          init_parameters:
            embedding_dim: 1024
        top_k: 20 # The number of results to return
        fuzziness: 0

    query_embedder:
      type: deepset_cloud_custom_nodes.embedders.nvidia.text_embedder.DeepsetNvidiaTextEmbedder
      init_parameters:
        normalize_embeddings: true
        model: "BAAI/bge-m3"

    embedding_retriever: # Selects the most similar documents from the document store
      type: haystack_integrations.components.retrievers.opensearch.embedding_retriever.OpenSearchEmbeddingRetriever
      init_parameters:
        document_store:
          type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
          init_parameters:
            embedding_dim: 1024
        top_k: 20 # The number of results to return

    document_joiner:
      type: haystack.components.joiners.document_joiner.DocumentJoiner
      init_parameters:
        join_mode: concatenate

    ranker:
      type: deepset_cloud_custom_nodes.rankers.nvidia.ranker.DeepsetNvidiaRanker
      init_parameters:
        model: "BAAI/bge-reranker-v2-m3"
        top_k: 5

    meta_field_grouping_ranker:
      type: haystack.components.rankers.meta_field_grouping_ranker.MetaFieldGroupingRanker
      init_parameters:
        group_by: file_id
        subgroup_by: null
        sort_docs_by: split_id

    image_downloader:
      type: deepset_cloud_custom_nodes.augmenters.deepset_file_downloader.DeepsetFileDownloader
      init_parameters:
        file_extensions:
          - ".pdf"

    pdf_to_image:
      type: deepset_cloud_custom_nodes.converters.pdf_to_image.DeepsetPDFDocumentToBase64Image
      init_parameters:
        detail: "high"

    prompt_builder:
      type: haystack.components.builders.prompt_builder.PromptBuilder
      init_parameters:
        template: |-
          Answer the question briefly and precisely based on the pictures.
          Give reasons for your answer.
          When answering the question only provide references within the answer text.
          Only use references in the form [NUMBER OF IMAGE] if you are using information from a image. 
          For example, if the first image is used in the answer add [1] and if the second image is used then use [2], etc. 
          Never name the images, but always enter a number in square brackets as a reference. 
          Question: {{ question }}
          Answer: 

        required_variables: "*"
    llm:
      type: deepset_cloud_custom_nodes.generators.deepset_amazon_bedrock_vision_generator.DeepsetAmazonBedrockVisionGenerator
      init_parameters:
        model: anthropic.claude-3-5-sonnet-20241022-v2:0
        aws_region_name: us-west-2
        max_length: 10000
        model_max_length: 200000
        temperature: 0

    answer_builder:
      type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
      init_parameters:
        reference_pattern: acm

connections:  # Defines how the components are connected
  - sender: bm25_retriever.documents
    receiver: document_joiner.documents
  - sender: query_embedder.embedding
    receiver: embedding_retriever.query_embedding
  - sender: embedding_retriever.documents
    receiver: document_joiner.documents
  - sender: document_joiner.documents
    receiver: ranker.documents
  - sender: ranker.documents
    receiver: meta_field_grouping_ranker.documents
  - sender: meta_field_grouping_ranker.documents
    receiver: image_downloader.documents
  - sender: image_downloader.documents
    receiver: pdf_to_image.documents
  - sender: pdf_to_image.base64_images
    receiver: llm.images
  - sender: prompt_builder.prompt
    receiver: llm.prompt
  - sender: image_downloader.documents
    receiver: answer_builder.documents
  - sender: prompt_builder.prompt
    receiver: answer_builder.prompt
  - sender: llm.replies
    receiver: answer_builder.replies

inputs:  # Define the inputs for your pipeline
    query:  # These components will receive the query as input
    - "bm25_retriever.query"
    - "query_embedder.text"
    - "ranker.query"
    - "prompt_builder.question"
    - "answer_builder.query"

    filters:  # These components will receive a potential query filter as input
    - "bm25_retriever.filters"
    - "embedding_retriever.filters"

outputs:  # Defines the output of your pipeline
    documents: "pdf_to_image.documents"  # The output of the pipeline is the retrieved documents
    answers: "answer_builder.answers"  # The output of the pipeline is the generated answers

Parameters

Inputs

Parameter	Type	Description
prompt	str	The prompt with instructions for the model.
images	List[Base64Image]	A list of Base64Images that represent the image content of the message. The base64 encoded images are passed to the large language model for text generation.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	A callback function to handle streaming chunks. To learn more about streaming, see Enable Streaming.
generation_kwargs	Optional[Dict[str, Any]]	Additional keyword arguments for text generation. These parameters potentially override the parameters passed in pipeline configuration.

Outputs

Parameter	Type	Description
replies	List[str]	A list of strings containing the generated responses.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
model	str		The model to use for text generation.
system_prompt	Optional[str]	None	A system prompt to use for role prompting.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	None	A callback function that is called when a new token is received from the stream. To learn more, see Enable Streaming.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
prompt	str		The prompt for the model.
images	List[Base64Image]		A list of Base64Images that represent the image content of the message.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	None	A callback function to handle streaming chunks. To learn more, see Enable Streaming.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for text generation. These parameters potentially override the parameters in pipeline configuration.