VertexAIImageCaptioner

Generate captions for images using Google Vertex AI imagetext model.

Key Features

• Generates descriptive captions for images using the Vertex AI imagetext model.
• Accepts images as ByteStream input.
• Authenticates using Google Cloud Application Default Credentials.
• Returns a list of captions for each input image.

Configuration

Authentication

This component authenticates using Google Cloud Application Default Credentials (ADCs). For more information, see the official Google documentation.

Create secrets for GCP_PROJECT_ID and optionally GCP_DEFAULT_REGION. For detailed instructions on creating secrets, see Create Secrets.

1. Drag the VertexAIImageCaptioner component onto the canvas from the Component Library.
2. Click the component to open the configuration panel.
3. On the General tab:
   1. Enter the model name (for example, imagetext).
4. Go to the Advanced tab to configure the project ID, location, and additional model kwargs.

Connections

VertexAIImageCaptioner accepts an image as ByteStream input (image). It outputs a list of generated captions (captions).

Connect the captions output to AnswerBuilder to format the final answer.

Usage Example

This pipeline uses VertexAIImageCaptioner to generate captions for images:

components:
  VertexAIImageCaptioner:
    type: haystack_integrations.components.generators.google_vertex.captioner.VertexAIImageCaptioner
    init_parameters:
      project_id:
      model: imagetext
      location:

  AnswerBuilder:
    type: haystack.components.builders.answer_builder.AnswerBuilder
    init_parameters:
      pattern:
      reference_pattern:

connections:
  - sender: VertexAIImageCaptioner.captions
    receiver: AnswerBuilder.replies

inputs:
  image:
    - VertexAIImageCaptioner.image
  query:
    - AnswerBuilder.query

outputs:
  answers: AnswerBuilder.answers

max_runs_per_component: 100

metadata: {}

Parameters

Inputs

Parameter	Type	Default	Description
image	ByteStream		The image to generate captions for.

Outputs

Parameter	Type	Default	Description
captions	List[str]		A list of captions generated by the model.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
project_id	Optional[str]	None	ID of the GCP project to use. By default, it is set during Google Cloud authentication.
model	str	imagetext	Name of the model to use.
location	Optional[str]	None	The default location to use when making API calls. If not set, uses us-central-1.
kwargs	Any		Additional keyword arguments to pass to the model. See the ImageTextModel.get_captions() documentation.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
image	ByteStream		The image to generate captions for.