VertexAIImageQA

Answer questions about images using Google Vertex AI generative models.

Key Features

• Answers questions about images using Google Vertex AI visual question answering models
• Accepts an image as ByteStream and a question as string inputs
• Returns answers based on the image content
• Authenticates using Google Cloud Application Default Credentials (ADCs)

Configuration

1. Drag the VertexAIImageQA component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Enter your GCP project ID. Create a secret with the key GCP_PROJECT_ID. For detailed instructions, see Create Secrets.
   2. Optionally, enter the location. If not set, uses us-central1.
   3. The default model is imagetext.
4. Go to the Advanced tab to configure additional model keyword arguments.

Connections

VertexAIImageQA accepts an image (ByteStream) and a question (str) as inputs. It outputs answers as replies (a list of strings).

Connect an image source to the image input and a question string to the question input. Connect the replies output to AnswerBuilder.

Source Code

To check this component's source code, open question_answering.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  VertexAIImageQA:
    type: haystack_integrations.components.generators.google_vertex.question_answering.VertexAIImageQA
    init_parameters:
      model: imagetext

This pipeline uses VertexAIImageQA to answer questions about images:

# haystack-pipeline
components:
  VertexAIImageQA:
    type: haystack_integrations.components.generators.google_vertex.question_answering.VertexAIImageQA
    init_parameters:
      project_id:
      model: imagetext
      location:

  AnswerBuilder:
    type: haystack.components.builders.answer_builder.AnswerBuilder
    init_parameters:
      pattern:
      reference_pattern:

connections:
- sender: VertexAIImageQA.replies
  receiver: AnswerBuilder.replies

outputs:
  answers: AnswerBuilder.answers

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - VertexAIImageQA.question
  - AnswerBuilder.query

Parameters

Inputs

Parameter	Type	Description
image	ByteStream	The image to ask the question about.
question	str	The question to ask.

Outputs

Parameter	Type	Description
replies	List[str]	A list of answers to the question.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
project_id	Optional[str]	None	ID of the GCP project to use. By default, it is set during Google Cloud authentication.
model	str	imagetext	Name of the model to use.
location	Optional[str]	None	The default location to use when making API calls. If not set, uses us-central-1.
kwargs	Any		Additional keyword arguments to pass to the model. See the ImageTextModel.ask_question() documentation.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Description
image	ByteStream	The image to ask the question about.
question	str	The question to ask.

Related Information

• Create Secrets
• Google Cloud Application Default Credentials