DeepsetTogetherAIGenerator

Generate text using large language models hosted on Together AI.

Deprecation Notice

This component is deprecated. It will continue to work in your existing pipelines for now. You can replace it with the TogetherAIChatGenerator component.

DeepsetTogetherAIGenerator generates answers to queries using models hosted on Together AI. For a complete list of models you can use, check Together AI documentation.

Key Features

• Supports a wide range of open-source models hosted on Together AI.
• String-based input and output interface, compatible with PromptBuilder.
• Supports streaming responses token by token.
• Supports system prompts at both initialization time and run time.
• Customizable generation via generation_kwargs, including max_tokens, temperature, and more.

Configuration

1. Drag the DeepsetTogetherAIGenerator component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   1. Enter the model name in the Model field (for example, deepseek-ai/DeepSeek-R1).
   2. Make sure Haystack Platform is connected to Together AI. You'll need a Together AI API key. For details, see Use Together AI Models.
4. Go to the Advanced tab to configure system_prompt, streaming_callback, api_base_url, generation_kwargs, timeout, and max_retries.

Connections

DeepsetTogetherAIGenerator receives the prompt from PromptBuilder through its prompt input. It outputs generated text as a list of strings through its replies output, which you connect to DeepsetAnswerBuilder.

Usage Examples

Basic Configuration

  DeepsetTogetherAIGenerator:
    type: deepset_cloud_custom_nodes.generators.togetherai.DeepsetTogetherAIGenerator
    init_parameters:
      model: deepseek-ai/DeepSeek-R1
      api_key:
        type: env_var
        env_vars:
        - TOGETHERAI_API_KEY
        strict: false

Using the Component in a Pipeline

This query pipeline uses the DeepSeek-R1 model hosted on Together AI:

components:
  # ...
  prompt_builder:
      type: haystack.components.builders.prompt_builder.PromptBuilder
      init_parameters:
        template: |-
          You are a technical expert.
          You answer questions truthfully based on provided documents.
          For each document check whether it is related to the question.
          Only use documents that are related to the question to answer it.
          Ignore documents that are not related to the question.
          If the answer exists in several documents, summarize them.
          Only answer based on the documents provided. Don't make things up.
          If the documents can't answer the question or you are unsure say: 'The answer can't be found in the text'.
          These are the documents:
          {% for document in documents %}
          Document[{{ loop.index }}]:
          {{ document.content }}
          {% endfor %}
          Question: {{question}}
          Answer:

  llm:
    type: deepset_cloud_custom_nodes.generators.togetherai.DeepsetTogetherAIGenerator
    init_parameters:
      api_key: {"type": "env_var", "env_vars": ["TOGETHERAI_API_KEY"], "strict": false}
      model: deepseek-ai/DeepSeek-R1
      generation_kwargs:
        max_tokens: 650
        temperature: 0
        seed: 0
       
  answer_builder:
      type: haystack.components.builders.answer_builder.AnswerBuilder

connections:
  # ...
  - sender: prompt_builder.prompt
    receiver: llm.prompt
  - sender: llm.replies
    receiver: answer_builder.replies

Parameters

Inputs

Parameter	Type	Description
prompt	str	The prompt with instructions for the model.
system_prompt	Optional[str]	The system prompt to use for text generation. If omitted, the system prompt defined at initialization time is used.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	A callback function called when a new token is received from the stream. For more information, see Enable Streaming.
generation_kwargs	Optional[Dict[str, Any]]	Additional keyword arguments for text generation. These parameters potentially override the parameters in pipeline configuration.

Outputs

Parameter	Type	Description
replies	List[str]	A list of strings containing the generated responses.
meta	List[Dict[str, Any]]	A list of dictionaries containing the metadata for each response.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Secret	Secret.from_env_var('TOGETHERAI_API_KEY')	The together.ai API key to connect.
model	str	deepseek-ai/DeepSeek-R1	The name of the model to use.
api_base_url	Optional[str]	None	The base URL of the together.ai API.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	None	A callback function called when a new token is received from the stream. For more information, see Enable Streaming.
system_prompt	Optional[str]	None	The system prompt to use for text generation. If not provided, the system prompt is omitted, and the default system prompt of the model is used.
generation_kwargs	Optional[Dict[str, Any]]	None	Other parameters to use for the model, sent directly to the together.ai endpoint. See together.ai documentation for more details.
timeout	Optional[float]	None	Timeout for together.ai Client calls. If not set, it is inferred from the TOGETHERAI_TIMEOUT environment variable or set to 30.
max_retries	Optional[int]	None	Maximum retries to establish contact with together.ai if it returns an internal error. If not set, it is inferred from the TOGETHER_MAX_RETRIES environment variable or set to 5.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
prompt	str		The prompt with instructions for the model.
system_prompt	Optional[str]	None	The system prompt to use for text generation. If omitted, the system prompt defined at initialization time is used.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	None	A callback function that is called when a new token is received from the stream.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for text generation. For more details, refer to the together.ai documentation.

Related Information

• Use Together AI Models
• Add Integrations