DeepsetTogetherAIGenerator

Generate text using large language models hosted on Together AI.

Basic Information

• Type: deepset_cloud_custom_nodes.generators.togetherai.DeepsetTogetherAIGenerator
• Components it can connect with:
  • PromptBuilder: DeepsetTogetherAIGenerator receives the prompt from PromptBuilder.
  • DeepsetAnswerBuilder: DeepsetTogetherAIGenerator sends the generated replies to DeepsetAnswerBuilder, which uses them to build GeneratedAnswer objects.

Inputs

Parameter	Type	Default	Description
prompt	str		The prompt with instructions for the model.
system_prompt	Optional[str]	None	The system prompt to use for text generation. If this run time system prompt is omitted, the system prompt, if defined at initialisation time, is used.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	None	A callback function called when a new token is received from the stream. For more information, see Enable Streaming.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for text generation. These parameters potentially override the parameters in pipeline configuration. For more details on supported parameters, refer to the together.ai documentation.

Outputs

Parameter	Type	Default	Description
replies	List[str]		A list of strings containing the generated responses.
meta	List[Dict[str, Any]]		A list dictionaries containing the metadata for each response.

Overview

DeepsetTogetherAIGenerator generates answers to queries using models hosted on Together AI. For a complete list of models you can use, check Together AI documentation.

Authentication

You need an API key from Together AI to use their models. For details on obtaining it, see Together AI Quickstart.

Once you have the API key, connect deepset AI Platform to Together AI on the Integrations page:

Add Workspace-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Workspace>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Organization>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.

For detailed instructions, see Use Together AI Models.

Usage Example

Initializing the Component

components:
  DeepsetTogetherAIGenerator:
    type: generators.togetherai.DeepsetTogetherAIGenerator
    init_parameters:

Using the Component in a Pipeline

This query pipeline uses the DeepSeek-R§ model hosted on Together AI:

components:
...
  prompt_builder:
      type: haystack.components.builders.prompt_builder.PromptBuilder
      init_parameters:
        template: |-
          You are a technical expert.
          You answer questions truthfully based on provided documents.
          For each document check whether it is related to the question.
          Only use documents that are related to the question to answer it.
          Ignore documents that are not related to the question.
          If the answer exists in several documents, summarize them.
          Only answer based on the documents provided. Don't make things up.
          If the documents can't answer the question or you are unsure say: 'The answer can't be found in the text'.
          These are the documents:
          {% for document in documents %}
          Document[{{ loop.index }}]:
          {{ document.content }}
          {% endfor %}
          Question: {{question}}
          Answer:

  llm:
    type: deepset_cloud_custom_nodes.generators.togetherai.DeepsetTogetherAIGenerator
    init_parameters:
      api_key: {"type": "env_var", "env_vars": ["TOGETHERAI_API_KEY"], "strict": false}
      model: deepseek-ai/DeepSeek-R1
      generation_kwargs:
        max_tokens: 650
        temperature: 0
        seed: 0
       
    answer_builder:
       type: haystack.components.builders.answer_builder.AnswerBuilder
      ...

connections:
...
 - sender: prompt_builder.prompt
   receiver: llm.prompt
 - sender: llm.replies
   receiver: answer_builder.replies
   ...

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Secret	Secret.from_env_var('TOGETHERAI_API_KEY')	The together.ai API key to connect.
model	str	deepseek-ai/DeepSeek-R1	The name of the model to use.
api_base_url	Optional[str]	None	The base URL of the together.ai API.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	None	A callback function called when a new token is received from the stream. The callback function accepts StreamingChunk as an argument. For more information, see Enable Streaming.
system_prompt	Optional[str]	None	The system prompt to use for text generation. If not provided, the system prompt is omitted, and the default system prompt of the model is used.
generation_kwargs	Optional[Dict[str, Any]]	None	Other parameters to use for the model. These parameters are all sent directly to the together.ai endpoint. See together.ai documentation for more details. Some of the supported parameters: - max_tokens: The maximum number of tokens the output text can have. - temperature: What sampling temperature to use. Higher values mean the model will take more risks. Try 0.9 for more creative applications and 0 (argmax sampling) for ones with a well-defined answer. - top_p: An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with top_p probability mass. So, 0.1 means only the tokens comprising the top 10% probability mass are considered. - n: How many completions to generate for each prompt. For example, if the LLM gets 3 prompts and n is 2, it will generate two completions for each of the three prompts, ending up with 6 completions in total. - stop: One or more sequences after which the LLM should stop generating tokens. - presence_penalty: What penalty to apply if a token is already present at all. Bigger values mean the model will be less likely to repeat the same token in the text. - frequency_penalty: What penalty to apply if a token has already been generated in the text. Bigger values mean the model will be less likely to repeat the same token in the text. - logit_bias: Add a logit bias to specific tokens. The keys of the dictionary are tokens, and the values are the bias to add to that token.
timeout	Optional[float]	None	Timeout for together.ai Client calls, if not set it is inferred from the TOGETHERAI_TIMEOUT environment variable or set to 30.
max_retries	Optional[int]	None	Maximum retries to establish contact with together.ai if it returns an internal error, if not set it is inferred from the TOGETHER_MAX_RETRIES environment variable or set to 5.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
prompt	str		The prompt with instructions for the model.
system_prompt	Optional[str]	None	The system prompt to use for text generation. If this run time system prompt is omitted, the system prompt, if defined at initialisation time, is used.
streaming_callback	Optional[Callable[[StreamingChunk], None]]	None	A callback function that is called when a new token is received from the stream.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for text generation. These parameters potentially override the parameters in pipeline configuration. For more details on the parameters supported by the together.ai API, refer to the together.ai documentation.