TogetherAIGenerator

Generates text using large language models running on Together AI, using a simple prompt string rather than the chat message format.

Key Features

• Simple text generation interface without requiring chat message formatting.
• Access to a wide range of open-source LLMs through Together AI's platform.
• Optional system prompt support for setting model behavior.
• Streaming support for real-time token delivery.
• Customizable generation with generation_kwargs for temperature, token limits, and more.

Configuration

Authentication

To use this component, connect Haystack Platform with Together AI first. For detailed instructions, see Use Together AI Models.

1. Drag the TogetherAIGenerator component onto the canvas from the Component Library.
2. Click the component to open the configuration panel.
3. On the General tab:
   1. Enter the name of the Together AI model to use (for example, meta-llama/Llama-3.3-70B-Instruct-Turbo). For supported models, see Together AI documentation.
4. Go to the Advanced tab to configure the API key, API base URL, generation parameters, and streaming callback.

Connections

TogetherAIGenerator accepts a prompt string as input. It outputs a list of generated text strings and a list of metadata dictionaries.

Connect a PromptBuilder to its prompt input. Connect its replies output to a DeepsetAnswerBuilder.

Usage Example

This is an example RAG pipeline with TogetherAIGenerator and DeepsetAnswerBuilder:

components:
  bm25_retriever:
    type: haystack_integrations.components.retrievers.opensearch.bm25_retriever.OpenSearchBM25Retriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: 'Standard-Index-English'
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
      top_k: 20
      fuzziness: 0

  query_embedder:
    type: deepset_cloud_custom_nodes.embedders.nvidia.text_embedder.DeepsetNvidiaTextEmbedder
    init_parameters:
      normalize_embeddings: true
      model: intfloat/e5-base-v2

  embedding_retriever:
    type: haystack_integrations.components.retrievers.opensearch.embedding_retriever.OpenSearchEmbeddingRetriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: 'Standard-Index-English'
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
      top_k: 20

  document_joiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate

  ranker:
    type: deepset_cloud_custom_nodes.rankers.nvidia.ranker.DeepsetNvidiaRanker
    init_parameters:
      model: intfloat/simlm-msmarco-reranker
      top_k: 8

  meta_field_grouping_ranker:
    type: haystack.components.rankers.meta_field_grouping_ranker.MetaFieldGroupingRanker
    init_parameters:
      group_by: file_id
      subgroup_by:
      sort_docs_by: split_id

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

  PromptBuilder:
    type: haystack.components.builders.prompt_builder.PromptBuilder
    init_parameters:
      template: "You are a helpful assistant answering the user's questions based on the provided documents.\nDo not use your own knowledge.\n\nProvided documents:\n{% for document in documents %}\nDocument [{{ loop.index }}]:\n{{ document.content }}\n{% endfor %}\n\nQuestion: {{ query }}\nAnswer:"

  TogetherAIGenerator:
    type: haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - TOGETHER_API_KEY
        strict: false
      model: meta-llama/Llama-3.3-70B-Instruct-Turbo
      streaming_callback:
      api_base_url: https://api.together.xyz/v1
      system_prompt:
      generation_kwargs:
      timeout:
      max_retries:

connections:
- sender: bm25_retriever.documents
  receiver: document_joiner.documents
- sender: query_embedder.embedding
  receiver: embedding_retriever.query_embedding
- sender: embedding_retriever.documents
  receiver: document_joiner.documents
- sender: document_joiner.documents
  receiver: ranker.documents
- sender: ranker.documents
  receiver: meta_field_grouping_ranker.documents
- sender: meta_field_grouping_ranker.documents
  receiver: answer_builder.documents
- sender: meta_field_grouping_ranker.documents
  receiver: PromptBuilder.documents
- sender: PromptBuilder.prompt
  receiver: TogetherAIGenerator.prompt
- sender: TogetherAIGenerator.replies
  receiver: answer_builder.replies

inputs:
  query:
  - "bm25_retriever.query"
  - "query_embedder.text"
  - "ranker.query"
  - "answer_builder.query"
  - "PromptBuilder.query"
  filters:
  - "bm25_retriever.filters"
  - "embedding_retriever.filters"

outputs:
  documents: "meta_field_grouping_ranker.documents"
  answers: "answer_builder.answers"

max_runs_per_component: 100

metadata: {}

Parameters

Inputs

Parameter	Type	Default	Description
prompt	str		The input prompt string for text generation.
system_prompt	Optional[str]	None	An optional system prompt to provide context or instructions for the generation.
streaming_callback	Optional[StreamingCallbackT]	None	A callback function called when a new token is received from the stream.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for text generation. These parameters override the parameters in pipeline configuration.

Outputs

Parameter	Type	Default	Description
replies	List[str]		A list of generated text completions as strings.
meta	List[Dict[str, Any]]		A list of metadata dictionaries containing information about each generation, including model name, finish reason, and token usage statistics.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
api_key	Secret	Secret.from_env_var('TOGETHER_API_KEY')	The Together AI API key.
model	str	meta-llama/Llama-3.3-70B-Instruct-Turbo	The name of the model to use.
api_base_url	Optional[str]	https://api.together.xyz/v1	The base URL of the Together AI API.
streaming_callback	Optional[StreamingCallbackT]	None	A callback function called when a new token is received from the stream. The callback function accepts StreamingChunk as an argument.
system_prompt	Optional[str]	None	The system prompt to use for text generation. If not provided, the system prompt is omitted and the default system prompt of the model is used.
generation_kwargs	Optional[Dict[str, Any]]	None	Other parameters to use for the model. These parameters are sent directly to the Together AI endpoint. See Together AI documentation for more details. Supported parameters include: max_tokens (maximum number of tokens the output text can have), temperature (sampling temperature for creativity control), top_p (nucleus sampling probability mass), n (number of completions to generate for each prompt), stop (sequences after which the LLM should stop generating tokens), presence_penalty (penalty for tokens already present), frequency_penalty (penalty for frequently generated tokens), logit_bias (add logit bias to specific tokens).
timeout	Optional[float]	None	Timeout for Together AI client calls. If not set, it is inferred from the OPENAI_TIMEOUT environment variable or set to 30.
max_retries	Optional[int]	None	Maximum retries to establish contact with Together AI if it returns an internal error. If not set, it is inferred from the OPENAI_MAX_RETRIES environment variable or set to five.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
prompt	str		The input prompt string for text generation.
system_prompt	Optional[str]	None	An optional system prompt to provide context or instructions for the generation. If not provided, the system prompt set during initialization is used.
streaming_callback	Optional[StreamingCallbackT]	None	A callback function called when a new token is received from the stream. If provided, this overrides the streaming_callback set during initialization.
generation_kwargs	Optional[Dict[str, Any]]	None	Additional keyword arguments for text generation. These parameters override the parameters passed during initialization. Supported parameters include temperature, max_tokens, top_p, and others.