Use Together AI Models

Use models, such as DeepSeek-R1, hosted on Together AI.

---

About This Task

You can use models hosted on Together AI in your deepset pipelines. For a complete list of models, see the Inference section of Together AI documentation.

Prerequisites

You need an API key from an active Together AI account. For details on obtaining it, see Together AI Quickstart.

Use Models Hosted on Together AI

First, connect deepset AI Platform to Together AI through the Integrations page. You can set up the connection for a single workspace or for the whole organization:

Add Workspace-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Workspace>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration

1. Click your profile icon and choose Settings.
2. Go to Organization>Integrations.
3. Find the provider you want to connect and click Connect next to them.
4. Enter the API key and any other required details.
5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.

Then, add the TogetherAIGenerator component to your pipeline and specify the path to the model you want to use.

Usage Examples

This is an example of a RAG pipeline that uses the DeepSeek-R1 model hosted on Together AI:

# If you need help with the YAML format, have a look at https://docs.cloud.deepset.ai/docs/create-a-pipeline-in-pipeline-builder#/.
# This section defines components that you want to use in your pipelines. Each component must have a name and a type. You can also set the component's parameters here.
# The name is up to you, you can give your component a friendly name. You then use components' names when specifying the connections in the pipeline.
# Type is the class path of the component. You can check the type on the component's documentation page.
components:
  retriever: # Selects the most similar documents from the document store
    type: haystack_integrations.components.retrievers.opensearch.open_search_hybrid_retriever.OpenSearchHybridRetriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          index: ''
          max_chunk_bytes: 104857600
          embedding_dim: 768
          return_embedding: false
          method:
          mappings:
          settings:
          create_index: true
          http_auth:
          use_ssl:
          verify_certs:
          timeout:
      top_k: 20 # The number of results to return
      fuzziness: 0
      embedder:
        type: deepset_cloud_custom_nodes.embedders.nvidia.text_embedder.DeepsetNvidiaTextEmbedder
        init_parameters:
          normalize_embeddings: true
          model: intfloat/e5-base-v2

  ranker:
    type: deepset_cloud_custom_nodes.rankers.nvidia.ranker.DeepsetNvidiaRanker
    init_parameters:
      model: intfloat/simlm-msmarco-reranker
      top_k: 8

  meta_field_grouping_ranker:
    type: haystack.components.rankers.meta_field_grouping_ranker.MetaFieldGroupingRanker
    init_parameters:
      group_by: file_id
      subgroup_by:
      sort_docs_by: split_id

  prompt_builder:
    type: haystack.components.builders.prompt_builder.PromptBuilder
    init_parameters:
      required_variables: "*"
      template: |-
        You are a technical expert.
        You answer questions truthfully based on provided documents.
        Ignore typing errors in the question.
        For each document check whether it is related to the question.
        Only use documents that are related to the question to answer it.
        Ignore documents that are not related to the question.
        If the answer exists in several documents, summarize them.
        Only answer based on the documents provided. Don't make things up.
        Just output the structured, informative and precise answer and nothing else.
        If the documents can't answer the question, say so.
        Always use references in the form [NUMBER OF DOCUMENT] when using information from a document, e.g. [3] for Document [3] .
        Never name the documents, only enter a number in square brackets as a reference.
        The reference must only refer to the number that comes in square brackets after the document.
        Otherwise, do not use brackets in your answer and reference ONLY the number of the document without mentioning the word document.

        These are the documents:
        {%- if documents|length > 0 %}
        {%- for document in documents %}
        Document [{{ loop.index }}] :
        Name of Source File: {{ document.meta.file_name }}
        {{ document.content }}
        {% endfor -%}
        {%- else %}
        No relevant documents found.
        Respond with "Sorry, no matching documents were found, please adjust the filters or try a different question."
        {% endif %}

        Question: {{ question }}
        Answer:

  llm:
    type: haystack_integrations.components.generators.togetherai.generator.TogetherAIGenerator
    init_parameters:
      api_key: {"type": "env_var", "env_vars": ["TOGETHER_API_KEY"], "strict": false}
      model: deepseek-ai/DeepSeek-R1
      generation_kwargs:
        max_tokens: 650
        temperature: 0
        seed: 0

  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

  attachments_joiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate
      weights:
      top_k:
      sort_by_score: true

  multi_file_converter:
    type: haystack.core.super_component.super_component.SuperComponent
    init_parameters:
      input_mapping:
        sources:
        - file_classifier.sources
      is_pipeline_async: false
      output_mapping:
        score_adder.output: documents
      pipeline:
        components:
          file_classifier:
            type: haystack.components.routers.file_type_router.FileTypeRouter
            init_parameters:
              mime_types:
              - text/plain
              - application/pdf
              - text/markdown
              - text/html
              - application/vnd.openxmlformats-officedocument.wordprocessingml.document
              - application/vnd.openxmlformats-officedocument.presentationml.presentation
              - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
              - text/csv

          text_converter:
            type: haystack.components.converters.txt.TextFileToDocument
            init_parameters:
              encoding: utf-8

          pdf_converter:
            type: haystack.components.converters.pdfminer.PDFMinerToDocument
            init_parameters:
              line_overlap: 0.5
              char_margin: 2
              line_margin: 0.5
              word_margin: 0.1
              boxes_flow: 0.5
              detect_vertical: true
              all_texts: false
              store_full_path: false

          markdown_converter:
            type: haystack.components.converters.txt.TextFileToDocument
            init_parameters:
              encoding: utf-8

          html_converter:
            type: haystack.components.converters.html.HTMLToDocument
            init_parameters:
              # A dictionary of keyword arguments to customize how you want to extract content from your HTML files.
              # For the full list of available arguments, see
              # the [Trafilatura documentation](https://trafilatura.readthedocs.io/en/latest/corefunctions.html#extract).
              extraction_kwargs:
                output_format: markdown # Extract text from HTML. You can also also choose "txt"
                target_language:       # You can define a language (using the ISO 639-1 format) to discard documents that don't match that language.
                include_tables: true  # If true, includes tables in the output
                include_links: true  # If true, keeps links along with their targets

          docx_converter:
            type: haystack.components.converters.docx.DOCXToDocument
            init_parameters:
              link_format: markdown

          pptx_converter:
            type: haystack.components.converters.pptx.PPTXToDocument
            init_parameters: {}

          xlsx_converter:
            type: haystack.components.converters.xlsx.XLSXToDocument
            init_parameters: {}

          csv_converter:
            type: haystack.components.converters.csv.CSVToDocument
            init_parameters:
              encoding: utf-8

          splitter:
            type: haystack.components.preprocessors.document_splitter.DocumentSplitter
            init_parameters:
              split_by: word
              split_length: 250
              split_overlap: 30
              respect_sentence_boundary: true
              language: en

          score_adder:
            type: haystack.components.converters.output_adapter.OutputAdapter
            init_parameters:
              template: |
                {%- set scored_documents = [] -%}
                {%- for document in documents -%}
                  {%- set doc_dict = document.to_dict() -%}
                  {%- set _ = doc_dict.update({'score': 100.0}) -%}
                  {%- set scored_doc = document.from_dict(doc_dict) -%}
                  {%- set _ = scored_documents.append(scored_doc) -%}
                {%- endfor -%}
                {{ scored_documents }}
              output_type: List[haystack.Document]
              custom_filters:
              unsafe: true

          text_joiner:
            type: haystack.components.joiners.document_joiner.DocumentJoiner
            init_parameters:
              join_mode: concatenate
              sort_by_score: false

          tabular_joiner:
            type: haystack.components.joiners.document_joiner.DocumentJoiner
            init_parameters:
              join_mode: concatenate
              sort_by_score: false
        connections:
        - sender: file_classifier.text/plain
          receiver: text_converter.sources
        - sender: file_classifier.application/pdf
          receiver: pdf_converter.sources
        - sender: file_classifier.text/markdown
          receiver: markdown_converter.sources
        - sender: file_classifier.text/html
          receiver: html_converter.sources
        - sender: file_classifier.application/vnd.openxmlformats-officedocument.wordprocessingml.document
          receiver: docx_converter.sources
        - sender: file_classifier.application/vnd.openxmlformats-officedocument.presentationml.presentation
          receiver: pptx_converter.sources
        - sender: file_classifier.application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
          receiver: xlsx_converter.sources
        - sender: file_classifier.text/csv
          receiver: csv_converter.sources
        - sender: text_joiner.documents
          receiver: splitter.documents
        - sender: text_converter.documents
          receiver: text_joiner.documents
        - sender: pdf_converter.documents
          receiver: text_joiner.documents
        - sender: markdown_converter.documents
          receiver: text_joiner.documents
        - sender: html_converter.documents
          receiver: text_joiner.documents
        - sender: pptx_converter.documents
          receiver: text_joiner.documents
        - sender: docx_converter.documents
          receiver: text_joiner.documents
        - sender: xlsx_converter.documents
          receiver: tabular_joiner.documents
        - sender: csv_converter.documents
          receiver: tabular_joiner.documents
        - sender: splitter.documents
          receiver: tabular_joiner.documents
        - sender: tabular_joiner.documents
          receiver: score_adder.documents

connections:  # Defines how the components are connected
- sender: retriever.documents
  receiver: ranker.documents
- sender: ranker.documents
  receiver: meta_field_grouping_ranker.documents
- sender: prompt_builder.prompt
  receiver: llm.prompt
- sender: prompt_builder.prompt
  receiver: answer_builder.prompt
- sender: llm.replies
  receiver: answer_builder.replies

- sender: multi_file_converter.documents
  receiver: attachments_joiner.documents
- sender: meta_field_grouping_ranker.documents
  receiver: attachments_joiner.documents
- sender: attachments_joiner.documents
  receiver: answer_builder.documents
- sender: attachments_joiner.documents
  receiver: prompt_builder.documents

inputs:  # Define the inputs for your pipeline
  query:  # These components will receive the query as input
  - "retriever.query"
  - "ranker.query"
  - "prompt_builder.question"
  - "answer_builder.query"

  filters:  # These components will receive a potential query filter as input
  - "retriever.filters_bm25"
  - "retriever.filters_embedding"

  files:
  - multi_file_converter.sources

outputs:  # Defines the output of your pipeline
  documents: "attachments_joiner.documents"  # The output of the pipeline is the retrieved documents
  answers: "answer_builder.answers"  # The output of the pipeline is the generated answers

max_runs_per_component: 100

metadata: {}