# Use NVIDIA Models

Use models from NVIDIA in your pipelines.

***

## About This Task

You can use self-hosted models from the[ NVIDIA API catalog](https://docs.api.nvidia.com/nim/reference/models-1) or models deployed on NVIDIA NIM. 

## Prerequisites

You need an active NVIDIA API key. For details on how to obtain it, see [NVIDIA documentation](https://org.ngc.nvidia.com/setup/personal-keys). 

## Use NVIDIA Models

First, connect <ProductName /> to NVIDIA through the Integrations page. You can set up the connection for a single workspace or for the whole organization:
<AddIntegration />

Then, add a component that uses a model hosted on NVIDIA to your pipeline. Here are the components by the model type they use:

- Embedding models:
  - [`NvidiaTextEmbedder`](/docs/reference/pipeline-components/integrations/nvidia/NvidiaTextEmbedder.mdx): Uses an embedding model to calculate vector representations of text. Often used in query pipelines to embed the query string and send it to an embedding retriever.
  - [`NvidiaDocumentEmbedder`](/docs/reference/pipeline-components/integrations/nvidia/NvidiaDocumentEmbedder.mdx): Uses an embedding model to calculate embeddings of documents. Often used in indexes to embed documents and send them to DocumentWriter. 
    <EmbeddingInfoCallout />

- LLMs:
  - [`NvidiaChatGenerator`](/docs/reference/pipeline-components/integrations/nvidia/NvidiaChatGenerator.mdx): Generates text using models from NVIDIA. Often used in RAG pipelines.

- Ranking models:
  - [`NvidiaRanker`](/docs/reference/pipeline-components/integrations/nvidia/NvidiaRanker.mdx): Ranks documents based on their similarity to the query using NVIDIA ranking models.

## Usage Examples

This is a YAML example of how to use embedding models and an LLM hosted on NVIDIA in an index and a query pipeline (each in a separate tab):

<Tabs>
<TabItem value="query" label="Query Pipeline" default>

```yaml Pipeline
# haystack-pipeline
components:
  # ...
  query_embedder:
      type: haystack_integrations.components.embedders.nvidia.text_embedder.NvidiaTextEmbedder
      init_parameters: 
        api_url: "https://ai.api.nvidia.com/v1/retrieval/nvidia" # custom API URL for NVIDIA NIM.
        model: "NV-Embed-QA" # the model to use
        
  retriever:
      type: haystack_integrations.components.retrievers.opensearch.embedding_retriever.OpenSearchEmbeddingRetriever
      init_parameters: 
        document_store:
          init_parameters:
            use_ssl: True
            verify_certs: False
            http_auth:
              - "${OPENSEARCH_USER}"
              - "${OPENSEARCH_PASSWORD}"
          type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        top_k: 20 
  prompt_builder:
      type: haystack.components.builders.chat_prompt_builder.ChatPromptBuilder
      init_parameters:
        template: >
          - _content:
                  - text: |-
                      You are a technical expert.
                      You answer questions truthfully based on provided documents.
                      For each document check whether it is related to the question.
                      Only use documents that are related to the question to answer it.
                      Ignore documents that are not related to the question.
                      If the answer exists in several documents, summarize them.
                      Only answer based on the documents provided. Don't make things up.
                      If the documents can't answer the question or you are unsure say: 'The answer can't be found in the text'.
                      These are the documents:
                      {% for document in documents %}
                      Document[{{ loop.index }}]:
                      {{ document.content }}
                      {% endfor %}
                      Question: {{ query }}
                  _role: user
        required_variables:
        variables:

  generator:
      type: haystack_integrations.components.generators.nvidia.chat.chat_generator.NvidiaChatGenerator
      init_parameters:
          model: "meta/llama3-70b-instruct" # here, pass the name of the model to use
          api_base_url: "https://integrate.api.nvidia.com/v1"
          generation_kwargs:
            temperature: 0.2
            top_p: 0.7
            max_tokens: 1024

  output_adapter:
      type: haystack.components.converters.output_adapter.OutputAdapter
      init_parameters:
        template: '{{ replies[0] }}'
        output_type: List[str]
        custom_filters:
        unsafe: false
                 
  answer_builder:
      init_parameters: {}
      type: haystack.components.builders.answer_builder.AnswerBuilder

connections:
  # ...
 - sender: query_embedder.embedding
   receiver: retriever.query_embedding
 - sender: retriever.documents
   receiver: prompt_builder.documents
 - sender: prompt_builder.prompt
   receiver: generator.messages
 - sender: generator.replies
   receiver: output_adapter.replies
 - sender: output_adapter.output
   receiver: answer_builder.replies
```
</TabItem>
<TabItem value="index" label="Index">
```yaml 
components:
  # ...
    splitter:
      type: haystack.components.preprocessors.document_splitter.DocumentSplitter
      init_parameters:
        split_by: word
        split_length: 250
        split_overlap: 30

    document_embedder:
      type: haystack_integrations.components.embedders.nvidia.document_embedder.NvidiaDocumentEmbedder
      init_parameters:
        api_url: "https://ai.api.nvidia.com/v1/retrieval/nvidia" # A required custom NVIDIA API URL for NVIDIA NIM
        model: "NV-Embed-QA" # the model to use
    
    writer:
      type: haystack.components.writers.document_writer.DocumentWriter
      init_parameters:
        document_store:
          type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
          init_parameters:
            embedding_dim: 768
            similarity: cosine
        policy: OVERWRITE
        
connections:  # Defines how the components are connected
  # ...
  - sender: splitter.documents
    receiver: document_embedder.documents
  - sender: document_embedder.documents
    receiver: writer.documents
```
</TabItem>
</Tabs>

## Related Information

- [Add Custom Model Definitions](/docs/how-to-guides/managing-access/add-custom-model-definitions.mdx)
- [LLM](/docs/reference/pipeline-components/ai/LLM.mdx)