Using LLMs hosted by the model provider is often the best option as it guarantees the model has sufficient infrastructure to run efficiently. deepset Cloud pipelines can use LLMs hosted in:

• Cohere
• OpenAI
• Amazon Sagemaker
• Amazon Bedrock

You use models directly in your pipelines through PromptNode. You simply pass the model name in PromptNode's model_name_or_path parameter. Check the sections below for details on how to use a model from a particular provider.

Using Cohere Models

1. Obtain the production key from your Cohere account.

2. Connect Cohere to deepset Cloud:

3. 1. Click your name in the top right corner and select Connections.
      
   2. Click Connect next to a model provider.
   3. Enter your user access token and submit it.

4. In your pipeline, pass the name of the Cohere model you want to use in the model_name_or_path parameter of PromptNode. For example, to use command-light, add:

components:
- name: PromptNode
  type: PromptNode
  params:
      model_name_or_path: command-light
      model_kwargs: # Specifies additional model settings
        temperature: 0 # Lower temperature works best for fact-based qa
        ...
     

Using OpenAI Models

1. Create a secret API key for your OpenAI account.
2. Connect OpenAI to deepset Cloud:
3. 1. Click your name in the top right corner and select Connections.
      
   2. Click Connect next to a model provider.
   3. Enter your user access token and submit it.

3. In your pipeline, pass the name of the OpenAI model you want to use in the model_name_or_path parameter of PromptNode. For example, to use gpt-3.5-turbo, add:

components:
- name: PromptNode
  type: PromptNode
  params:
      model_name_or_path: gpt-3.5-turbo
      model_kwargs: # Specifies additional model settings
        temperature: 0 # Lower temperature works best for fact-based qa
        ...

Using OpenAI Models Hosted on Microsoft Azure

To use a remote model from Microsoft Azure, pass the API version, base URL, and deployment name in model_kwargs:

...
components:
 - name: PromptNode
   type: PromptNode
   params:
      model_name_or_path: gpt-3.5-turbo
      api_key: <azure_openai_api_key>
      default_prompt_template: <your_prompt_template>
      model_kwargs:
      	api_version: 2022-12-01
        azure_base_url: https://<your-endpoint>.openai.azure.com
        azure_deployment_name: <your-deployment-name>

You can find the azure_base_url parameter in your Azure account's Keys and Endpoint tab. You choose the azure_deployment_name when you deploy a model through your Azure account in the Model deployments tab. For available models and versions for the service, check Azure documentation.

Using Amazon Sagemaker Models

You can use LLMs hosted on Amazon Sagemaker in PromptNode. Contact your deepset Cloud representative to set up the model for you. Once it's ready, you'll get the model name that you then pass in the model_name_or_path parameter of PromptNode, like this:

...
components:
  - name: PromptNode 
    type: PromptNode
    params:
      model_name_or_path: <the_model_name_you_got_from_deepset_Cloud_rep>
      model_kwargs:
      	temperature: 0.6 #these are additional model parameters that you can configure
        ...

Using Models Hosted on Amazon Bedrock

You can use text generation models hosted on deepset's Bedrock account or on your private Bedrock account.

Using Models Hosted in deepset's Bedrock Account

1. Check the ID of the model you want to use. You can find the IDs in Amazon Documentation.
2. Pass the model ID preceded by deepset-cloud in the model_name_or_path parameter of PromptNode, like this:
   YAML

   components:
     - name: PromptNode 
       type: PromptNode
       params:
         model_name_or_path: deepset-cloud-anthropic.claude-v2 #This PromptNode uses the Claude 2 model hosted on Bedrock
         model_kwargs:
         	temperature: 0.6
           ...
   

Using Models Hosted in Your Private Bedrock Account

1. In Amazon Identity and Access Management (IAM), create a dedicated user to connect to deepset Cloud. When creating the user, make sure you save the access key ID and secret key. For detailed instructions, see Amazon IAM User Guide.
2. Give the user the following permission policies:
   JSON

   "bedrock:ListFoundationModels",
   "bedrock:GetFoundationModel",
   "bedrock:InvokeModel",
   "bedrock:InvokeModelWithResponseStream",
   "bedrock:GetFoundationModelAvailability",
   "bedrock:GetCustomModel",
   "bedrock:ListCustomModels"
   

3. Go back to deepset Cloud, and connect Bedrock to deepset Cloud:
   1. 1. Click your initials in the top right corner and select Connections.
         
      2. Click Connect next to a model provider.
      3. Enter your user access token and submit it.

You're now ready to use the model in your pipeline.

4. Pass the model ID in the model_name_or_path parameter of PromptNode, like this:

components:
  - name: PromptNode 
    type: PromptNode
    params:
      model_name_or_path: anthropic.claude-v2 #This PromptNode uses the Claude 2 model hosted on Bedrock
      model_kwargs:
      	temperature: 0.6
        ...