Models in Your Pipelines

You use models through pipeline nodes. There are a couple of pipeline components that use models. Have a look at this table for an overview of model applications and components that use them:

To use a model, simply pass its name as a parameter to the node (it's usually the model parameter). If you're using a proprietary model, you can either pass the API key to the node or connect deepset Cloud to the model provider. deepset Cloud takes care of loading the models.

You can run models locally or remotely. Smaller models, like question answering or embedding models, are fast when run locally. Large models, like GPT-3.5 or GPT-4, are faster to run remotely as they need optimized hardware. To run a model remotely, you may need to pass additional parameters in model kwargs. For detailed instructions, see the Pipeline Components for the component that uses the model.

When using LLMs with Generators, you can specify all additional model settings, like temperature, in the kwargs parameter, for example:

components:
  generator:
      type: haystack_integrations.components.generators.amazon_bedrock.generator.AmazonBedrockGenerator
      init_parameters:
        aws_access_key_id: {"type": "env_var", "env_vars": ["AWS_ACCESS_KEY_ID"], "strict": False}
        aws_secret_access_key: {"type": "env_var", "env_vars": ["AWS_SECRED_ACCESS_KEY"], "strict": False}
        aws_region_name: {"type": "env_var", "env_vars": ["AWS_DEFAULT_REGION"], "strict": False}
        model: "anthropic.claude-v2"
        kwargs:
          temperature: 0.0

Recommended Models

Larger models are generally more accurate at the cost of speed.

If you don't know which model to start with, you can use one of the models we recommend.

Large Language Models for RAG

This table lists the models that we recommend for retrieval augmented generation (RAG) QA. You can use them with Generators in your pipelines.

For an overview of the models, see Large Language Models Overview.

Hosted LLMs

Currently, supported connections to model providers are:

• Amazon Bedrock and SageMaker
• Azure OpenAI
• Cohere
• Google AI
• Hugging Face (only needed for private models hosted there)
• Nvidia
• OpenAI
• SearchAPI
• Voyage AI

Supported connections to data processing services and data sources are:

• Azure Document Intelligence
• DeepL
• Snowflake
• unstructured.io

Reader Models for Question Answering

This table describes the models that we recommend for the question answering task. You can use them with your Readers.

You can also view state-of-the-art question answering models on the Hugging Face leaderboard.

Embedding Models

This table describes the embedding models we recommend for calculating vector representations of text. You can use them with dedicated Embedders.

*In language models, vector dimension refers to the number of elements a vector representing a word contains, capturing its meaning and usage. The more dimensions, or numbers, in this vector, the more detail the model can understand about the word, though it also makes the model more complex to handle. It's best to try out different models and see what works best for your data.

Ranker Models

This table lists models to use with Rankers to rank documents.