PromptNode Parameters

Check the init and runtime parameters you can specify for PromptNode types, PromptTemplate, and PromptModel.


YAML Init Parameters

These are the parameters you can specify in pipeline YAML:

PromptNode Parameters

ParameterTypePossible ValuesDescription
model_name_or_pathStringDefault: google/flan-t5-baseThe name of the model you want to use with the PromptNode or the path to a locally stored model.
To use an OpenAI model through Microsoft Azure service, pass the api_version, azure_base_url and azure_deployment_name parameters in model_kwargs. For more information, see the model_kwargs argument description.
For a list of supported models, see the PromptNode documentation.
Mandatory.
default_prompt_templateStringName of the out-of-the-box template or the name of your custom template.The prompt template you want to use with the PromptNode. The template contains instructions for the model. If you don't specify it, the model tries to guess what task you want it to do based on your query. For best results, we recommend specifying the template.
Optional.
output_variableStringDefault: resultsThe name of the output variable where you want to store the inference results. If not specified, PromptNode uses the output_variable from the PromptTemplate. If PromptTemplate's output_variable is not set, the default name is results.
Optional.
max_lengthIntegerDefault: 100The maximum number of tokens of the text output the PromptNode generates.
The length of the prompt and the length of the generated output must not be larger than the number of tokens the LLM can process.
Optional.
api_keyStringDefault: NoneThe API key for the model provider, like OpenAI, Cohere, etc.
Optional.
use_auth_tokenStringDefault: NoneThe Hugging Face authentication token for your private model.
Optional.
use_gpuBooleanTrue
False
Default: None
Specifies if you want to use GPU when running PromptNode.
Optional.
devicesList of stringsExample: [torch.device('cuda:0'), "mps", "cuda:1"]The list of torch devices to use.
Optional.
stop_wordsList of stringsDefault: NoneIf the PromptNode encounters any of the words you specify here, it stops generating text.
Optional.
top_kIntegerDefault: 1The number of answers (generated texts) you want PromptNode to return.
Mandatory.
debugBooleanTrue
False (default)
Enables debug output.
Optional.
model_kwargsDictionaryDefault: NoneAny additional keyword arguments you want to pass to the model, for example: model_kwargs: temperature: 0.7
To use a remote OpenAI model through Microsoft Azure, pass api_version, azure_base_url and azure_deployment_name in model_kwargs.
You can find the azure_base_url parameter in the Keys and Endpoint tab of your Azure account. You choose the azure_deployment_name when you deploy a model through your Azure account in the Model deployments tab. For available models and versions for the service, check Azure documentation.

Here are the parameters you can specify:
- For OpenAI models, you can specify the following parameters: max_tokens, temperature, top_p, n, stop, presence_penalty, frequency_penalty, logit_bias, response_format, seed. See the OpenAI documentation for these parameters.

- For Azure OpenAI models, you can specify the following parameters: max_tokens, temperature, top_p, n, stop, presence_penalty, frequency_penalty, logit_bias. See Azure OpenAI documentation for these parameters.

- For Cohere models, you can specify the following parameters: end_sequences, frequency_penalty, k, max_tokens, model, num_generations, p, presence_penalty, return_likelihoods, temperature, and truncate. See Cohere documentation for these parameters.
- For AWS Bedrock models:

- Anthropic Claude 2, you can specify all parameters listed in Anthropic documentation except for prompt and stream.
- Anthropic Claude 3, you can specify all parameters listed in Anthropic documentation except for prompt, stream, and tools.
- Cohere Command, you can specify all parameters listed in Cohere documentation except for stream and prompt.
- Meta Llama2, deepset Cloud supports all parameters documented in AWS documentation .
- Amazon Titan, you can specify all parameters listed in the Amazon documentation listed in textGenerationConfig.
Optional.
truncateBooleanTrue
False
Default: True
Truncates the prompt to the maximum token limit before sending it to the model.
Mandatory.
timeoutFloatDefault: NoneSets the timeout for PromptNode.
Optional.

CNPromptNodeAsClassifier Parameters

In pipeline YAML, CNPromptNodeAsClassifier takes the same parameters as PromptNode and an additional labels parameter:

ParameterTypePossible ValuesDescription
labelsList of stringsA list of labelsList of labels used for classification. Each label is routed to a different output edge.
Required.

PromptTemplate Parameters

These are the parameters you can configure for the PromptTemplate:

ParameterTypePossible ValuesDescription
promptStringThe instructions for the model.
It can contain variables in the f-string syntax, for example: {documents}. These variables need to be filled in the prompt_text for the model to perform the task. Note that other than strict f-string syntax, you can safely use the following backslash characters in text parts of the prompt text: \n, \t, \r. If you want to use them in f-string expressions, use new_line, tab, carriage_return instead. Double quotes (") are automatically replaced with single quotes (') in the prompt text. To use double quotes in the prompt text, use {double_quote} instead.
Mandatory.
output_parserStringAnswerParser()Applies a parser that converts the model to an Answer object. If you want to use it, always set it to AnswerParser().
Optional.

PromptTemplate Function Parameters

These are the parameters you can use in prompt template functions:

ParameterTypeDescription
documentsListThe documents whose rendering you want to format. Mandatory.
patternStringThe regex pattern used for parsing.
You can use the following placeholders in pattern:
- $content: The content of the document.
- $idx: The index of the document in the list.
- $id: The ID of the document.
- $META_FIELD: The values of the metadata field called META_FIELD (Only fields that match python's string Template pattern are supported. That is: [\_a-z][_a-z0-9]\*). (Only fields that match python's string Template pattern are supported. That is: [\_a-z][_a-z0-9]\*).
Optional.
delimiterString
Default: " " (single space)
Specifies the delimiter you want to use to separate documents. Used in the join function.
Mandatory.
str_replaceDictionary of stringsSpecifies the characters you want to replace. Use the format `str_replace={"r":"R"}.
Optional.

PromptModel Parameters

ParameterTypePossible ValuesDescription
model_name_or_pathStringDefault: google/flan-t5-baseThe name of the model or the path to a locally stored model. You can use the following models:

- Hugging Face transformers (all text2text-generation and text-generation models).
- OpenAI InstructGPT models.
- Azure OpenAI InstructGPT models.
- Cohere models
You can also use a remote model hosted on AWS SageMaker or an OpenAI model through Microsoft Azure.
To use a model hosted on SageMaker, contact your deepset Cloud representative.
To use a model through Azure, pass api_version, azure_deployment_name, and azure_base_url in model_kwargs. You can find the azure_base_url parameter in the Keys and Endpoint tab of your Azure account. You choose the azure_deployment_name when you deploy a model through your Azure account in the Model deployments tab. For available models and versions for the service, check Azure documentation.
See also the model_kwargs description below.
To use gpt-4o as a visual QA model, set the detail parameter through kwargs. Possible values are auto, high, and low.
Mandatory.
max_lengthIntegerDefault: 100The maximum length of the output text the model generates.
Optional.
api_keyStringDefault: NoneThe API key to use for the model.
Optional.
use_auth_tokenUnionk:parameterDefault: NoneIf the model is hosted on Hugging Face, this is the token to use for the model.
Optional.
use_gpuBooleanDefault: NoneUses GPU if available.
Optional.
devicesStringA list of GPU devices
Example: [torch.device('cuda:0'), "mps", "cuda:1"]
Contains a list of GPU devices to limit inference to certain GPUs and not use all available GPUs. As multi-GPU training is currently not implemented for DPR, training only uses the first device provided in this list.
Optional.
invocation_layer_classPromptModelInvocationLayerDefault: None
Possible values:
The custom invocation layer class to use. If you created your own invocation layer, you can pass it here.
If you don't specify it, it uses one of the known invocation layers (anthropic_claude, azure_chatgpt, azure_open_ai, chatgpt, cohere, hugging_face, hugging_face_inference, open_ai, sagemaker).
Optional.
model_kwargsDictionaryDefault: NoneAdditional parameters passed to the model, such as temperature. Model parameters are specific to the model.
To use a remote OpenAI model through Microsoft Azure service,
pass these two parameters: azure_base_url (the URL for the Azure OpenAI API endpoint, usually in the form https://<your-endpoint>.openai.azure.com, you can find it in the Keys and Endpoint tab of your Azure account) and azure_deployment_name (the name of the Azure OpenAI API deployment you specify when deploying a model through Azure in the Model deployments tab). You should pass these parameters in the model_kwargs dictionary.

- For OpenAI models, you can specify the following parameters: max_tokens, temperature, top_p, n, stop, presence_penalty, frequency_penalty, logit_bias, response_format, seed. See the OpenAI documentation for these parameters.

- For Azure OpenAI models, you can specify the following parameters: max_tokens, temperature, top_p, n, stop, presence_penalty, frequency_penalty, logit_bias. See Azure OpenAI documentation for these parameters.

- For Cohere models, you can specify the following parameters at runtime: end_sequences, frequency_penalty, k, max_tokens, model, num_generations, p, presence_penalty, return_likelihoods, temperature, and truncate. See Cohere documentation for these parameters.
- For AWS Bedrock models:

- Anthropic Claude 2, you can specify all parameters listed in Anthropic documentation except for prompt and stream.

- Anthropic Claude 3, you can specify all parameters listed in Anthropic documentation except for prompt, stream, and tools.
- Cohere Command, you can specify all parameters listed in Cohere documentation except for stream and prompt.
- Meta Llama2, deepset Cloud supports all parameters documented in AWS documentation .
- Amazon Titan, you can specify all parameters listed in the Amazon documentation listed in textGenerationConfig.

REST API Runtime Parameters

These are the runtime parameters you can pass in the body of the request to the Search endpoint:

PromptNode and CNPromptNodeAsClassifier Parameters

REST API parameters are the same for both PromptNode types.

ParameterTypePossible ValuesDescription
generation_kwargsDictionary [String, Any]Default: noneAdditional parameters to configure how PromptNode generates the text. These parameters depend on the model PromptNode uses.
- For OpenAI models, you can specify the following parameters: max_tokens, temperature, top_p, n, stop, presence_penalty, frequency_penalty, logit_bias, response_format, seed. See the OpenAI documentation for these parameters.

- For Azure OpenAI models, you can specify the following parameters: max_tokens, temperature, top_p, n, stop, presence_penalty, frequency_penalty, logit_bias. See Azure OpenAI documentation for these parameters.

- For Cohere models, you can specify the following parameters: end_sequences, frequency_penalty, k, max_tokens, model, num_generations, p, presence_penalty, return_likelihoods, temperature, and truncate. See Cohere documentation for these parameters.
- For AWS Bedrock models:
- Anthropic Claude 2, you can specify all parameters listed in Anthropic documentation except for prompt and stream.
- Anthropic Claude 3, you can specify all parameters listed in Anthropic documentation except for prompt, stream, and tools.
- Cohere Command, you can specify all parameters listed in Cohere documentation except for stream and prompt.
- Meta Llama2, deepset Cloud supports all parameters documented in AWS documentation.
- Amazon Titan, you can specify all parameters listed in the Amazon documentation listed in textGenerationConfig.
Optional.
prompt_templateStringNone
PromptTemplate
string
Default: none
Specifies the prompt template for PromptNode. You can set it to the following values:
- None: Uses the default prompt template.
- PromptTemplate: Uses the prompt template you pass to it.
- String: You can pass the following values as a string:
the template name (then the prompt template with the given name si used), prompt template YAML (uses the prompt template from the YAML), prompt text (uses a copy of the default prompt template with this prompt text).
Optional.