YAML Init Parameters

These are the parameters you can specify in pipeline YAML:

PromptNode Parameters

Parameter	Type	Possible Values	Description
`model_name_or_path`	String	Default: `google/flan-t5-base`	The name of the model you want to use with the PromptNode or the path to a locally stored model. To use an OpenAI model through Microsoft Azure service, pass the `api_version`, `azure_base_url` and `azure_deployment_name` parameters in `model_kwargs`. For more information, see the `model_kwargs` argument description. For a list of supported models, see the PromptNode documentation. Mandatory.
`default_prompt_template`	String	Name of the out-of-the-box template or the name of your custom template.	The prompt template you want to use with the PromptNode. The template contains instructions for the model. If you don't specify it, the model tries to guess what task you want it to do based on your query. For best results, we recommend specifying the template. Optional.
`output_variable`	String	Default: `results`	The name of the output variable where you want to store the inference results. If not specified, PromptNode uses the `output_variable` from the PromptTemplate. If PromptTemplate's `output_variable` is not set, the default name is `results`. Optional.
`max_length`	Integer	Default: `100`	The maximum number of tokens of the text output the PromptNode generates. The length of the prompt and the length of the generated output must not be larger than the number of tokens the LLM can process. Optional.
`api_key`	String	Default: `None`	The API key for the model provider, like OpenAI, Cohere, etc. Optional.
`use_auth_token`	String	Default: `None`	The Hugging Face authentication token for your private model. Optional.
`use_gpu`	Boolean	`True` `False` Default: `None`	Specifies if you want to use GPU when running PromptNode. Optional.
`devices`	List of strings	Example: `[torch.device('cuda:0'), "mps", "cuda:1"]`	The list of torch devices to use. Optional.
`stop_words`	List of strings	Default: `None`	If the PromptNode encounters any of the words you specify here, it stops generating text. Optional.
`top_k`	Integer	Default: `1`	The number of answers (generated texts) you want PromptNode to return. Mandatory.
`debug`	Boolean	`True` `False` (default)	Enables debug output. Optional.
`model_kwargs`	Dictionary	Default: `None`	Any additional keyword arguments you want to pass to the model, for example: `model_kwargs: temperature: 0.7` To use a remote OpenAI model through Microsoft Azure, pass `api_version`, `azure_base_url` and `azure_deployment_name` in `model_kwargs`. You can find the `azure_base_url` parameter in the Keys and Endpoint tab of your Azure account. You choose the `azure_deployment_name` when you deploy a model through your Azure account in the Model deployments tab. For available models and versions for the service, check Azure documentation. Here are the parameters you can specify: - For OpenAI models, you can specify the following parameters: `max_tokens`, `temperature`, `top_p`, `n`, `stop`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `response_format`, `seed`. See the OpenAI documentation for these parameters. - For Azure OpenAI models, you can specify the following parameters: `max_tokens`, `temperature`, `top_p`, `n`, `stop`, `presence_penalty`, `frequency_penalty`, `logit_bias`. See Azure OpenAI documentation for these parameters. - For Cohere models, you can specify the following parameters: `end_sequences`, `frequency_penalty`, `k`, `max_tokens`, `model`, `num_generations`, `p`, `presence_penalty`, `return_likelihoods`, `temperature`, and `truncate`. See Cohere documentation for these parameters. - For AWS Bedrock models: - Anthropic Claude 2, you can specify all parameters listed in Anthropic documentation except for `prompt` and `stream`. - Anthropic Claude 3, you can specify all parameters listed in Anthropic documentation except for `prompt`, `stream`, and `tools`. - Cohere Command, you can specify all parameters listed in Cohere documentation except for `stream` and `prompt`. - Meta Llama2, deepset Cloud supports all parameters documented in AWS documentation . - Amazon Titan, you can specify all parameters listed in the Amazon documentation listed in `textGenerationConfig`. Optional.
`truncate`	Boolean	`True` `False` Default: `True`	Truncates the prompt to the maximum token limit before sending it to the model. Mandatory.
`timeout`	Float	Default: `None`	Sets the timeout for PromptNode. Optional.

CNPromptNodeAsClassifier Parameters

In pipeline YAML, CNPromptNodeAsClassifier takes the same parameters as PromptNode and an additional labels parameter:

Parameter	Type	Possible Values	Description
`labels`	List of strings	A list of labels	List of labels used for classification. Each label is routed to a different output edge. Required.

PromptTemplate Parameters

These are the parameters you can configure for the PromptTemplate:

Parameter	Type	Possible Values	Description
`prompt`	String		The instructions for the model. It can contain variables in the f-string syntax, for example: `{documents}`. These variables need to be filled in the prompt_text for the model to perform the task. Note that other than strict f-string syntax, you can safely use the following backslash characters in text parts of the prompt text: `\n,` `\t,` `\r`. If you want to use them in f-string expressions, use `new_line`, `tab`, `carriage_return` instead. Double quotes (") are automatically replaced with single quotes (') in the prompt text. To use double quotes in the prompt text, use `{double_quote}` instead. Mandatory.
`output_parser`	String	`AnswerParser()`	Applies a parser that converts the model to an Answer object. If you want to use it, always set it to `AnswerParser()`. AnswerParser parses the model output to extract the answer into a proper `Answer` object using regex patterns. It adds the `document_ids` of the documents used to generate the answer and the prompts used to the `Answer` object. It takes `reference_pattern` as an argument. It extracts the document_ids of the answer from the model output. It's a regex pattern you want to use to parse the answer. Examples: `[^\\n]+$` finds "this is an answer" in string "this is an argument.\this is an answer". `Answer: (.)` finds "this is an answer" in the string "this is an argument. Answer: this is an answer". You can also use it to add references to the answers in your RAG pipelines. For example, a reference pattern `\[(?:(\\d+),?\\s\)+\]` matches any references in the formats: [1], [2][3], [4,5], [6, 7]. We've added an abbreviation `acm` for this pattern. So to show references like `[1], [2], [3]`, set `reference_pattern: acm`. If not specified, the whole string is used as the answer. If specified, the first group of the regex is used as the answer. If there is no group, the whole match is used as the answer. Optional.

PromptTemplate Function Parameters

These are the parameters you can use in prompt template functions:

Parameter	Type	Description
`documents`	List	The documents whose rendering you want to format. Mandatory.
`pattern`	String	The regex pattern used for parsing. You can use the following placeholders in pattern: - `$content`: The content of the document. - `$idx`: The index of the document in the list. - `$id`: The ID of the document. - `$META_FIELD`: The values of the metadata field called `META_FIELD` (Only fields that match python's string Template pattern are supported. That is: `[\_a-z][_a-z0-9]\`). (Only fields that match python's string Template pattern are supported. That is: `[\_a-z][_a-z0-9]\`). Optional.
`delimiter`	String Default: " " (single space)	Specifies the delimiter you want to use to separate documents. Used in the `join` function. Mandatory.
`str_replace`	Dictionary of strings	Specifies the characters you want to replace. Use the format `str_replace={"r":"R"}. Optional.

PromptModel Parameters

Parameter	Type	Possible Values	Description
`model_name_or_path`	String	Default: `google/flan-t5-base`	The name of the model or the path to a locally stored model. You can use the following models: - Hugging Face transformers (all text2text-generation and text-generation models). - OpenAI InstructGPT models. - Azure OpenAI InstructGPT models. - Cohere models You can also use a remote model hosted on AWS SageMaker or an OpenAI model through Microsoft Azure. To use a model hosted on SageMaker, contact your deepset Cloud representative. To use a model through Azure, pass `api_version`, `azure_deployment_name`, and `azure_base_url` in `model_kwargs`. You can find the `azure_base_url` parameter in the Keys and Endpoint tab of your Azure account. You choose the `azure_deployment_name` when you deploy a model through your Azure account in the Model deployments tab. For available models and versions for the service, check Azure documentation. See also the `model_kwargs` description below. To use gpt-4o as a visual QA model, set the `detail` parameter through kwargs. Possible values are `auto`, `high`, and `low`. Mandatory.
`max_length`	Integer	Default: `100`	The maximum length of the output text the model generates. Optional.
`api_key`	String	Default: `None`	The API key to use for the model. Optional.
`use_auth_token`	Unionk:parameter	Default: `None`	If the model is hosted on Hugging Face, this is the token to use for the model. Optional.
`use_gpu`	Boolean	Default: `None`	Uses GPU if available. Optional.
`devices`	String	A list of GPU devices Example: `[torch.device('cuda:0'), "mps", "cuda:1"]`	Contains a list of GPU devices to limit inference to certain GPUs and not use all available GPUs. As multi-GPU training is currently not implemented for DPR, training only uses the first device provided in this list. Optional.
`invocation_layer_class`	PromptModelInvocationLayer	Default: `None` Possible values:	The custom invocation layer class to use. If you created your own invocation layer, you can pass it here. If you don't specify it, it uses one of the known invocation layers (`anthropic_claude`, `azure_chatgpt`, `azure_open_ai`, `chatgpt`, `cohere`, `hugging_face`, `hugging_face_inference`, `open_ai`, `sagemaker`). Optional.
`model_kwargs`	Dictionary	Default: `None`	Additional parameters passed to the model, such as `temperature`. Model parameters are specific to the model. To use a remote OpenAI model through Microsoft Azure service, pass these two parameters: `azure_base_url` (the URL for the Azure OpenAI API endpoint, usually in the form `https://<your-endpoint>.openai.azure.com`, you can find it in the Keys and Endpoint tab of your Azure account) and `azure_deployment_name` (the name of the Azure OpenAI API deployment you specify when deploying a model through Azure in the Model deployments tab). You should pass these parameters in the `model_kwargs` dictionary. - For OpenAI models, you can specify the following parameters: `max_tokens`, `temperature`, `top_p`, `n`, `stop`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `response_format`, `seed`. See the OpenAI documentation for these parameters. - For Azure OpenAI models, you can specify the following parameters: `max_tokens`, `temperature`, `top_p`, `n`, `stop`, `presence_penalty`, `frequency_penalty`, `logit_bias`. See Azure OpenAI documentation for these parameters. - For Cohere models, you can specify the following parameters at runtime: `end_sequences`, `frequency_penalty`, `k`, `max_tokens`, `model`, `num_generations`, `p`, `presence_penalty`, `return_likelihoods`, `temperature`, and `truncate`. See Cohere documentation for these parameters. - For AWS Bedrock models: - Anthropic Claude 2, you can specify all parameters listed in Anthropic documentation except for `prompt` and `stream`. - Anthropic Claude 3, you can specify all parameters listed in Anthropic documentation except for `prompt`, `stream`, and `tools`. - Cohere Command, you can specify all parameters listed in Cohere documentation except for `stream` and `prompt`. - Meta Llama2, deepset Cloud supports all parameters documented in AWS documentation . - Amazon Titan, you can specify all parameters listed in the Amazon documentation listed in `textGenerationConfig`.

REST API Runtime Parameters

These are the runtime parameters you can pass in the body of the request to the Search endpoint:

PromptNode and CNPromptNodeAsClassifier Parameters

REST API parameters are the same for both PromptNode types.

Parameter	Type	Possible Values	Description
`generation_kwargs`	Dictionary [String, Any]	Default: `none`	Additional parameters to configure how PromptNode generates the text. These parameters depend on the model PromptNode uses. - For OpenAI models, you can specify the following parameters: `max_tokens`, `temperature`, `top_p`, `n`, `stop`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `response_format`, `seed`. See the OpenAI documentation for these parameters. - For Azure OpenAI models, you can specify the following parameters: `max_tokens`, `temperature`, `top_p`, `n`, `stop`, `presence_penalty`, `frequency_penalty`, `logit_bias`. See Azure OpenAI documentation for these parameters. - For Cohere models, you can specify the following parameters: `end_sequences`, `frequency_penalty`, `k`, `max_tokens`, `model`, `num_generations`, `p`, `presence_penalty`, `return_likelihoods`, `temperature`, and `truncate`. See Cohere documentation for these parameters. - For AWS Bedrock models: - Anthropic Claude 2, you can specify all parameters listed in Anthropic documentation except for `prompt` and `stream`. - Anthropic Claude 3, you can specify all parameters listed in Anthropic documentation except for `prompt`, `stream`, and `tools`. - Cohere Command, you can specify all parameters listed in Cohere documentation except for `stream` and `prompt`. - Meta Llama2, deepset Cloud supports all parameters documented in AWS documentation. - Amazon Titan, you can specify all parameters listed in the Amazon documentation listed in `textGenerationConfig`. Optional.
`prompt_template`	String	`None` `PromptTemplate` string Default: `none`	Specifies the prompt template for PromptNode. You can set it to the following values: - `None`: Uses the default prompt template. - `PromptTemplate`: Uses the prompt template you pass to it. - String: You can pass the following values as a string: the template name (then the prompt template with the given name si used), prompt template YAML (uses the prompt template from the YAML), prompt text (uses a copy of the default prompt template with this prompt text). Optional.