With PromptNode, you can use large language models directly in your pipelines. It's a very versatile node used in query pipelines, but its position depends on what you want it to do. You pass a prompt to it to specify the NLP task the PromptNode should perform, and a model to use.

📘
Changes in Release 2023.07
The latest version of deepset Cloud simplifies the PromptTemplate parameters. Only two parameters are now supported: prompt and answer_parser. All your deployed pipelines will keep running as before. But if you undeploy a pipeline, you'll need to update your PromptTemplate structure.

What are large language models?

Large language models are huge models trained on enormous amounts of data. Interacting with such a model resembles talking to another person. These models have general knowledge of the world. You can ask them anything, and they'll be able to answer.

Large language models are trained to perform many NLP tasks with little training data. What's astonishing about them is that a single model can perform various NLP tasks with good accuracy.

Some examples of large language models include flan-t5-base, flan-paLM, Claude, Falcon, and GPT-3 variants, such as gpt-3.5-turbo.

For more information, see also Large Language Models.

Basic Information

Pipeline type: Used in query pipelines.
Position in a pipeline: The position depends on the NLP task you want it to do.
Node Input: Depends on the NLP task it performs. Some examples are query, documents, and the output of the preceding node. You define the input in the PromptTemplate you pass to PromptNode.
Node Output: Depends on the NLP task it performs. You can specify the output you want in the prompt you're passing to PromptNode.
Available classes: PromptNode
Supported models:
- flan t5 base (default)
- Hugging Face transformers (all text2text-generation models)
- OpenAI InstructGPT models, including ChatGPT and GPT-4
- Azure OpenAI InstructGPT models
- Cohere's command models
- Anthropic's Claude models
- Open source models hosted on Amazon SageMaker

Usage Examples

Stand Alone

You can run PromptNode in your SDK. Just initialize the node and ask a question. The model has general knowledge about the world, so you can ask it anything.

📘
You can use PromptNode as a stand-alone node in deepset Cloud only through Notebooks.

from haystack.nodes import PromptNode

# Initialize the node:
prompt_node = PromptNode()

# Run a prompt
prompt_node("What is the capital of Germany?")

# Here's the output:
['berlin']

With a Custom Prompt

To use a custom prompt, declare PromptTemplate as your pipeline component and paste the prompt text in the prompt parameter. Then, pass the name of the PromptTemplate in the default_prompt_template parameter of PromptNode:

...
components:
  - name: question-answering # give this pipeline any name
    type: PromptTemplate
    params: # the prompt parameter contains the text you copied from the template in Prompt Explorer
    	prompt: "You are a technical expert. \
        You answer questions truthfully based on provided documents. \
        For each document check whether it is related to the question. \
        Only use documents that are related to the question to answer it. \
        Ignore documents that are not related to the question. \
        If the answer exists in several documents, summarize them. \
        Only answer based on the documents provided. Don't make things up. \
        Always use references in the form [NUMBER OF DOCUMENT] when using information from a document. e.g. [3], for Document[3]. \
        The reference must only refer to the number that comes in square brackets after passage. \
        Otherwise, do not use brackets in your answer and reference ONLY the number of the passage without mentioning the word passage. \
        If the documents can't answer the question or you are unsure say: 'The answer can't be found in the text'. \
        {new_line}\
        These are the documents:\
        {join(documents, delimiter=new_line, pattern=new_line+'Document[$idx]:'+new_line+'$content')}\
        {new_line}\
        Question: {query}\
        {new_line}\
        Answer:\
        {new_line}"
	- name: PromptNode 
    type: PromptNode
    params:
      default_prompt_template: question-answering # This tells PromptNode to use the PromptTemplate you configured above
   ...
   pipelines:
  - name: query
    nodes:
     - name: Retriever
     	 inputs: [Query]
     - name: PromptNode
       inputs: [Retriever]
    ...

In your pipeline, you can have multiple PromptNodes, each with a different prompt. This way, each of them performs a different task. See the In a Pipeline section.

With a Model Specified

To use a proprietary model, make sure you first Connect to Model Providers. That's safer than adding the API key in the YAML.

You specify the model in the model_name_or_path parameter. Additional model parameters go into model_kwargs, for example:

components:
	- name: PromptNode 
    type: PromptNode
    params:
      model_name_or_path: google/flan-t5-xl
      model_kwargs:
      	temperature: 0.6

To use Azure OpenAI service, you must pass the API version, base URL, and deployment name in model_kwargs like this:

components:
	- name: PromptNode 
    type: PromptNode
    params:
      model_name_or_path: text-davinci-003
      api_key: <azure_openai_api_key>
      model_kwargs:
      	api_version: 2022-12-01
        azure_base_url: https://<your-endpoint>.openai.azure.com
        azure_deployment_name: <your-deployment-name>

You can find the azure_base_url parameter in the Keys and Endpoint tab of your Azure account. You choose the azure_deployment_name when you deploy a model through your Azure account in the Model deployments tab. For available models and versions for the service, check Azure documentation.

For instructions on using a model hosted on SageMaker, see Using a Model Hosted on AWS SageMaker.

For more information about model parameters, see the PromptModel Arguments section.

In a Pipeline

The real power of PromptNode shows when you use it in a pipeline. Look at the example to get an idea of what's possible.

Conversational Summary

Here's an example of how you could use PromptNode to generate a summary of a chat transcript:


name: 'Chat_summary'
version: '1.21.0'

components: 
	- name: DocumentStore
    type: DeepsetCloudDocumentStore # The only supported document store in deepset Cloud
  - name: Retriever # Selects the most relevant documents from the document store so that the OpenAI model can base it's generation on it. 
    type: EmbeddingRetriever # Uses a Transformer model to encode the document and the query
    params:
      document_store: DocumentStore
      embedding_model: sentence-transformers/multi-qa-mpnet-base-dot-v1 # Model optimized for semantic search 
      model_format: sentence_transformers
      top_k: 1 # The number of documents
 	- name: PromptNode
   	type: PromptNode
   	params:
    	default_prompt_template: deepset/conversational-summary # this is a ready-made prompt template for summarizing transcripts. For more templates, see Prompt Explorer.
    	model_name_or_path: text-davinci-003
    	api_key: <api_key>
	 - name: FileTypeClassifier # Routes files based on their extension to appropriate converters, by default txt, pdf, md, docx, html
   	 type: FileTypeClassifier
	 - name: TextConverter # Converts files into documents
   	 type: TextConverter
	 - name: PDFConverter # Converts PDFs into documents
	   type: PDFToTextConverter
	 - name: Preprocessor # Splits documents into smaller ones and cleans them up
	   type: PreProcessor
	   params:
	     split_by: word # The unit by which you want to split the documents
 	     split_length: 250 # The max number of words in a document
 	     split_overlap: 20 # Enables the sliding window approach
 	     language: en
 	     split_respect_sentence_boundary: True 
     
 pipelines:
  - name: query
    nodes:
     - name: Retriever
     	 inputs: [Query]
     - name: PromptNode
       inputs: [Retriever]
   - name: indexing
     nodes: 
      - name: FileTypeClassifier
        inputs: [File]
      - name: TextConverter
        inputs: [FileTypeClassifier.output_1] # Ensures that this converter receives txt files
      - name: PDFConverter
        inputs: [FileTypeClassifier.output_2] # Ensures that this converter receives PDFs
      - name: Preprocessor
        inputs: [TextConverter, PDFConverter]
      - name: Retriever
        inputs: [Preprocessor]
      - name: DocumentStore
        inputs: [Retriever]

Mulitple PromptNodes

You can reuse the instance of a model for multiple PromptNodes in your pipeline. This saves resources. To do this, configure the PromptModel as a component and then pass its name in the model_name_or_path parameter in each instance of PromptNode.

components: 
	- name: DocumentStore
    type: DeepsetCloudDocumentStore # The only supported document store in deepset Cloud
  - name: Retriever # Selects the most relevant documents from the document store so that the OpenAI model can base it's generation on it. 
    type: EmbeddingRetriever # Uses a Transformer model to encode the document and the query
    params:
      document_store: DocumentStore
      embedding_model: sentence-transformers/multi-qa-mpnet-base-dot-v1 # Model optimized for semantic search 
      model_format: sentence_transformers
      top_k: 1 # The number of documents
  - name: MyPromptModel
  	type: PromptModel
    params:
    	model_name_or_path: text-davinci-003
      use_gpu: True
 	- name: PromptNode1
   	type: PromptNode
   	params:
    	default_prompt_template: deepset/question-generation
    	model_name_or_path: MyPromptModel
    	api_key: <api_key>
 	- name: PromptNode2
   	type: PromptNode
   	params:
    	default_prompt_template: deepset/question-answering-per-document
    	model_name_or_path: MyPromptModel
    	api_key: <api_key>
  # You would also need to configure the components for the indexing pipeline
  # We're skipping this part in this example
  
  pipelines:
  - name: query
    nodes:
      - name: Retriever
        inputs: [Query]
      - name: PromptNode1
        inputs: [Retriever]
      - name: PromptNode2
        inputs: [PromptNode1]
# Here would come the indexing pipeline, which we're skipping in this example

from haystack import Pipeline
from haystack.nodes.prompt import PromptNode, PromptModel
# You'd also need to import a Retriever and a DocumentStore, 
# we're skipping this in this example

top_k = 10
query = "Who is Paul Atreides' father?"

prompt_model = PromptModel()
node = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="query")
node2 = PromptNode(prompt_model, default_prompt_template="question-answering-per-document")

# You'd also need to initialize a Retriever with a DocumentStore
# We're skipping this step in this example to simplify it
pipe = Pipeline()
pipe.add_node(component=retriever, name="retriever", inputs=["Query"])
pipe.add_node(component=node, name="prompt_node", inputs=["retriever"])
pipe.add_node(component=node2, name="prompt_node_2", inputs=["prompt_node"])
output = pipe.run(query=query, params={"retriever": {"top_k": 10}})

dict(zip(output["query"], output["answers"]))

Using Different Models in One Pipeline

You can also specify different LLMs for each PromptNode in your pipeline. This way, you create multiple PromptNode instances that use a single PromptNode, which saves computational resources.

# In YAML, you simply specify two PromptNodes, each with a different name and a different model
# Bear in mind that this example is not a complete pipeline, you'd still need to create the indexing pipeline
# and define its components


components:
 - name: PromptNodeOpenAI 
   type: PromptNode
   params:
   	default_prompt_template: deepset/question-answering
    model_name_or_path: text-davinci-003
    api_key: <my_openai_key>
 - name: PromptNodeDefault
   type: PromptNode
   params:
    default_prompt_template: deepset/question-generation
    model_name_or_path: google/flan-t5-large

# And now you could put the two nodes together in the query pipeline:
pipelines:
 - name: query
   nodes:
    - name: PromptNodeDefault
      inputs: [Query]
    - name: PromptNodeOpenAI
      inputs: [PromptNodeDefault]

from haystack.nodes. import PromptTemplate, PromptNode, PromptModel
from haystack.pipelines import Pipeline
 
api_key = "<type your OpenAI API key>"

# Specify the model you want to use:
prompt_open_ai = PromptModel(model_name_or_path="text-davinci-003", api_key=api_key)

# This sets up the default model:
prompt_model = PromptModel()

# Now let make one PromptNode use the default model and the other one the OpenAI model:
node_default_model = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="questions")
node_openai = PromptNode(prompt_open_ai, default_prompt_template="question-answering")

pipeline = Pipeline()
pipeline.add_node(component=node_default_model, name="prompt_node1", inputs=["Query"])
pipe.add_node(component=node_openai, name="prompt_node_2", inputs=["prompt_node1"])
output = pipe.run(query="not relevant", documents=[Document("Berlin is the capital of Germany")])
output["results"]

Prompt Templates

PromptTemplates work a bit like pipeline components. To use a custom prompt, you declare it like a pipeline component and then you pass its name to PromptNode. See the Usage Example: With a Custom Prompt section.

deepset Cloud comes with a library of out-of-the-box prompt templates ready for you to use for multiple NLP tasks. Each template contains the prompt, which is the instruction for the model. You can browse the available templates in Prompt Explorer by clicking Templates. You can also save your custom prompts there.

PromptTemplate Structure

You can create your own prompt template and save it in the library of Prompt Explorer for future use. If you create a template in Prompt Explorer, it guides you through the structure it should have. But you can also create a prompt directly in your pipeline. If you do this, follow this structure:

components:
 - name: my_prompt 
   type: PromptTemplate
   params:
   	prompt: "Here comes the prompt text"

prompt contains the prompt for the task you want the model to do. It also specifies input variables in curly brackets. The variables can be document, query, label.
At runtime, these variables must be present in the execution context of the node. You can apply functions to those variables. For example, you can combine the list of documents into a string by applying the join function to execute only one prompt instead of one prompt per one document.
You can use \n, \t, and \r characters in the prompt text.
output_parser converts the output to an Answerobject. If you need PromptNode to output an Answer object, set this parameter to AnswerParser().

For a detailed explanation, see the PromptTemplate Arguments section.

Functions in Prompts

You can add the join and to_strings functions to your template to control how the documents, query, or any other variables are rendered in the output. Functions are useful if you want to, for example, make the model reference a document ID or metadata in the output.

Functions use the Python f-string format and you can use any list comprehension inside a function. You can't use \n, \t, or \r inside a function. Also, double quotes (") are automatically replaced with single quotes (') in the function. Here are the variables you can use in the functions instead:

Special characters not allowed in functions	PromptTemplate variable to use instead
`\n`	`new_line`
`\t`	`tab`
`\r`	`carriage_return`
"	`double_quote`

For detailed parameters, you can use in a function, see the Function Arguments section.

The `join()` Function

The join function joins all documents into a single string with the content of each document separated by the delimiter you specify. Here's a simple example (line 6):

- name: summary-custom
    type: PromptTemplate
    params:
      prompt: >
        Summarize this document: join{documents} \n\n 
        If the document is not there, answer with: "I can''t answer the question based on the information provided." \n\n
        Summary:

The function here joins all documents into a single string. This way, PromptNode only sends one prompt, which is faster. If you don't join the documents, PromptNode sends one prompt for each document.

This example joins all documents in a single string, but it also formats the resulting string so that the output is the meta name of each document followed by its contents:

Context: {' '.join([d.meta['name']+': '+d.content for d in documents])}

' '.join([d.meta['name']+': '+d.content for d in documents])

Here's what the resulting string would look like:

DocumentMetaName1: DocumentContent1 DocumentMetaName2: DocumentContent2 # and so on, for each document

Now, you could tell the model in the prompt to print answers together with the meta name of the document containing the answer.

The `to_strings()` Function

This function extracts the content field of documents and returns a list of strings. In the example below, it renders each document by its name (document.meta["name"]) followed by a new line and the contents of the document:

{to_strings(documents, pattern='$name'+new_line+'$content', str_replace={new_line: ' ', '[': '(', ']': ')'})}

{to_strings(documents, pattern='$name'+new_line+'$content', str_replace={new_line: ' ', '[': '(', ']': ')'})}

Arguments

PromptNode Arguments

Use these parameters to configure the PromptNode:

Argument	Type	Possible Values	Description
`model_name_or_path`	String	Default: `google/flan-t5-base`	The name of the model you want to use with the PromptNode or the path to a locally stored model. Mandatory.
`default_prompt_template`	String	Name of the out-of-the-box template or the name of your custom template.	The prompt template you want to use with the PromptNode. The template contains instructions for the model. If you don't specify it, the model tries to guess what task you want it to do based on your query. For best results, we recommend specifying the template. Optional.
`output_variable`	String	Default: `None`	The name of the output variable where you want to store the inference results. If not specified, PromptNode uses the `output_variable` from the PromptTemplate. If PromptTemplate's `output_variable` is not set, the default name is `results`. Optional.
`max_length`	Integer	Default: `100`	The maximum number of tokens of the text output the PromptNode generates. The length of the prompt and the length of the generated output must not be larger than the number of tokens the LLM can process. Optional.
`api_key`	String	Default: `None`	The API key for the model. Specify it to use a model by OpenAI. Optional.
`use_auth_token`	String	Default: `None`	The Hugging Face authentication token for your private model. Optional.
`use_gpu`	Boolean	`True` `False` Default: `None`	Specifies if you want to use GPU when running PromptNode. Optional.
`devices`	List of strings	Example: `[torch.device('cuda:0'), "mps", "cuda:1"]`	The list of torch devices to use. Optional.
`stop_words`	List of strings	Default: `None`	If the PromptNode encounters any of the words you specify here, it stops generating text. Optional.
`top_k`	Integer	Default: `1`	The number of answers (generated texts) you want PromptNode to return. Mandatory.
`debug`	Boolean	`True` `False` (default)	Enables debug output. Optional.
`model_kwargs`	Dictionary	Default: `None`	Any additional keyword arguments you want to pass to the model, for example: `model_kwargs: temperature: 0.7` Optional.

PromptTemplate Arguments

These are the arguments you can use when creating your own PromptTemplate:

Argument	Type	Possible Values	Description
`prompt`	String		The instructions for the model. It can contain variables in the f-string syntax, for example: `{documents}`. These variables need to be filled in the prompt_text for the model to perform the task. Note that other than strict f-string syntax, you can safely use the following backslash characters in text parts of the prompt text: `\n,` `\t,` `\r`. If you want to use them in f-string expressions, use `new_line`, `tab`, `carriage_return` instead. Double quotes (") are automatically replaced with single quotes (') in the prompt text. To use double quotes in the prompt text, use `{double_quote}` instead. Mandatory.
`output_parser`	String	`AnswerParser()`	Applies a parser that converts the model to an Answer object. If you want to use it, always set it to `AnswerParser()`. Optional.

Function Arguments

Parameter	Type	Description
`documents`	List	The documents whose rendering you want to format. Mandatory.
`pattern`	String	The regex pattern used for parsing. You can use the following placeholders in pattern: - `$content`: The content of the document. - `$idx`: The index of the document in the list. - `$id`: The ID of the document. - `$META_FIELD`: The values of the metadata field called `META_FIELD` (Only fields that match python's string Template pattern are supported. That is: `[\_a-z][_a-z0-9]\`). (Only fields that match python's string Template pattern are supported. That is: `[\_a-z][_a-z0-9]\`). Optional.
`delimiter`	String Default: " " (single space)	Specifies the delimiter you want to use to separate documents. Used in the `join` function. Mandatory.
`str_replace`	Dictionary of strings	Specifies the characters you want to replace. Use the format `str_replace={"r":"R"}. Optional.

PromptModel Arguments

Argument	Type	Possible Values	Description
`model_name_or_path`	String	Default: `google/flan-t5-base`	The name of the model or the path to a locally stored model. You can use the following models: - Hugging Face transformers (all text2text-generation and text-generation models). - OpenAI InstructGPT models. - Azure OpenAI InstructGPT models. Mandatory.
`max_length`	Integer	Default: `100`	The maximum length of the output text the model generates. Optional.
`api_key`	String	Default: `None`	The API key to use for the model. Optional.
`use_auth_token`	Unionk:parameter	Default: `None`	If the model is hosted on Hugging Face, this is the token to use for the model. Optional.
`use_gpu`	Boolean	Default: `None`	Uses GPU if available. Optional.
`devices`	String	A list of GPU devices Example: `[torch.device('cuda:0'), "mps", "cuda:1"]`	Contains a list of GPU devices to limit inference to certain GPUs and not use all available GPUs. As multi-GPU training is currently not implemented for DPR, training only uses the first device provided in this list. Optional.
`invocation_layer_class`	PromptModelInvocationLayer	Default: `None`	The custom invocation layer class to use. If you don't specify it, it uses one of the known invocation layers. Optional.
`model_kwargs`	Dictionary	Default: `None`	Additional keyword arguments passed to the underlying model. Note that Azure OpenAI InstructGPT models require two additional parameters: `azure_base_url` (the URL for the Azure OpenAI API endpoint, usually in the form `https://<your-endpoint>.openai.azure.com`, you can find it in the Keys and Endpoint tab of your Azure account) and `azure_deployment_name` (the name of the Azure OpenAI API deployment you specify when deploying a model through Azure in the Model deployments tab). You should pass these parameters in the `model_kwargs` dictionary.

Basic Information

Usage Examples

Stand Alone

With a Custom Prompt

With a Model Specified

In a Pipeline

Conversational Summary

Mulitple PromptNodes

Using Different Models in One Pipeline

Prompt Templates

PromptTemplate Structure

Functions in Prompts

The join() Function

The to_strings() Function

Arguments

PromptNode Arguments

PromptTemplate Arguments

Function Arguments

PromptModel Arguments

The `join()` Function

The `to_strings()` Function