PromptNode
PromptNode is an easy-to-use, customizable node that brings you the power of large language models. You can use it in your pipelines for various NLP tasks.
With PromptNode, you can use large language models directly in your pipelines. It's a very versatile node used in query pipelines, but its position depends on what you want it to do. You pass a prompt to it to specify the NLP task the PromptNode should perform, and a model to use.
Changes in Release 2023.07
The latest version of deepset Cloud simplifies the PromptTemplate parameters. Only two parameters are now supported:
prompt
andanswer_parser
. All your deployed pipelines will keep running as before. But if you undeploy a pipeline, you'll need to update your PromptTemplate structure.
What are large language models?
Large language models are huge models trained on enormous amounts of data. Interacting with such a model resembles talking to another person. These models have general knowledge of the world. You can ask them anything, and they'll be able to answer.
Large language models are trained to perform many NLP tasks with little training data. What's astonishing about them is that a single model can perform various NLP tasks with good accuracy.
Some examples of large language models include flan-t5-base, flan-paLM, Claude, Falcon, and GPT-3 variants, such as gpt-3.5-turbo.
For more information, see also Large Language Models.
Basic Information
- Pipeline type: Used in query pipelines.
- Position in a pipeline: The position depends on the NLP task you want it to do.
- Node Input: Depends on the NLP task it performs. Some examples are
query
,documents
, and the output of the preceding node. You define the input in the PromptTemplate you pass to PromptNode. - Node Output: Depends on the NLP task it performs. You can specify the output you want in the prompt you're passing to PromptNode.
- Available classes: PromptNode
- Supported models:
- flan t5 base (default)
- Hugging Face transformers (all text2text-generation models)
- OpenAI InstructGPT models, including ChatGPT and GPT-4
- Azure OpenAI InstructGPT models
- Cohere's command models
- Anthropic's Claude models
- Open source models hosted on Amazon SageMaker
Usage Examples
Stand Alone
You can run PromptNode in your SDK. Just initialize the node and ask a question. The model has general knowledge about the world, so you can ask it anything.
You can use PromptNode as a stand-alone node in deepset Cloud only through Notebooks.
from haystack.nodes import PromptNode
# Initialize the node:
prompt_node = PromptNode()
# Run a prompt
prompt_node("What is the capital of Germany?")
# Here's the output:
['berlin']
With a Custom Prompt
To use a custom prompt, declare PromptTemplate
as your pipeline component and paste the prompt text in the prompt
parameter. Then, pass the name of the PromptTemplate in the default_prompt_template
parameter of PromptNode:
...
components:
- name: question-answering # give this pipeline any name
type: PromptTemplate
params: # the prompt parameter contains the text you copied from the template in Prompt Explorer
prompt: "You are a technical expert. \
You answer questions truthfully based on provided documents. \
For each document check whether it is related to the question. \
Only use documents that are related to the question to answer it. \
Ignore documents that are not related to the question. \
If the answer exists in several documents, summarize them. \
Only answer based on the documents provided. Don't make things up. \
Always use references in the form [NUMBER OF DOCUMENT] when using information from a document. e.g. [3], for Document[3]. \
The reference must only refer to the number that comes in square brackets after passage. \
Otherwise, do not use brackets in your answer and reference ONLY the number of the passage without mentioning the word passage. \
If the documents can't answer the question or you are unsure say: 'The answer can't be found in the text'. \
{new_line}\
These are the documents:\
{join(documents, delimiter=new_line, pattern=new_line+'Document[$idx]:'+new_line+'$content')}\
{new_line}\
Question: {query}\
{new_line}\
Answer:\
{new_line}"
- name: PromptNode
type: PromptNode
params:
default_prompt_template: question-answering # This tells PromptNode to use the PromptTemplate you configured above
...
pipelines:
- name: query
nodes:
- name: Retriever
inputs: [Query]
- name: PromptNode
inputs: [Retriever]
...
In your pipeline, you can have multiple PromptNodes, each with a different prompt. This way, each of them performs a different task. See the In a Pipeline section.
With a Model Specified
To use a proprietary model, make sure you first Connect to Model Providers. That's safer than adding the API key in the YAML.
You specify the model in the model_name_or_path
parameter. Additional model parameters go into model_kwargs
, for example:
components:
- name: PromptNode
type: PromptNode
params:
model_name_or_path: google/flan-t5-xl
model_kwargs:
temperature: 0.6
To use Azure OpenAI service, you must pass the API version, base URL, and deployment name in model_kwargs
like this:
components:
- name: PromptNode
type: PromptNode
params:
model_name_or_path: text-davinci-003
api_key: <azure_openai_api_key>
model_kwargs:
api_version: 2022-12-01
azure_base_url: https://<your-endpoint>.openai.azure.com
azure_deployment_name: <your-deployment-name>
You can find the azure_base_url
parameter in the Keys and Endpoint tab of your Azure account. You choose the azure_deployment_name
when you deploy a model through your Azure account in the Model deployments tab. For available models and versions for the service, check Azure documentation.
For instructions on using a model hosted on SageMaker, see Using a Model Hosted on AWS SageMaker.
For more information about model parameters, see the PromptModel Arguments section.
In a Pipeline
The real power of PromptNode shows when you use it in a pipeline. Look at the example to get an idea of what's possible.
Conversational Summary
Here's an example of how you could use PromptNode to generate a summary of a chat transcript:
name: 'Chat_summary'
version: '1.21.0'
components:
- name: DocumentStore
type: DeepsetCloudDocumentStore # The only supported document store in deepset Cloud
- name: Retriever # Selects the most relevant documents from the document store so that the OpenAI model can base it's generation on it.
type: EmbeddingRetriever # Uses a Transformer model to encode the document and the query
params:
document_store: DocumentStore
embedding_model: sentence-transformers/multi-qa-mpnet-base-dot-v1 # Model optimized for semantic search
model_format: sentence_transformers
top_k: 1 # The number of documents
- name: PromptNode
type: PromptNode
params:
default_prompt_template: deepset/conversational-summary # this is a ready-made prompt template for summarizing transcripts. For more templates, see Prompt Explorer.
model_name_or_path: text-davinci-003
api_key: <api_key>
- name: FileTypeClassifier # Routes files based on their extension to appropriate converters, by default txt, pdf, md, docx, html
type: FileTypeClassifier
- name: TextConverter # Converts files into documents
type: TextConverter
- name: PDFConverter # Converts PDFs into documents
type: PDFToTextConverter
- name: Preprocessor # Splits documents into smaller ones and cleans them up
type: PreProcessor
params:
split_by: word # The unit by which you want to split the documents
split_length: 250 # The max number of words in a document
split_overlap: 20 # Enables the sliding window approach
language: en
split_respect_sentence_boundary: True
pipelines:
- name: query
nodes:
- name: Retriever
inputs: [Query]
- name: PromptNode
inputs: [Retriever]
- name: indexing
nodes:
- name: FileTypeClassifier
inputs: [File]
- name: TextConverter
inputs: [FileTypeClassifier.output_1] # Ensures that this converter receives txt files
- name: PDFConverter
inputs: [FileTypeClassifier.output_2] # Ensures that this converter receives PDFs
- name: Preprocessor
inputs: [TextConverter, PDFConverter]
- name: Retriever
inputs: [Preprocessor]
- name: DocumentStore
inputs: [Retriever]
Mulitple PromptNodes
You can reuse the instance of a model for multiple PromptNodes in your pipeline. This saves resources. To do this, configure the PromptModel as a component and then pass its name in the model_name_or_path
parameter in each instance of PromptNode.
components:
- name: DocumentStore
type: DeepsetCloudDocumentStore # The only supported document store in deepset Cloud
- name: Retriever # Selects the most relevant documents from the document store so that the OpenAI model can base it's generation on it.
type: EmbeddingRetriever # Uses a Transformer model to encode the document and the query
params:
document_store: DocumentStore
embedding_model: sentence-transformers/multi-qa-mpnet-base-dot-v1 # Model optimized for semantic search
model_format: sentence_transformers
top_k: 1 # The number of documents
- name: MyPromptModel
type: PromptModel
params:
model_name_or_path: text-davinci-003
use_gpu: True
- name: PromptNode1
type: PromptNode
params:
default_prompt_template: deepset/question-generation
model_name_or_path: MyPromptModel
api_key: <api_key>
- name: PromptNode2
type: PromptNode
params:
default_prompt_template: deepset/question-answering-per-document
model_name_or_path: MyPromptModel
api_key: <api_key>
# You would also need to configure the components for the indexing pipeline
# We're skipping this part in this example
pipelines:
- name: query
nodes:
- name: Retriever
inputs: [Query]
- name: PromptNode1
inputs: [Retriever]
- name: PromptNode2
inputs: [PromptNode1]
# Here would come the indexing pipeline, which we're skipping in this example
from haystack import Pipeline
from haystack.nodes.prompt import PromptNode, PromptModel
# You'd also need to import a Retriever and a DocumentStore,
# we're skipping this in this example
top_k = 10
query = "Who is Paul Atreides' father?"
prompt_model = PromptModel()
node = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="query")
node2 = PromptNode(prompt_model, default_prompt_template="question-answering-per-document")
# You'd also need to initialize a Retriever with a DocumentStore
# We're skipping this step in this example to simplify it
pipe = Pipeline()
pipe.add_node(component=retriever, name="retriever", inputs=["Query"])
pipe.add_node(component=node, name="prompt_node", inputs=["retriever"])
pipe.add_node(component=node2, name="prompt_node_2", inputs=["prompt_node"])
output = pipe.run(query=query, params={"retriever": {"top_k": 10}})
dict(zip(output["query"], output["answers"]))
Using Different Models in One Pipeline
You can also specify different LLMs for each PromptNode in your pipeline. This way, you create multiple PromptNode instances that use a single PromptNode, which saves computational resources.
# In YAML, you simply specify two PromptNodes, each with a different name and a different model
# Bear in mind that this example is not a complete pipeline, you'd still need to create the indexing pipeline
# and define its components
components:
- name: PromptNodeOpenAI
type: PromptNode
params:
default_prompt_template: deepset/question-answering
model_name_or_path: text-davinci-003
api_key: <my_openai_key>
- name: PromptNodeDefault
type: PromptNode
params:
default_prompt_template: deepset/question-generation
model_name_or_path: google/flan-t5-large
# And now you could put the two nodes together in the query pipeline:
pipelines:
- name: query
nodes:
- name: PromptNodeDefault
inputs: [Query]
- name: PromptNodeOpenAI
inputs: [PromptNodeDefault]
from haystack.nodes. import PromptTemplate, PromptNode, PromptModel
from haystack.pipelines import Pipeline
api_key = "<type your OpenAI API key>"
# Specify the model you want to use:
prompt_open_ai = PromptModel(model_name_or_path="text-davinci-003", api_key=api_key)
# This sets up the default model:
prompt_model = PromptModel()
# Now let make one PromptNode use the default model and the other one the OpenAI model:
node_default_model = PromptNode(prompt_model, default_prompt_template="question-generation", output_variable="questions")
node_openai = PromptNode(prompt_open_ai, default_prompt_template="question-answering")
pipeline = Pipeline()
pipeline.add_node(component=node_default_model, name="prompt_node1", inputs=["Query"])
pipe.add_node(component=node_openai, name="prompt_node_2", inputs=["prompt_node1"])
output = pipe.run(query="not relevant", documents=[Document("Berlin is the capital of Germany")])
output["results"]
Prompt Templates
PromptTemplates work a bit like pipeline components. To use a custom prompt, you declare it like a pipeline component and then you pass its name to PromptNode. See the Usage Example: With a Custom Prompt section.
deepset Cloud comes with a library of out-of-the-box prompt templates ready for you to use for multiple NLP tasks. Each template contains the prompt, which is the instruction for the model. You can browse the available templates in Prompt Explorer by clicking Templates. You can also save your custom prompts there.
PromptTemplate Structure
You can create your own prompt template and save it in the library of Prompt Explorer for future use. If you create a template in Prompt Explorer, it guides you through the structure it should have. But you can also create a prompt directly in your pipeline. If you do this, follow this structure:
components:
- name: my_prompt
type: PromptTemplate
params:
prompt: "Here comes the prompt text"
prompt
contains the prompt for the task you want the model to do. It also specifies input variables in curly brackets. The variables can bedocument
,query
,label
.
At runtime, these variables must be present in the execution context of the node. You can apply functions to those variables. For example, you can combine the list of documents into a string by applying thejoin
function to execute only one prompt instead of one prompt per one document.
You can use\n
,\t
, and\r
characters in the prompt text.output_parser
converts the output to an Answerobject. If you need PromptNode to output an Answer object, set this parameter toAnswerParser()
.
For a detailed explanation, see the PromptTemplate Arguments section.
Functions in Prompts
You can add the join
and to_strings
functions to your template to control how the documents, query, or any other variables are rendered in the output. Functions are useful if you want to, for example, make the model reference a document ID or metadata in the output.
Functions use the Python f-string format and you can use any list comprehension inside a function. You can't use \n
, \t
, or \r
inside a function. Also, double quotes (") are automatically replaced with single quotes (') in the function. Here are the variables you can use in the functions instead:
Special characters not allowed in functions | PromptTemplate variable to use instead |
---|---|
\n | new_line |
\t | tab |
\r | carriage_return |
" | double_quote |
For detailed parameters, you can use in a function, see the Function Arguments section.
The join()
Function
join()
FunctionThe join
function joins all documents into a single string with the content of each document separated by the delimiter you specify. Here's a simple example (line 6):
- name: summary-custom
type: PromptTemplate
params:
prompt: >
Summarize this document: join{documents} \n\n
If the document is not there, answer with: "I can''t answer the question based on the information provided." \n\n
Summary:
The function here joins all documents into a single string. This way, PromptNode only sends one prompt, which is faster. If you don't join the documents, PromptNode sends one prompt for each document.
This example joins all documents in a single string, but it also formats the resulting string so that the output is the meta name of each document followed by its contents:
Context: {' '.join([d.meta['name']+': '+d.content for d in documents])}
' '.join([d.meta['name']+': '+d.content for d in documents])
Here's what the resulting string would look like:
DocumentMetaName1: DocumentContent1 DocumentMetaName2: DocumentContent2 # and so on, for each document
Now, you could tell the model in the prompt to print answers together with the meta name of the document containing the answer.
The to_strings()
Function
to_strings()
FunctionThis function extracts the content field of documents and returns a list of strings. In the example below, it renders each document by its name (document.meta["name"]
) followed by a new line and the contents of the document:
{to_strings(documents, pattern='$name'+new_line+'$content', str_replace={new_line: ' ', '[': '(', ']': ')'})}
{to_strings(documents, pattern='$name'+new_line+'$content', str_replace={new_line: ' ', '[': '(', ']': ')'})}
Arguments
PromptNode Arguments
Use these parameters to configure the PromptNode:
Argument | Type | Possible Values | Description |
---|---|---|---|
model_name_or_path | String | Default: google/flan-t5-base | The name of the model you want to use with the PromptNode or the path to a locally stored model. Mandatory. |
default_prompt_template | String | Name of the out-of-the-box template or the name of your custom template. | The prompt template you want to use with the PromptNode. The template contains instructions for the model. If you don't specify it, the model tries to guess what task you want it to do based on your query. For best results, we recommend specifying the template. Optional. |
output_variable | String | Default: None | The name of the output variable where you want to store the inference results. If not specified, PromptNode uses the output_variable from the PromptTemplate. If PromptTemplate's output_variable is not set, the default name is results .Optional. |
max_length | Integer | Default: 100 | The maximum number of tokens of the text output the PromptNode generates. The length of the prompt and the length of the generated output must not be larger than the number of tokens the LLM can process. Optional. |
api_key | String | Default: None | The API key for the model. Specify it to use a model by OpenAI. Optional. |
use_auth_token | String | Default: None | The Hugging Face authentication token for your private model. Optional. |
use_gpu | Boolean | True False Default: None | Specifies if you want to use GPU when running PromptNode. Optional. |
devices | List of strings | Example: [torch.device('cuda:0'), "mps", "cuda:1"] | The list of torch devices to use. Optional. |
stop_words | List of strings | Default: None | If the PromptNode encounters any of the words you specify here, it stops generating text. Optional. |
top_k | Integer | Default: 1 | The number of answers (generated texts) you want PromptNode to return. Mandatory. |
debug | Boolean | True False (default) | Enables debug output. Optional. |
model_kwargs | Dictionary | Default: None | Any additional keyword arguments you want to pass to the model, for example: model_kwargs: temperature: 0.7 Optional. |
PromptTemplate Arguments
These are the arguments you can use when creating your own PromptTemplate:
Argument | Type | Possible Values | Description |
---|---|---|---|
prompt | String | The instructions for the model. It can contain variables in the f-string syntax, for example: {documents} . These variables need to be filled in the prompt_text for the model to perform the task. Note that other than strict f-string syntax, you can safely use the following backslash characters in text parts of the prompt text: \n, \t, \r . If you want to use them in f-string expressions, use new_line , tab , carriage_return instead. Double quotes (") are automatically replaced with single quotes (') in the prompt text. To use double quotes in the prompt text, use {double_quote} instead.Mandatory. | |
output_parser | String | AnswerParser() | Applies a parser that converts the model to an Answer object. If you want to use it, always set it to AnswerParser() .Optional. |
Function Arguments
Parameter | Type | Description |
---|---|---|
documents | List | The documents whose rendering you want to format. Mandatory. |
pattern | String | The regex pattern used for parsing. You can use the following placeholders in pattern: - $content : The content of the document.- $idx : The index of the document in the list.- $id : The ID of the document.- $META_FIELD : The values of the metadata field called META_FIELD (Only fields that match python's string Template pattern are supported. That is: [\_a-z][_a-z0-9]\* ). (Only fields that match python's string Template pattern are supported. That is: [\_a-z][_a-z0-9]\* ).Optional. |
delimiter | String Default: " " (single space) | Specifies the delimiter you want to use to separate documents. Used in the join function.Mandatory. |
str_replace | Dictionary of strings | Specifies the characters you want to replace. Use the format `str_replace={"r":"R"}. Optional. |
PromptModel Arguments
Argument | Type | Possible Values | Description |
---|---|---|---|
model_name_or_path | String | Default: google/flan-t5-base | The name of the model or the path to a locally stored model. You can use the following models: - Hugging Face transformers (all text2text-generation and text-generation models). - OpenAI InstructGPT models. - Azure OpenAI InstructGPT models. Mandatory. |
max_length | Integer | Default: 100 | The maximum length of the output text the model generates. Optional. |
api_key | String | Default: None | The API key to use for the model. Optional. |
use_auth_token | Unionk:parameter | Default: None | If the model is hosted on Hugging Face, this is the token to use for the model. Optional. |
use_gpu | Boolean | Default: None | Uses GPU if available. Optional. |
devices | String | A list of GPU devices Example: [torch.device('cuda:0'), "mps", "cuda:1"] | Contains a list of GPU devices to limit inference to certain GPUs and not use all available GPUs. As multi-GPU training is currently not implemented for DPR, training only uses the first device provided in this list. Optional. |
invocation_layer_class | PromptModelInvocationLayer | Default: None | The custom invocation layer class to use. If you don't specify it, it uses one of the known invocation layers. Optional. |
model_kwargs | Dictionary | Default: None | Additional keyword arguments passed to the underlying model. Note that Azure OpenAI InstructGPT models require two additional parameters: azure_base_url (the URL for the Azure OpenAI API endpoint, usually in the form https://<your-endpoint>.openai.azure.com , you can find it in the Keys and Endpoint tab of your Azure account) and azure_deployment_name (the name of the Azure OpenAI API deployment you specify when deploying a model through Azure in the Model deployments tab). You should pass these parameters in the model_kwargs dictionary. |
Updated 2 days ago