Engineering Prompts

Sometimes, you need to iterate on your prompts to get the best results. deepset Cloud offers Prompt Studio, where you can test your prompts and compare how they perform. It comes with a library of ready-to-use prompts where you can also save your own prompts.

About This Task

Prompt Studio is a sandbox environment where you can:

  • Test different versions of your prompts before bringing them to production.
  • Modify your prompts.
  • Compare the same prompt across up to three pipelines.
  • Browse through and use curated prompt templates from a library.
  • Save custom prompts to the library and reuse them later.

Accessing Prompt Studio

Prompt Studio is only available for deployed pipelines that use at least one PromptNode.

In the navigation, click Prompt Studio:

deepset Cloud navigation with Prompt Studio highlighted

Testing and Modifying Prompts

Use the Prompt Editor within the Prompt Studio to modify your prompts and observe how they affect the results. You can experiment with different versions of prompts for the same query and review the history of your results.

Prompt Studio supports Markdown formatting of prompts.

Using the expanding list at the top, you can select a pipeline. Only deployed pipelines using at least one PromptNode are available for selection. Once you choose a pipeline, its prompt immediately shows in the Prompt Editor at the bottom. If there are multiple PromptNodes in the pipeline, you can choose the one you want to work with.

You can use the Editor to modify your prompt and run queries. To check the history of your queries, click the clock icon next to the query (it's only available if you select a single pipeline).

Prompt Explorer window. At the top, there's a pipeline selection field. Then, the middle of the page shows search history for queries "How to cure swollen gum after removing a wisdom tooth" and "My head aches, what do I do?". At the bottom, there's the Prompt Editor section showing the current prompt that can be edited. At the very bottom of the page, there's a text field where you can type your query, and a Submit button next to it.

By clicking the More Actions icon next to a result, you can view the prompt that generated this result.

Using Prompt Templates

Choose a prompt from a library and test it in Prompt Studio. The prompt you choose shows in Prompt Editor where you can further modify it if needed.

In Prompt Editor, click Templates. The Custom tab lists the prompts you saved, and the deepset tab lists prompts curated and tested by deepset.

Prompt Explorer with the prompt showing in prompt editor and the templates button highlighted.

Saving Your Custom Prompts

Save your prompts in the library so you can use them in the future.

  1. In Prompt Studio, click Templates.
  2. On the Custom tab, click Create Custom Prompt.
  3. Name your prompt, type the prompt text, and save your prompt. Use curly brackets to insert a variable that will be filled in at search time. For example {documents}.

Updating the Prompt in the Pipeline

Once you've tested the new prompt and decided you want to use it in your pipeline, you must update your pipeline with the prompt:

  1. Copy the prompt you want to use in your pipeline into a Notepad. (You can copy the prompt from Prompt Editor or from the templates if you saved it there.)

  2. In the navigation, click Pipelines and find the pipeline whose prompt you want to update.

  3. Click the More Actions menu next to the pipeline and choose Edit.

    The More Actions menu expanded with the Edit option highlighted
  4. In Pipeline Designer, find the PromptTemplate component and paste the new prompt its prompt parameter. This is an example of what it can look like:

    Pipeline YAML with the PromptTemplate component indicated and its prompt parameter highlighted
  5. Save and deploy the pipeline.

Comparing Pipelines

You can compare up to three pipelines. This is useful if you have pipelines using different models to understand how they perform. You pass the same query to all the pipelines you're comparing, but you can modify the prompts for each pipeline.

To add a pipeline to comparison, click Add Pipeline at the top of the Playground.

Add pipeline button highlighted in Prompt Explorer window

You can then add or remove pipelines using the plus (+) and minus (-) icons:

The add/remove pipeline buttons highlighted.

Each pipeline receives the same query, but you can change the prompt for each pipeline.