📘

You must be an Admin user to perform this task.

About This Task

When creating an experiment run, you can run it immediately or save it as a draft and run it later.

To learn more about experiments, see About Experiments.

Prerequisites

• Prepare the evaluation dataset. For more information, see Evaluation Datasets and Tutorial: Creating an Evaluation Dataset for Document Search.
• If you want to use SDK or REST API, Generate an API Key.

Create an Experiment Run from the UI

1. Log in to deepset Cloud and go to Experiments > New Experiment.
2. Choose the pipeline that you want to evaluate.
3. Choose the evaluation dataset that you want to use for this experiment.
4. Give your experiment a meaningful name. You can also add tags that will let you identify the experiment later.
   You create tags for the whole workspace; they're not tied to a single experiment. You can use the same tag for multiple experiments.
5. Choose one of the following:

• To start the experiment now, click Start Experiment. The experiment starts running.
• To save your experiment as a draft, click Save as Draft. You'll be able to run the experiment later.

Create an Experiment Run with REST API

Before you start, you must prepare a couple of things:

• Generate an API Key.
• Find out the name of the pipeline and the evaluation set you want to use for this experiment run. You can use the following endpoints to list them:
  • get evaluation sets
  • list pipelines

You're now ready to create an experiment run:

1. Define your experiment run. Use the create eval run API endpoint. Follow the step-by-step code explanation:

Or copy this code:

curl --request POST \
     --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/eval_runs \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --header 'Content-Type: application/json' \
     --data '
{
     "tags": [
          "<your_tag1>",
          "<your_tag2>"
     ],
     "comment": "This is a comment",
     "debug": true,
     "evaluation_set_name": "<eval_set_name>",
     "name": "<experiment_run_name>",
     "pipeline_name": "<pipeline_name>"
}
'

2. Start the experiment run using the start eval run endpoint. Follow the step-by-step code explanation:

Or copy this code:

curl --request POST \
     --url https://api.cloud.deepset.ai/api/v1/workspaces/<your_workspace>/eval_runs/<your_eval_run/start \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer <your API key>'