Create an Experiment Run

Experiments let you evaluate your pipeline in a systematic way. An experiment run is a single trial of an experiment that lets you monitor your pipeline performance.

๐Ÿ“˜

You must be an Admin user to perform this task.

About This Task

When creating an experiment run, you can run it immediately or save it as a draft and run it later.

To learn more about experiments, see About Experiments.

Prerequisites

Create an Experiment Run from the UI

  1. Log in to deepset Cloud and go to Experiments > New Experiment.
  2. Choose the pipeline that you want to evaluate.
  3. Choose the evaluation dataset that you want to use for this experiment.
  4. Give your experiment a meaningful name. You can also add tags that will let you identify the experiment later.
    You create tags for the whole workspace; they're not tied to a single experiment. You can use the same tag for multiple experiments.
  5. Choose one of the following:
  • To start the experiment now, click Start Experiment. The experiment starts running.
  • To save your experiment as a draft, click Save as Draft. You'll be able to run the experiment later.

Create an Experiment Run with REST API

Before you start, you must prepare a couple of things:

You're now ready to create an experiment run:

  1. Define your experiment run. Use the create eval run API endpoint. Follow the step-by-step code explanation:

Or copy this code:

curl --request POST \
     --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/eval_runs \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --header 'Content-Type: application/json' \
     --data '
{
     "tags": [
          "<your_tag1>",
          "<your_tag2>"
     ],
     "comment": "This is a comment",
     "debug": true,
     "evaluation_set_name": "<eval_set_name>",
     "name": "<experiment_run_name>",
     "pipeline_name": "<pipeline_name>"
}
'
  1. Start the experiment run using the start eval run endpoint. Follow the step-by-step code explanation:

Or copy this code:

curl --request POST \
     --url https://api.cloud.deepset.ai/api/v1/workspaces/<your_workspace>/eval_runs/<your_eval_run/start \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer <your API key>'