Upload an Evaluation Set

You need an evaluation set to run an experiment. An evaluation dataset is a sample of data that your model never saw during the training. It contains gold answers which are then compared against the actual answers that your pipeline retrieved during evaluation.


You must be an Admin to perform this task.

The evaluation dataset in deepset Cloud is based on the files that you uploaded to Data>Files. After you add your evaluation set, deepset Cloud validates if it matches at least one of the files in your workspace. If it doesn't, deepset Cloud lets you know.

The evaluation dataset must be a .csv file with the following columns:

  • question
  • text
  • context
  • file_name
  • answer_start
  • answer_end
  • filters (optional)
Why would I need filters?

If you want to use the same filters you'd use in a real-life search for your experiment, you can add your filters to labels in your evaluation set using the _Filters _column. Add the filters in the following format: {"key1":"value1", "key1":"value2", "key2":"value3"}.

An example evaluation set

This example is meant to help you understand the formatting of the file and what a sample answer can look like.

Upload an Evaluation Dataset from the UI

  1. Log in to deepset Cloud and go to Data>Evaluation Sets>Import Eval Sets.
  2. Drop your file on the Evaluation Set Import page.
  3. Click Upload File under your file.

You can find your evaluation set in Data_Evaluation Sets.

Upload an Evaluation Dataset with Python SDK

You can also run it from Jupyter Notebooks within deepset Cloud; just go to Notebooks in the left navigation. If you use Notebooks, make sure you upload the evaluation dataset to the Notebooks folder.

You need to Generate an API Key first.

Use this code to upload your evaluation sets:

# The necessary imports
import os
from pathlib import Path
from haystack.utils import DeepsetCloud

# Set the API key and API endpoint:
os.environ["DEEPSET_CLOUD_API_ENDPOINT"] = "https://api.cloud.deepset.ai/api/v1"

# Upload your eval sets:
evaluation_set_client = DeepsetCloud.get_evaluation_set_client(workspace="your_workspace")
# If you don't specify the workspace for the evaluation set client, it uses the default one.
evaluation_set_path = Path("my-eval-set.csv")

evaluation_set_client.get_evaluation_set("my-eval-set")  # Provide only the name without the extension

Upload an Evaluation Dataset with REST API

Here's the code to use. You need to Generate an API Key first.

curl --request POST \
       --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/evaluation_sets/import \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --header 'Content-Type: multipart/form-data' \
     --form '[email protected]_NAME.csv'

For more information about this endpoint, see Import Evaluation Set API.

Related Links