Register to deepset CloudLog in to deepset Cloud

Upload an Evaluation Set

You need an evaluation set to run an experiment. An evaluation dataset is a sample of data that your model never saw during the training. It contains gold answers which are then compared against the actual answers that your pipeline retrieved during evaluation.

Suggest Edits

📘

You must be an Admin to perform this task.

The evaluation dataset in deepset Cloud is based on the files you uploaded to Data>Files. After you add your evaluation set, deepset Cloud validates if it matches at least one of the files in your workspace. If it doesn't, deepset Cloud lets you know.

The evaluation dataset must be a .csv file with the following columns:

  • question
  • text
  • context
  • file_name
  • answer_start
  • answer_end
  • filters (optional)
Why would I need filters?

If you want to use the same filters you'd use in a real-life search for your experiment, you can add your filters to labels in your evaluation set using the _Filters_column. Add the filters in the following format: {"key1":"value1", "key1":"value2", "key2":"value3"}.

An example evaluation set

This example is meant to help you understand the file's formatting and what a sample answer can look like.

drive.google.comdrive.google.com
annotations_jazz_short.csv

Upload an Evaluation Dataset from the UI

  1. Log in to deepset Cloud and go to Data>Evaluation Sets>Upload Evaluation Sets.
  2. Drop your file into the Upload Evaluation Sets window and click Upload.
  3. Wait for the upload to finish.

Upload an Evaluation Dataset with REST API

Here's the code to use. You need to Generate an API Key first.

Follow the step-by-step instructions:

▶️
Import Evaluation Dataset
Open Recipe

Or copy this code:

curl --request POST \
       --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/evaluation_sets/import \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --header 'Content-Type: multipart/form-data' \
     --form 'file=@FILE_NAME.csv'

For more information about this endpoint, see Import Evaluation Set API.

Updated about 18 hours ago

Related Links