Register to deepset CloudLog in to deepset Cloud

Upload an Evaluation Set

You need an evaluation set to run an experiment. An evaluation dataset is a sample of data that your model never saw during the training. It contains gold answers which are then compared against the actual answers that your pipeline retrieved during evaluation.

Suggest Edits

📘

You must be an Admin to perform this task.

The evaluation dataset in deepset Cloud is based on the files you uploaded to Data>Files. After you add your evaluation set, deepset Cloud validates if it matches at least one of the files in your workspace. If it doesn't, deepset Cloud lets you know.

The evaluation dataset must be a .csv file with the following columns:

  • question
  • text
  • context
  • file_name
  • answer_start
  • answer_end
  • filters (optional)
Why would I need filters?

If you want to use the same filters you'd use in a real-life search for your experiment, you can add your filters to labels in your evaluation set using the _Filters_column. Add the filters in the following format: {"key1":"value1", "key1":"value2", "key2":"value3"}.

An example evaluation set

This example is meant to help you understand the file's formatting and what a sample answer can look like.

drive.google.comdrive.google.com
annotations_jazz_short.csv

Upload an Evaluation Dataset from the UI

  1. Log in to deepset Cloud and go to Data>Evaluation Sets>Upload Evaluation Sets.
  2. Drop your file into the Upload Evaluation Sets window and click Upload.
  3. Wait for the upload to finish.

Upload an Evaluation Dataset with Python SDK

You can also run it from Jupyter Notebooks within deepset Cloud; just go to Notebooks in the left navigation. If you use Notebooks, upload the evaluation dataset to the Notebooks folder.

You need to Generate an API Key first.

Use this code to upload your evaluation sets:

# The necessary imports
import os
from pathlib import Path
from haystack.utils import DeepsetCloud

# Set the API key and API endpoint:
os.environ["DEEPSET_CLOUD_API_KEY"] = "<YOUR_API_KEY>"
os.environ["DEEPSET_CLOUD_API_ENDPOINT"] = "https://api.cloud.deepset.ai/api/v1"

# Upload your eval sets:
evaluation_set_client = DeepsetCloud.get_evaluation_set_client(workspace="your_workspace")
# If you don't specify the workspace for the evaluation set client, it uses the default one.
evaluation_set_path = Path("my-eval-set.csv")
evaluation_set_client.upload_evaluation_set(file_path=evaluation_set_path)

evaluation_set_client.get_evaluation_set("my-eval-set")  # Provide only the name without the extension

Upload an Evaluation Dataset with REST API

Here's the code to use. You need to Generate an API Key first.

curl --request POST \
       --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/evaluation_sets/import \
     --header 'Accept: application/json' \
     --header 'Authorization: Bearer <YOUR_API_KEY>' \
     --header 'Content-Type: multipart/form-data' \
     --form '[email protected]_NAME.csv'

For more information about this endpoint, see Import Evaluation Set API.

Updated about 2 months ago

Related Links