Tutorial: Creating an Experiment for a QA Pipeline

Use this tutorial to learn how to create an experiment to evaluate your question answering pipeline. The instructions here guide you through the steps to create your first experiment. They also contain all the data you need for it.

  • Level: Beginner
  • Time to complete: 15 minutes
  • Prerequisites:
    • This tutorial assumes a basic knowledge of NLP and the concept of pipeline evaluation. If you need more information, look at About Experiments.
    • You must be an Admin to complete this tutorial.
  • Goal: After completing this tutorial, you will have created and run an experiment to evaluate a question answering pipeline.
    This tutorial contains all the necessary files and an evaluation dataset, but you can replace them with your own. It also guides you through creating a QA pipeline from a ready-made template but you can also use a previously created pipeline.

Upload Files

Your pipeline will run the search on these files.

  1. Download the .zip file with sample files and unpack it on your computer.

  2. Log in to deepset Cloud, make sure you're in the right workspace, and go to Files.

  3. Click Upload Files.

  4. Select all the files you extracted and drop them into the Upload Files window. There should be 344 files in total.

  5. Click Upload and wait until the files are uploaded.

Result: Your files are in your workspace, and you can see them on the Files page.

A screenshot of the Files page with the files uploaded and listed there

Upload an Evaluation Set

You need a set of annotated data your pipeline will be evaluated against.

  1. Download the CSV file and save it on your computer.
  2. In deepset Cloud, go to Evaluation Sets and click Upload Evaluation Sets.
  3. Drop the CSV file you downloaded in step 1 in the Upload Evaluation Sets window and click Upload. Wait for the confirmation that it uploaded without problems.

Result: The evaluation dataset is uploaded, and you can see it on the Evaluation Sets page.

A screenshot of the Evaluation Sets page with the annotations_jaz eval set uploaded and displayed

Create a Pipeline to Run the Experiment On

If you already have a pipeline you want to evaluate, you can skip this part.

  1. In deepset Cloud, go to Pipeline Templates.
  2. Choose Basic QA, hover your mouse over the Extractive Question Answering template, and click Use Template.
The Create a pipeline in YAML Editor window with test_experiment as the pipeline name, the extractive question answering template selected and the create button at the bottom highlighted.
  1. Type test_experiment and the pipeline name and click Create Pipeline.

Result: You have created a question answering pipeline that you'll evaluate next. Your pipeline is displayed on the Pipelines page.

A screenshot of the Pipelines page with the Test Experiment pipeline displayed under in development.

Create an Experiment

Now it's time to evaluate your pipeline.

  1. Go to Experiments>New Experiment.

  2. Choose test_experiment as the pipeline.

  3. Choose annotations_jazz as the evaluation set.

  4. Type jazz as the experiment name and add test as a tag.

  5. Click Start Experiment. You can see that jazz is running. Wait until it completes. It may take a couple of minutes.

    The Experiments page with the newly started experiment running.
  6. When the experiment status changes to Completed, click its name to view its details, such as the data and the pipeline, the metrics, and predictions.

Result: Congratulations! You just created an experiment and ran it to evaluate your pipeline.

A screenshot of the experiments page with the jazz experiment displayed as completed

You can now review the results on the experiment details page.

A screenshot of the Experiment Details page showing all the experiment data such as evaluation set used, date started, debug information, and metrics.