- Time to complete: 15 minutes
- This tutorial assumes a basic knowledge of NLP and the concept of pipeline evaluation. If you need more information, have a look at About Pipeline Evaluation.
- You must be an Admin to complete this tutorial.
- Goal: After completing this tutorial, you will have created and run an experiment to evaluate a pipeline.
Your pipeline will run the search on these files.
Download the .zip file with sample files and unpack it on your computer.
Log in to deepset Cloud, make sure you're in the right workspace, and go to Data>Files.
Click Upload Files.
Select all the files you extracted and drop them on the Upload Your Files page. There should be 344 files in total.
Click Upload and wait until the files are uploaded.
Result: Your files are in your workspace and you can see them on the Files page.
You need a set of annotated data your pipeline will be evaluated against.
- Download the CSV file and save it on your computer.
- In deepset Cloud, go to Data>Evaluation Sets and click Import Eval Sets.
- Drop the CSV file you downloaded on the Evaluation Set Import page and click Upload File. Wait for the confirmation that it uploaded without problems and click Go to Evaluation Sets.
Result: The evaluation dataset is uploaded and you can see it on the Evaluation Sets page.
In deepset Cloud, go to Pipelines>New Pipeline.
In YAML Editor, click Create Pipeline and select From Template.
Choose the English Question Answering template.
In the YAML editor, in line 8, find
nameand change it to
In line 17, change the retriever type to
ElasticsearchRetrieverand delete lines 20 and 21 (
top_kparameter value in line 22 to
This is what your pipeline should look like now:
- Save your pipeline.
Result: You have created a question answering pipeline that you're going to evaluate next. Your pipeline is displayed on the Pipelines page.
Now it's time to evaluate your pipeline.
Go to Experiments>New Experiment.
Choose Test_experiment as the pipeline.
Choose annotations_jazz as the evaluation set.
Type jazz as the experiment name and add test as a tag.
Click Start Experiment. You can see that jazz is running. Wait until it completes. It may take a couple of minutes.
When the experiment status changes to
Completed, click its name to view its details, such as the data and the pipeline used, the metrics, and predictions.
Result: Congratulations! You just created an experiment and ran it to evaluate your pipeline.
You can now review the results on the experiment details page.
Updated about 1 month ago