Create and run experiments to evaluate your deepset Cloud pipelines using your Python methods.
Prerequisites
- You must be an Admin to perform this task.
- If you work from your local SDK, you must have Haystack installed. For more information, see Haystack Installation. If you use Jupyter Notebooks in deepset Cloud, you don't have to worry about that.
- Add API endpoint and API key to the environment variables. The API endpoint is
<https://api.cloud.deepset.ai/api/v1>. See Generate an API Key.
Server restart
If the Notebooks server closes while you're still working, all the files you saved are still there. When you restart the server, you'll be able to work on them again.
Manage Experiment Runs
These are the DeepsetCloudExperiment class methods that you can use to manage your experiments runs in deepset Cloud. This class also has methods to list your pipelines and evaluation sets, as you'll need them to manage your experiment runs.
To run an experiment from SDK:
- List available pipelines to choose the pipeline you want to evaluate.
- List evaluation sets to choose the evaluation set you want to use in your experiment.
- Create an experiment run and start it.
- Track the experiment run.
- Review the run results.
List Pipelines
To start an experiment run, you must first choose a pipeline that you want to evaluate in your experiment. To do this, list all the available pipelines in deepset Cloud.
This table lists the parameters that you can use with the list_pipelines() method:
| Method | Parameters | Description |
|---|---|---|
list_pipelines() | workspace - specifies the deepset Cloud workspace containing the pipelines. String. Optional. Set this value to default.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1>. | Lists all the pipelines available on deepset Cloud as a list of dictionaries. Each dictionary contains the name of the pipeline and pipeline metadata, such as status or creation date. |
Example of usage
from haystack.utils import DeepsetCloudExperiments
pipeline_list = DeepsetCloudExperiments.list_pipelines(workspace="default")
print(pipeline_list)
List Evaluation Sets
When creating an experiment run, specify the evaluation set you want to use.
This table lists the parameters that you can use to list evaluation sets:
| Method | Parameters | Description |
|---|---|---|
list_evaluation_sets() | workspace - specifies the deepset Cloud workspace containing the evaluation sets. String. Optional. Set this value to default.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1>. | Lists all the evaluation sets available on deepset Cloud. It returns a list of dictionaries. Each dictionary contains the evaluation set name that you can use in create_run() and evaluation set metadata, such as the date when it was created or the number of labels it contains. |
Example of usage
from haystack.utils import DeepsetCloudExperiments
eval_sets = DeepsetCloudExperiments.list_evaluation_sets(workspace="default")
print(eval_sets)
Create an Experiment Run
Specify all the parameters for your experiment run and save it without starting.
This table describes the parameters that you can use when creating an experiment run:
| Method | Parameters | Description |
|---|---|---|
create_run() | eval_run_name - specifies the name of the evaluation run. String. Required.workspace - specifies the deepset Cloud workspace containing the experiment runs. String. Optional. Set this value to default.pipeline_config_name - specifies the name of the pipeline you want to use in this experiment run. You can use list_pipelines() to see all the pipelines in deepset Cloud. String. Optional.evaluation_set - specifies the evaluation set you want to use for this experiment run. Use list_evaluation_sets() to view all evaluation sets in deepset Cloud. String. Optional.eval_mode - the evaluation mode that you want to use. Possible values are: integrated and isolated. The default value is integrated. Required.debug - enables collecting debugging information. Possible values are False and True. The default value is False. Required.comment - adds a comment to the experiment run. String. Optional.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1>. | Creates an evaluation run in deepset Cloud. |
Example of usage
from haystack.utils import DeepsetCloudExperiments
my_experiment = DeepsetCloudExperiments.create_run(
eval_run_name="Experiment1")
Update an Experiment Run
Edit your experiment run. You can update all the parameters.
This table describes the parameters that you can use when updating an experiment run:
| Method | Parameters | Description |
|---|---|---|
update_run() | eval_run_name - specifies the name of the experiment run that you want to update. String. Required.workspace - specifies the deepset Cloud workspace containing the experiment runs. String. Optional. Set this value to default.pipeline_config_name - specifies the name of the pipeline you want to use in this experiment run. You can use list_pipelines() to see all the pipelines in deepset Cloud. String. Optional.evaluation_set - specifies the evaluation set you want to use for this experiment run. Use list_evaluation_sets() to view all evaluation sets in deepset Cloud. String. Optional.eval_mode - the evaluation mode you want to use. Possible values: integrated and isolated. Default: integrated. Required.debug - enables collecting debugging information. Possible values: False and True. Default: False. Required.comment - adds a comment to the experiment run. String. Optional.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1>.tags - contains a list of tags that you want to associate with this evaluation run. It's optional. | Modifies an experiment run. |
Example of usage
from haystack.utils import DeepsetCloudExperiments
updated_run = DeepsetCloudExperiments.create_run(
eval_run_name="Experiment1", pipeline_config_name="new_pipeline")
List Experiment Runs
You can view a particular experiment run or all experiment runs created in deepset Cloud.
This table describes the parameters that you can use when listing experiment runs:
| Method | Parameters | Description |
|---|---|---|
get_run() | eval_run_name - specifies the name of the experiment run that you want to view. String. Required.workspace - specifies the deepset Cloud workspace containing the experiment run. String. Optional. Set this value to default.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1> | List a single experiment run. It returns a dictionary containing the name of the experiment and its parameters and metrics. |
get_runs() | workspace - specifies the deepset Cloud workspace containing the experiment runs. String. Optional. Set this value to default.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1> | Lists all the experiment runs created in deepset Cloud. It returns a list of dictionaries. Each dictionary contains the name of the experiment run and additional information such as parameters, metrics, status, and more. |
Example of usage
from haystack.utils import DeepsetCloudExperiments
experiment_runs = DeepsetCloudExperiments.get_runs(workspace="default")
print(experiment_runs)
Start an Experiment Run
Use this method to start an experiment you saved as a draft earlier.
This table describes the parameters that you can use when starting an experiment run:
| Method | Parameters | Description |
|---|---|---|
start_run() | eval_run_name - the name of the experiment run that you want to run. String. Required.workspace - the name of the deepset Cloud workspace where you want to run the experiment. If none is specified, the default workspace is used. String. Required.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1>pipeline_config_name - specifies the name of the pipeline that you want to evaluate. To list all the pipelines, use list_pipelines() String. Optional.evaluation_set - specifies the name of the evaluation set that you want to use for the experiment. To list all available evaluation sets, use list_evaluation_sets(). | Starts an experiment run in deepset Cloud. |
Example of usage
form haystack.utils import DeepsetCloudExperiments
my_experiment = DeepsetCloudExperiments.start_run(eval_run_name="Experiment1")
Create and Start an Experiment Run
You can create an experiment run and start it using one method. An alternative is to create an experiment run and save it as a draft.
This table lists the parameters that you can use when creating and starting an experiment run:
| Method | Parameters | Description |
|---|---|---|
create_and_start_run() | eval_run_name - the name of the experiment run that are creating. String. Required.workspace - the name of the deepset Cloud workspace where you want to create and run the experiment. If none is specified, the default workspace is used. String. Required.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1>pipeline_config_name - specifies the name of the pipeline that you want to evaluate. To list all available pipelines, use list_pipelines(). String. Optional.evaluation_set - specifies the name of the evaluation set to use for this experiment. Use list_evaluation_sets() to see all available evaluation sets. String. Optional.eval_mode - specifies the evaluation mode to use. Possible values are integrated and isolated. The default value is integrated. String. Required.debug - enables the collection of debug information for the experiment. Possible values are True and False. The default value is False. Reqiured.comment - adds a comment to this experiment. String. Optional. | Creates and starts an experiment run. |
Example of usage
from haystack.utils import DeepsetCloudExperiments
my_experiment = DeepsetCloudExperiments.create_and_start_run(
eval_run_name="Experiment1", pipeline_config_name="Hybrid Pipeline", debug="True")
Review an Experiment Run
Check the results of your experiment run.
Here are the parameters that you can pass to this method:
| Methods | Parameters | Description |
|---|---|---|
get_run_result() | eval_run_name - type the name of the evaluation run whose results you want to check.workspace - specify the workspace where the run is. If you set it to None, the EvaluationRunClient uses its default workspace.api_key - the secret value of the API key to your deepset Cloud workspace. If you don't enter the key, it's read from the DEEPSET_CLOUD_API_KEY_ environment variable.api_endpoint - the URL of the deepset Cloud API. If you don't specify it, it's read from the DEEPSET_CLOUD_API_ENDPOINT envirnoment variable. If you don't have the environment variable set, it uses the default value, which is <https://api.cloud.deepset.ai/api/v1>. | Fetches the results of an evaluation run. |
Delete an Experiment Run
You can delete an experiment run from deepset Cloud.
This table describes the parameters that you can use when deleting an experiment run:
| Method | Parameters | Description |
|---|---|---|
delete_run | eval_run_name - the name of the experiment run that you want to delete. String. Required.workspace - the name of the deepset Cloud workspace where the experiment run exists. If none is specified, the default workspace is used. String. Required.api_key - contains the secret value of the deepset Cloud API key. If no value is specified, it is read from the DEEPSET_CLOUD_API_KEY environment variable. String. Optional.api_endpoint - specifies the URL of the deepset Cloud API. If not specified, it is read from the DEEPSET_CLOUD_API_ENDPOINT environment variable. String. Optional.The endpoint should be <https://api.cloud.deepset.ai/api/v1> | Removes an experiment run from deepset Cloud. |
Example of usage
from haystack.utils import DeepsetCloudExperiments
DeepsetCloudExperiments.delete_run(eval_run_name="Experiment1")