• Level: Intermediate
• Time to complete: 10 minutes
• Prerequisites:
  • You must be an Admin to complete this tutorial.
  • You need a basic knowledge of Python.
  • The workspace where you want to upload the files must already be created in deepset Cloud. In this tutorial, it's called hotel_reviews.
• Goal: After completing this tutorial, you will have uploaded a set of hotel reviews with metadata to a deepset Cloud workspace using a Python script. You can replace this dataset with your custom one.

Prepare Your Files

This tutorial uses a set of hotel reviews with some metadata in them. You can also use your own files; just make sure they're in the TXT or PDF format.

1. Download the hotel reviews dataset.
2. Extract the files to a folder called hotel_reviews in your Documents folder. This can take a couple of minutes.

Result: You have 5,956 files in the \Documents\hotel_reviews folder, 2978 TXT files and 2978 JSON files. Each TXT file is accompanied by a .meta.json file containing the text file metadata.

Install the SDK

1. Open the command line and run:
   PowerShell

   pip install deepset-cloud-sdk
   

2. Wait until the installation finishes with a success message.

Result: You have installed the deepset Cloud SDK. It comes with a command line interface that we'll use to upload the files.

Obtain the API Key

1. Log in to deepset Cloud.
2. Click your name in the top right corner and select Connections.
3. Under API Keys, click Add new key.
4. Select the expiration date for your key and click Generate key.
5. Copy the key and save it to a notepad.
6. Click Done.

Result: You have an API key saved in a file. You can now use it to upload your files.

Upload Files

1. Write a script that will upload the files from a specified path:
   1. In the same folder where you saved the hotel_reviews files, create a Python script called hotel_reviews_upload.py.
   2. Follow the step-by-step explanation or copy the code from the example below:

from pathlib import Path

from deepset_cloud_sdk.workflows.sync_client.files import upload

## Uploads all files from a given path.
upload(
    paths=[Path("<your_path_to_the_hotel_reviews_folder>")], # provide a path to the folder on your computer where you saved the hotel_reviews folder
    api_key="<your_deepsetCloud_api_key>", # the API key to connect to deepset Cloud
    workspace_name="<default_workspace>", # the workspace where you want to upload files
    blocking=True,  # waits until the files are displayed in deepset Cloud,
    # this may take a couple of minutes
    timeout_s=300,  # the timeout for the `blocking` parameter in number of seconds
    show_progress=True,  # shows the progress bar
    recursive=True,  # uploads files from all subfolders as well
)

3. Save the script and run it:
   Python

   python hotel_reviews_upload.py
   

4. Wait until the upload finishes successfully. You should see this message:
   
   5956 files are uploaded, and half of them, 2978, are listed in deepset Cloud. (This is because the metadata files are not shown in deepset Cloud).

Result: You have uploaded all your files, including the ones from the subfolders. Let's now see if they're showing up in deepset Cloud.

Verify the Upload

• In the command line, list the uploaded files by running:

  MacOSWindows

  deepset-cloud list-files
  

  python -m deepset_cloud_sdk.cli list-files
  

  You should see a list of files with file ID, URL, name, size, metadata, and the date when it was created.

  
  With the number of files we uploaded, it's easier to verify if they uploaded correctly in the deepset Cloud UI.

• In deepset Cloud:

  1. Switch to the hotel_reviews workspace where you uploaded the files and check the workspace statistics. You should see "3K" above files.
  2. In the navigation, click Files and check if the files are showing on the Files page.

• Now, let's check if the metadata was uploaded:

  1. In the navigation, click Files .
  2. Click any file that has metadata and choose View Metadata. You should see the file's metadata.