About This Task

There are two ways to add metadata to your files:

• During upload: You upload metadata along with the actual files. You can use an API endpoint or the Python SDK to do this. When uploading a file, include a corresponding metadata file with the meta.json extension. For example, if you have a file called myfile.pdf, its metadata would be stored in a separate file called myfile.pdf.meta.json. This approach ensures the metadata is associated with the correct file.
• Update files already uploaded. You can do it in two ways:
  • Use the Update File Meta endpoint.
  • Update the file metadata directly on the Files page in deepset Cloud. You can add, edit, and delete metadata for each file.

Metadata Format

Metadata is always a dictionary in this format: {"meta_key1": "value", "meta_key2": "value2"}. The value for a specific key in a key:value pair must be of the same type across all files. For example, if one file has a key called "category" and its value is of type string {"category":"news"}, then the values of "category" in all other metadata files must also be strings. An upload will fail if another metadata file lists "category" with a value that's not a string but an integer or some other type.

📢

Important

Do not use "id" or "_id" as metadata keys or values, as they're reseved for internal processing of deepset Cloud. If you add them, they'll be overwritten or rejected.

Formatting Guidelines for Uploading with SDK

When uploading with SDK, include one metadata file for each file you're uploading. The name of the metadata file should match the original file's name, but with a meta.jsonextension. Format the metadata inside the metadata file like this: {"meta_key1": "value", "meta_key2": "value2"}.

Positives:
location and how clean it is

Negatives:
lack of variety in food any thing you ask for will charge you 5 euro the WiFi is not covering all my room dogs are allowed

{"Hotel_Address": "Schlossallee 8 14 Penzing 1140 Vienna Austria", "Review_Date": "2016-02-22", "Average_Score": 8.8, "Hotel_Name": "Austria Trend Hotel Park Royal Palace Vienna", "Reviewer_Score": 9.6}

Formatting Guidelines for Uploading with REST API

If you're uploading files through the API, pass metadata for each file in a separate dictionary formatted like this: meta={"key1":"value1", "key2":"value2"}. For each file you're uploading, provide a corresponding metadata dictionary. The order matters: the first dictionary you pass is linked to the first file, the second dictionary to the second file, and so on.

Allowed Values

In your metadata dictionaries, you can pass the following:

• Numerical data
• Dates
• Keyword fields
• Lists

Metadata limitations:

• One level of nesting works best.
• The size limit for a single value in the key:value pair is 32 766 bytes, which is roughly 32 766 characters.
• The total size of metadata (all keys and values) is 250 000 bytes maximum, roughly 250 000 characters.

Example

Here are simple examples of metadata:

metadata = [{"type": "article", "source": "New York Times", "title": "Leaders Look into the Future", "year published": "2020"}]

{
	"type": "book",
  "topic": "business",
  "author": "Ben Horowitz",
  "book_titles": [
  	"The Hard Thing About Hard Things",
    "What You Do Is Who You Are"
    ]
 } 
    

Add Metadata with the Python SDK

The easiest way is to upload metadata together with your files. For detailed instructions, see Tutorial: Uploading Files with CLI or Tutorial: Uploading Files with Python Methods.

Add Metadata with REST API

Here's the code you can use to add metadata to files when uploading them through REST API:

curl --request POST \
     --url 'https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/files?write_mode=OVERWRITE' \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <YOUR_API_KEY>' \
     --header 'content-type: multipart/form-data' \
     --form file=@<YOUR_FILE.PDF>
		// Here is where you define the metadata for your file:
     --form 'meta={"year":"2009", "type":"financial report"}'

For a step-by-step walkthrough of the code, see:

You can also add or update metadata of files that are already uploaded to deepset Cloud, using the Update File Meta endpoint. You need the ID of the file whose metadata you want to change. (You can check it with the list files endpoint.)

curl --request PUT \
     --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/files/<FILE_ID>/meta \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <YOUR_API_KEY>' \
     --header 'content-type: application/json'
     // Here is where you pass the metadata:
     --data '
{
     "year": "2009",
     "type": "financial report"
}
'

Add Metadata from the UI

This is an easy way to add, update, and remove metadata from files.

1. Log in to deepset Cloud and go to Files.

2. Find the file whose metadata you want to update, click More actions next to it, and choose View Metadata.

   

3. Update the metadata as needed. You must choose a type for every field and then add its key and value. When you're done, save your changes.

4. Continue for each file whose metadata you want to update.