Metadata as Filters

For example, if you have a set of financial reports covering a decade, you may want to search only through the reports for a given year, so you'll want to select a year in your search. To do this, add metadata to your files, for example: {"year":"2009"}.

Here's what it will look like on the Search page:

You can add metadata to files when uploading them with an API endpoint or using the Python SDK.

To add metadata to files that already exist in a deepset Cloud workspace, use the update file meta endpoint.

To learn more about using metadata as filters, see Filtering Logic.

Metadata Format

You pass metadata in a dictionary this way: meta={"key1":"value1", "key2":"value2"}.

Pass one dictionary for one file you're uploading, so if you're uploading ten files, you should pass ten metadata dictionaries. The first dictionary attaches metadata to the first file you're passing, the second one to the second file, and so on.

In your metadata dictionaries, you can pass:

Numerical data
Dates
Keyword fields

Metadata limitations:

One level of nesting works best.
The size limit for a single value in the key:value pair is 32 766 bytes, which is roughly 32 766 characters.
The total size of metadata (all keys and values) is 250 000 bytes maximum, roughly 250 000 characters.

Example

Here are simple examples of metadata:

metadata = [{"type": "article", "source": "New York Times", "title": "Leaders Look into the Future", "year published": "2020"}]

{
    "type": "book",
  "topic": "business",
  "author": "Ben Horowitz",
  "book_titles": [
    "The Hard Thing About Hard Things",
    "What You Do Is Who You Are"
    ]
 }

Add Filters with the Python SDK

Below is an example of Python code you can use to add metadata to files when uploading them:

# The first five lines are all the necessary imports to make it work
import os
from haystack.utils import DeepsetCloud
from pathlib import Path

# Set the API key and API endpoint:
os.environ["DEEPSET_CLOUD_API_KEY"] = "<YOUR_API_KEY>"
os.environ["DEEPSET_CLOUD_API_ENDPOINT"] = "https://api.cloud.deepset.ai/api/v1"

file_client = DeepsetCloud.get_file_client(api_key=os.environ["DEEPSET_CLOUD_API_KEY"], 
                                           workspace="<WORKSPACE_NAME>")
# Specify the paths to your files here:
file_paths = [
  Path("C:\Users\OneDrive\Documents\file1.txt"),
  Path("C:\Users\OneDrive\Documents\file2.pdf")
]

# Here you're defining the metadata. Add one dictionary with key:value pairs per file.
# In this example, we need two dictionaries because we're uploading two files.
# 
metas = [{"year": "2009", "type":"financial report"}, {"year": "2010"}]

# Here you're uploading the files to deepset Cloud together with their metadata
my_files = file_client.upload_files(file_paths=file_paths, metas=metas)

In this example, the first file, file1.txt gets the following metadata: "year":"2009" and "type":"financial report" and the second file, file2.txt gets "year":"2010". The Search page will show two filters: year and type.

Add Filters with REST API

Here's the code you can use to add metadata to files when uploading them through REST API:

curl --request POST \
     --url 'https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/files' \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <YOUR_API_KEY>' \
     --header 'content-type: multipart/form-data' \
     --form [email protected]<YOUR_FILE.PDF>
        // Here is where you define the metadata for your file:
     --form 'meta={"year":"2009", "type":"financial report"}'

You can also add or update metadata of files that are already uploaded to deepset Cloud, using the update file meta endpoint. You need the ID of the file whose metadata you want to change. (You can check it with the list files endpoint.)

curl --request PUT \
     --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/files/<FILE_ID>/meta \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <YOUR_API_KEY>' \
     --header 'content-type: application/json'
     // Here is where you pass the metadata:
     --data '
{
     "year": "2009",
     "type": "financial report"
}
'