Add Search Filters

Add filters to narrow down the scope of your search. You do this by attaching metadata to the files you upload to deepset Cloud. These metadata then turn into filters at query time.

Metadata as Filters

For example, if you have a set of financial reports covering a decade, you may want to search only through the reports for a given year, so you'll want to select a year in your search. To do this, add metadata to your files, for example: {"year":"2009"}.

Here's what it will look like on the Search page:

A screenshot showing the Search page with filters displayed on the left. The Year filter is expanded and shows three years: 2011, 2010, and 2009.A screenshot showing the Search page with filters displayed on the left. The Year filter is expanded and shows three years: 2011, 2010, and 2009.

You can add metadata to files when uploading them with an API endpoint or using the Python SDK. You simply pass them in a dictionary this way: meta={"key1":"value1", "key2":"value2"}. Each dictionary defines metadata for one file you're uploading, so if you're uploading ten files, you should pass ten metadata dictionaries. The first dictionary attaches metadata to the first file you're passing, the second one to the second file, and so on.

deepset Cloud supports numerical data, dates, and keyword fields as metadata.

To add metadata to files that already exist in a deepset Cloud workspace, use the update file meta endpoint.

Add Filters with the Python SDK

This is an example of ,,Python code you can use to add metadata to files when uploading them:

# The first five lines are all the necessary imports to make it work
import os
from haystack.utils import DeepsetCloud
from pathlib import Path

# Set the API key and API endpoint:
os.environ["DEEPSET_CLOUD_API_KEY"] = "<YOUR_API_KEY>"
os.environ["DEEPSET_CLOUD_API_ENDPOINT"] = "https://api.cloud.deepset.ai/api/v1"

file_client = DeepsetCloud.get_file_client(api_key=os.environ["DEEPSET_CLOUD_API_KEY"], 
                                           workspace="<WORKSPACE_NAME>")
# Specify the paths to your files here:
file_paths = [
  Path("C:\Users\OneDrive\Documents\file1.txt"),
  Path("C:\Users\OneDrive\Documents\file2.pdf")
]

# Here you're defining the metadata. Add one dictionary with key:value pairs per file.
# In this example, we need two dictionaries because we're uploading two files.
# 
metas = [{"year": "2009", "type":"financial report"}, {"year": "2010"}]

# Here you're uploading the files to deepset Cloud together with their metadata
my_files = file_client.upload_files(file_paths=file_paths, metas=metas)

In this example, the first file, file1.txt gets the following metadata: "year":"2009" and "type":"financial report" and the second file, file2.txt gets "year":"2010". The Search page will show two filters: year and type.

Add Filters with REST API

Here's the code you can use to add metadata to files when uploading them through REST API:

curl --request POST \
     --url 'https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/files' \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <YOUR_API_KEY>' \
     --header 'content-type: multipart/form-data' \
     --form [email protected]<YOUR_FILE.PDF>
        // Here is where you define the metadata for your file:
     --form 'meta={"year":"2009", "type":"financial report"}'

You can also add or update metadata of files that are already uploaded to deepset Cloud, using the update file meta endpoint. You need the ID of the file whose metadata you want to change. (You can check it with the list files endpoint.)

curl --request PUT \
     --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/files/<FILE_ID>/meta \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <YOUR_API_KEY>' \
     --header 'content-type: application/json'
     // Here is where you pass the metadata:
     --data '
{
     "year": "2009",
     "type": "financial report"
}
'

Related Links