Use Metadata in Your Search System

Attach metadata to the files you upload to deepset Cloud and take advantage of them in your search system. Learn about different ways you can use metadata.

Applications of Metadata

Metadata in deepset Cloud can serve as filters that narrow down the selection of documents for generating the final answer or to customize retrieval and ranking.

Let's say we have a document with the following metadata:

{ "title": "Mathematicians prove Pólya's conjecture for the eigenvalues of a disk", "subtitle": "A 70-year old math problem proven", "authors": ["Alice Wonderland"], "published_date": "2024-03-01", "category": "news" }

We'll refer to these metadata throughout this guide to illustrate different applications.

To learn how to add metadata, see Add Metadata to Your Files.

Metadata as Filters

Metadata acts as filters that narrow down the scope of your search. All metadata from your files are shown as filters on the Playground:

But you can also use metadata to add a preset filter to your pipeline or when searching through REST API.

Filtering with a Preset Filter

You can configure your pipeline to use only documents with specific metadata values. Depending on whether you use BM25Retriever or CNStaticFilterEmbeddingRetriever, this involves custom OpenSearch queries or direct filter parameters.

With BM25Retriever, use the custom_query parameter to pass your OpenSearch query. For example, to retrieve only documents of the news category ("category": "news" in metadata), you could pass this query:

- name: BM25Retriever type: BM25Retriever params: document_store: DocumentStore top_k: 10 custom_query: > { "query": { "bool": { "must": [ { "multi_match": { "query": $query, "type": "most_fields", "fields": ["content", "title", "authors"], "operator": "OR" } } ], "filter": { "term": { "category": "news" } } } } } ...

With the CNStaticFilterEmbeddingRetriever, specify your filters directly in the filters parameter, like this:

components: - name: EmbeddingRetriever type: CNStaticFilterEmbeddingRetriever params: document_store: DocumentStore embedding_model: ... model_format: sentence_transformers top_k: 20 filters: {"category": "news"}

You can also assign weight to certain metadata values to prioritize them. To learn more about using OpenSearch queries in your pipelines, see Boosting Retrieval with OpenSearch Queries.

Filtering at Query Time with REST API

When making a search request through API, add the filter to the payload when making the request like this:

curl --request POST \ --url https://api.cloud.deepset.ai/api/v1/workspaces/workspace_name/pipelines/pipeline_name/search \ --header 'accept: application/json' \ --header 'content-type: application/json' \ --data ' { "debug": false, "filters": { "category": "news" } } '

See also Filtering Logic.

Metadata to Customize Retrieval

By default, retrievers search only through the content of the documents in the Document Store, but with some retrievers, you can indicate the metadata fields that the retriever should check in addition to the contents.

Metadata with BM25Retriever

You can indicate the metadata fields you want the keyword-based BM25Retriever to search. Pass the names of these fields in the search_fields parameter of the Document Store used with the retriever. For example, to get an answer to a question like: "What was the latest article Alice wrote?", you may want the retriever to search through the title and the author of the documents, not only their content. Here's how you could configure BM25Retriever to do this:

components: - name: DocumentStore type: DeepsetCloudDocumentStore params: search_fields: ["title", "authors"] - name: Retriever type: BM25Retriever params: document_store: DocumentStore

Metadata with EmbeddingRetriever

EmbeddingRetriever can vectorize not only the document text but also the metadata you indicate in the embed_meta_fields parameter. In this example, EmbeddingRetriever embeds the title and subtitle of documents:

components: - name: EmbeddingRetriever type: EmbeddingRetriever params: document_store: DocumentStore embedding_model: ... model_format: sentence_transformers top_k: 20 embed_meta_fields: ["title", "subtitle"]

This means that the title and subtitle would be prepended to the document content.

Metadata for Ranking

When using SentenceTransformersRanker, CohereRanker, and RecentnessRanker, you can assign a higher rank to documents with certain metadata values. To learn more about rankers, see Ranker.

Both SentenceTransformersRanker and CohereRanker take the embed_meta_fields parameter, where you can pass the metadata fields you want to prioritize. Like with EmbeddingRetriever, the values of these fields are then prepended to the document content and embedded there. This means that the content of the metadata fields you indicate are taken into consideration during the ranking. In this example, the Ranker prioritizes documents with fields category and authors:

components: - name: Reranker type: SentenceTransformersRanker params: model_name_or_path: svalabs/cross-electra-ms-marco-german-uncased top_k: 15 embed_meta_fields: [category, authors] ...

RecentnessRanker is specifically designed to prioritize the latest documents. All you need to do is pass the name of the metadata field containing the date in the date_meta_field parameter, like this:

components: - name: RecentnessRanker type: RecentnessRanker params: date_meta_field: published_date ...

To learn more, see Improving Your Document Search Pipeline.

Metadata in Prompts

You can pass documents' metadata in prompts for additional context and then instruct the LLM to use them. To pass the metadata, use the $ (dollar sign) as a prefix and then pass the name of the metadata field in a function. This prompt, apart from the documents' contents, contains their titles and (the variables are replaced with real values at runtime). Additionally, it instructs the LLM to prefer the most recent documents:

You are a media expert. You answer questions truthfully based on provided documents. For each document check whether it is related to the question. Only use documents that are related to the question to answer it. Ignore documents that are not related to the question. If the answer exists in several documents, summarize them. Only answer based on the documents provided. Don't make things up. Always use references in the form [NUMBER OF DOCUMENT] when using information from a document. e.g. [3], for Document[3]. The reference must only refer to the number that comes in square brackets after passage. Otherwise, do not use brackets in your answer and reference ONLY the number of the passage without mentioning the word passage. If the documents can't answer the question or you are unsure say: 'The answer can't be found in the text'. For contradictory information, prefer recent documents. These are the documents: {join(documents, delimiter=new_line, pattern=new_line+'Document[$idx]:'+new_line+'Title: $title'+new_line+'Subtitle: $subtitle'+new_line+'Published Date: $published_date'+new_line+'$content')} Question: {query} Answer:

For more information, see Functions in prompts.

Example

Here's an example pipeline that uses metadata during the retrieval, ranking, and answer generation stages. During retrieval, it focuses on documents from the "news" category and embeds the title and subtitle into the document's text. Then, when ranking the documents, it takes the document's title and subtitle into consideration. And finally, when generating the answer, it passes the document's published date in the prompt, instructing the LLM to prefer the most recent documents.

components: - name: DocumentStore type: DeepsetCloudDocumentStore params: embedding_dim: 768 similarity: cosine search_fields: ["title", "authors"] # We also enable (sparse) search over these fields - name: BM25Retriever # The keyword-based retriever type: BM25Retriever params: document_store: DocumentStore top_k: 10 # The number of results to return # Custom Query to only return documents from the `news` category custom_query: > { "query": { "bool": { "must": [ { "multi_match": { "query": $query, "type": "most_fields", "fields": ["content", "title", "authors"], "operator": "OR" } } ], "filter": { "term": { "category": "news" } } } } } - name: EmbeddingRetriever type: CNStaticFilterEmbeddingRetriever params: document_store: DocumentStore embedding_model: intfloat/e5-base-v2 # Model optimized for semantic search. It has been trained on 215M (question, answer) pairs from diverse sources. model_format: sentence_transformers top_k: 10 embed_meta_fields: ["title", "subtitle"] # We also add these fields before embedding filters: {"category": "news"} # Also only return documents from the `news` category - name: JoinResults # Joins the results from both retrievers type: JoinDocuments params: join_mode: concatenate # Combines documents from multiple retrievers - name: Reranker # Uses a cross-encoder model to rerank the documents returned by the two retrievers type: SentenceTransformersRanker params: model_name_or_path: intfloat/simlm-msmarco-reranker # Fast model optimized for reranking top_k: 5 # The number of results to return batch_size: 20 # Try to keep this number equal or larger to the sum of the top_k of the two retrievers so all docs are processed at once embed_meta_fields: ["title", "subtitle"] # We also add these fields before embedding model_kwargs: # Additional keyword arguments for the model torch_dtype: torch.float16 - name: RecentnessRanker type: RecentnessRanker params: date_meta_field: published_date # Rerank based on recency as determined by this metadata field - name: qa_template type: PromptTemplate params: output_parser: type: AnswerParser # We also refer to the metadata fields $title, $subtitle, $published_date in the prompt. prompt: > You are a media expert. {new_line}You answer questions truthfully based on provided documents. {new_line}For each document check whether it is related to the question. {new_line}Only use documents that are related to the question to answer it. {new_line}Ignore documents that are not related to the question. {new_line}If the answer exists in several documents, summarize them. {new_line}Only answer based on the documents provided. Don't make things up. {new_line}Always use references in the form [NUMBER OF DOCUMENT] when using information from a document. e.g. [3], for Document[3]. {new_line}The reference must only refer to the number that comes in square brackets after passage. {new_line}Otherwise, do not use brackets in your answer and reference ONLY the number of the passage without mentioning the word passage. {new_line}If the documents can't answer the question or you are unsure say: 'The answer can't be found in the text'. {new_line}For contradictory information, prefer recent documents. {new_line}These are the documents: {join(documents, delimiter=new_line, pattern=new_line+'Document[$idx]:'+new_line+'Title: $title'+new_line+'Subtitle: $subtitle'+new_line+'Published Date: $published_date'+new_line+'$content')} {new_line}Question: {query} {new_line}Answer: - name: PromptNode type: PromptNode params: default_prompt_template: qa_template max_length: 400 # The maximum number of tokens the generated answer can have model_kwargs: # Specifies additional model settings temperature: 0 # Lower temperature works best for fact-based qa model_name_or_path: gpt-3.5-turbo - name: FileTypeClassifier # Routes files based on their extension to appropriate converters, by default txt, pdf, md, docx, html type: FileTypeClassifier - name: TextConverter # Converts files into documents type: TextConverter - name: PDFConverter # Converts PDFs into documents type: PDFToTextConverter - name: Preprocessor # Splits documents into smaller ones and cleans them up type: PreProcessor params: # With a vector-based retriever, it's good to split your documents into smaller ones split_by: word # The unit by which you want to split the documents split_length: 250 # The max number of words in a document split_overlap: 20 # Enables the sliding window approach language: en split_respect_sentence_boundary: True # Retains complete sentences in split documents # Here you define how the nodes are organized in the pipelines # For each node, specify its input pipelines: - name: query nodes: - name: BM25Retriever inputs: [Query] - name: EmbeddingRetriever inputs: [Query] - name: JoinResults inputs: [BM25Retriever, EmbeddingRetriever] - name: Reranker inputs: [JoinResults] - name: RecentnessRanker inputs: [Reranker] - name: PromptNode inputs: [RecentnessRanker] - name: indexing nodes: # Depending on the file type, we use a Text or PDF converter - name: FileTypeClassifier inputs: [File] - name: TextConverter inputs: [FileTypeClassifier.output_1] # Ensures that this converter receives txt files - name: PDFConverter inputs: [FileTypeClassifier.output_2] # Ensures that this converter receives PDFs - name: Preprocessor inputs: [TextConverter, PDFConverter] - name: EmbeddingRetriever inputs: [Preprocessor] - name: DocumentStore inputs: [EmbeddingRetriever]

Did this page help you?