Choose Files to Index
Choose which files your index processes. You can filter files by name prefix
About This Task
By default, an index processes all files in your workspace. You can restrict indexing to a subset of those files:
- By file name prefix: Index only files whose names start with one of the prefixes you set. For example,
legal*matcheslegal-contract.pdfandlegal_memo.docx.
- By metadata: Index only files whose metadata matches a condition you set. For example, index files where
category == reports.
Only files that match your filters are indexed, counted, and listed on the Index Files tab. All other files are skipped.
You can use both filters at the same time. When both are set, a file must match the prefix filter and the metadata filter to be indexed.
You can update these filters only while the index is disabled. Disable the index first, then change the filters. Changing the filter does not re-index files that were already indexed.
Changing a file's metadata after it was already indexed does not automatically re-index that file.
Prerequisites
- A disabled index. If the index is enabled, disable it before you change filters.
- File metadata, if you want to filter by metadata. For details, see Add Metadata to Your Files.
Set a File Name Prefix Filter
- In Haystack Enterprise Platform, go to Indexes and click the index you want to configure. This opens the Index in Builder.
- Open the Settings tab.
- In the File Name Prefix Filter section, type one or more prefixes in the Prefixes field.
- Separate multiple prefixes with commas. A file is indexed if its name matches any of the prefixes.
- You can end a prefix with
*as a wildcard, for exampledoc*orcontracts/. - Each prefix can include letters, digits,
.,_,-, and/.
Clear the File Name Prefix Filter
- Open the Settings tab for the disabled index.
- Clear the Prefixes field in the File Name Prefix Filter section.
Set a File Name Prefix Filter with the API
Use the Update Index endpoint to set or clear the prefix filter.
Pass a comma-separated list of prefixes in file_name_prefix_filter. A file is indexed if its name starts with any of the prefixes. The trailing * is optional: docs/ and docs/* both match names that start with docs/.
To remove the prefix filter, set file_name_prefix_filter to null.
Example: Index files whose names start with legal or contracts/
PATCH /api/v1/workspaces/{workspace_name}/indexes/{index_name}
{
"file_name_prefix_filter": "legal*,contracts/"
}
Set a Metadata Filter in the UI
The File Meta Filter in the UI uses one equality condition: you specify a metadata field name and a value, and only files where that field equals that value are indexed. Files without that field, or with a different value, are skipped.
- In Haystack Enterprise Platform, go to Indexes and click the index you want to configure. This opens the Index in Builder.
- Open the Settings tab.
- In the File Meta Filter section, type the metadata field name in the first field. For example,
category. - The operator is fixed to
==(equals). - Type the value to match in the second field. For example,
reports. - Click Save.
The index now processes only files whose metadata field matches the value you entered.
Clear the File Meta Filter
To remove the filter and index all files again:
- Open the Settings tab for the disabled index.
- In the File Meta Filter section, clear both the field name and the value fields.
The filter applies to file metadata, not to file content. Add the relevant metadata to your files before you enable the index. For details, see Add Metadata to Your Files.
In the UI you can set one metadata rule at a time, and the operator is always ==. To use != or combine several conditions, set the filter with the API.
Set a Metadata Filter with the API
Use the Update Index endpoint to set or clear the metadata filter.
Each condition in file_meta_filter must include:
field— The metadata field name to filter on.operator— The comparison operator:==(equal) or!=(not equal).value— The value to compare against. Supported types: string, integer, number, boolean, or null.
All conditions are combined with AND logic: a file must satisfy every condition to be indexed.
To remove a metadata filter, set file_meta_filter to null.
Example: Index only files in the "legal" category that are not in German
PATCH /api/v1/workspaces/{workspace_name}/indexes/{index_name}
{
"file_meta_filter": [
{"field": "category", "operator": "==", "value": "legal"},
{"field": "language", "operator": "!=", "value": "de"}
]
}
You can set file_name_prefix_filter and file_meta_filter in the same request.
What To Do Next
After you update the filters, enable the index so it applies the new conditions to incoming files.
Was this page helpful?