Configure Index Settings
Control how an index runs and which files it processes. Turn on GPU acceleration, pin a Haystack version, filter files by name prefix
About This Task
Most index settings can be changed only while the index is disabled, because an enabled index is read-only. Disable the index first, change the setting, and enable the index again. For details, see Edit an Index.
The indexing pod size is the exception. You can change it while the index is enabled.
Enable GPU Acceleration
By default, indexes run on CPU. You can enable GPU acceleration for an index to speed it up. This is especially useful for pipelines that include components that run faster on a GPU, like DoclingConverter or SentenceTransformersDocumentEmbedder.
When you enable GPU acceleration, the index checks whether a component needs a GPU and assigns one automatically. GPUs are only used when required and are not reserved for the entire index processing.
If GPU acceleration is disabled and your index includes components that rely on a GPU, those components run on the CPU instead. This can slow down processing and may cause timeouts, especially for larger or more complex indexes.
To enable GPU acceleration:
- Go to Indexes and click the index you want to enable GPU acceleration for. This opens the Builder.
- Open the Settings tab and click the GPU Acceleration toggle to turn it on.
If your index needs more CPU or memory rather than a GPU, for example to convert large files, set a larger indexing pod size. For details, see Set the Indexing Instance Size in Index Settings.
Pin a Haystack Version
By default, an index runs on the Haystack version that Haystack Platform provides. To use a specific haystack-ai version or extra packages, add a top-level dependencies block to the index YAML. The index then processes your files in an isolated environment built from that block. The block works the same way in index YAML you create or update through the API. For details, see Pinning a Haystack Version for an Index.
The index must be disabled.
-
Log in to Haystack Enterprise Platform and open Indexes.
-
Click the name of the index you want to update and switch to Build.
-
In the index YAML, add a top-level
dependenciesblock. Pin thehaystack-aiversion with==and list any extra packages:dependencies:- haystack-ai==2.18.0- <extra-package> -
Save the index.
-
Enable the index. For details, see Enable an Index. When indexing starts, we validate the
dependenciesblock. -
Open the Index Details page and check the logs. If the block is invalid or empty, the indexing job fails right away with an error message that describes the problem, before any file is processed. Fix the block and enable the index again. If the block is valid, indexing continues and processes your files.
On the Index Details page, check the processed files to confirm your files were indexed.
To remove the pin, delete the dependencies block from a disabled index and enable it again. The index then runs on the version Haystack Platform provides.
Choose Files to Index
By default, an index processes all files in your workspace. You can restrict indexing to a subset of those files:
- By file name prefix: Index only files whose names start with one of the prefixes you set. For example,
legal*matcheslegal-contract.pdfandlegal_memo.docx.
- By metadata: Index only files whose metadata matches a condition you set. For example, index files where
category == reports.
Only files that match your filters are indexed, counted, and listed on the Index Files tab. All other files are skipped.
You can use both filters at the same time. When both are set, a file must match the prefix filter and the metadata filter to be indexed.
You can update these filters only while the index is disabled. Disable the index first, then change the filters. Changing the filter does not re-index files that were already indexed.
Changing a file's metadata after it was already indexed does not automatically re-index that file.
To filter by metadata, your files need metadata first. For details, see Add Metadata to Your Files.
Set a File Name Prefix Filter
- In Haystack Enterprise Platform, go to Indexes and click the index you want to configure. This opens the Index in Builder.
- Open the Settings tab.
- In the File Name Prefix Filter section, type one or more prefixes in the Prefixes field.
- Separate multiple prefixes with commas. A file is indexed if its name matches any of the prefixes.
- You can end a prefix with
*as a wildcard, for exampledoc*orcontracts/. - Each prefix can include letters, digits,
.,_,-, and/.
Clear the File Name Prefix Filter
- Open the Settings tab for the disabled index.
- Clear the Prefixes field in the File Name Prefix Filter section.
Set a File Name Prefix Filter with the API
To set or clear the prefix filter with the REST API, see Filter Files to Index with the REST API.
Set a Metadata Filter in the UI
The File Meta Filter in the UI uses one equality condition: you specify a metadata field name and a value, and only files where that field equals that value are indexed. Files without that field, or with a different value, are skipped.
- In Haystack Enterprise Platform, go to Indexes and click the index you want to configure. This opens the Index in Builder.
- Open the Settings tab.
- In the File Meta Filter section, type the metadata field name in the first field. For example,
category. - The operator is fixed to
==(equals). - Type the value to match in the second field. For example,
reports. - Click Save.
The index now processes only files whose metadata field matches the value you entered.
Clear the File Meta Filter
To remove the filter and index all files again:
- Open the Settings tab for the disabled index.
- In the File Meta Filter section, clear both the field name and the value fields.
The filter applies to file metadata, not to file content. Add the relevant metadata to your files before you enable the index. For details, see Add Metadata to Your Files.
In the UI you can set one metadata rule at a time, and the operator is always ==. To use != or combine several conditions, set the filter with the REST API.
Set a Metadata Filter with the API
To set or clear the metadata filter with the REST API, see Filter Files to Index with the REST API.
Set the Indexing Instance Size in Index Settings
If an index processes big files and indexing is slow or fails, choose a larger instance size in the index settings.
The Instance size list shows only the sizes your organization has enabled. An organization admin enables sizes in Settings > Organization > Compute. For steps, see Managing Compute Settings.
If an admin turned off the size your index already uses, that size still appears in the list, so the index keeps showing its current size. You can keep it or switch to an enabled size. While an index has an instance size set, deployed or not, an organization admin can't turn that size off. To free the size, switch the index to another size, or reset it to the default pod size.
Each size sets the memory limit of the indexing pod: small has 4 GiB, medium has 8 GiB, large has 16 GiB, and xl has 32 GiB. An index without a size uses the default of 8 GiB. To check whether the pod is close to running out of memory, compare its usage with this limit in the index metrics. If it's close, set a larger size. For details, see Check Index Hardware Metrics.
To change the instance size:
- Log in to Haystack Enterprise Platform and open Indexes.
- Click the index name and open the index settings.
- In Instance size, select the size you want. You need permission to edit indexes to change it. While the list is loading, it shows only the index's current size.
A new size doesn't apply right away. The index keeps indexing with the old size until the next time its pod is deployed, and then it uses the new size.
What To Do Next
After you change the settings, enable the index so it applies them. Wait until indexing finishes before you run queries.
Related Information
Was this page helpful?