Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Upload Files

Upload your data to Haystack Enterprise Platform. The files you upload are then turned into documents and indexed to make them searchable for your pipelines.


Things to note:
  • Currently, Haystack Enterprise Platformsupports files encoded in UTF-8 format. For files with different encoding formats, you may experience issues with display or formatting.
  • A single file cannot be larger than 200 MB. If a file exceeds this size, it won't be uploaded. This applies to uploads from the UI and to the Upload File REST API endpoint, which returns a 413 error. To upload larger files, use the Haystack Platform SDK or upload sessions (see Upload Sessions).
  • Whenever you upload a file, it's indexed by all indexes enabled in this workspace. This means all deployed pipelines then use this file to resolve queries.

About This Task​

Supported File Types​

You can upload any file types, there are no restrictions. To preprocess most file types, you can add Converters to your index. For the file types for which a converter is unavailable, we recommend preprocessing the files outside of Haystack Enterprise Platform. You can use one of Haystack Platform's integrations to do that. For details, see Using Hosted Models and External Services.

To learn more, see also PreProcessing Data with Pipeline Components.

Once you upload your files, you can preview them in Haystack Platform. Preview is supported for TXT, PDF, JSON, HTML, MD, XML, CSV, DOCX, DOC, XLSX, XLS, PNG, JPG, JPEG, GIF, SVG, and WEBP formats.

Metadata​

You can upload files together with metadata using the SDK, the REST API, or directly from the Haystack Enterprise Platform interface. When uploading from the Haystack Enterprise Platform interface, you can either attach a JSON metadata object to all the files in a batch directly in the upload dialog, or add metadata to your files one by one after they're already uploaded. You can also add a <file_name>.meta.json file next to a file to give that file its own metadata when you upload it.

If you're uploading files with metadata, the resulting documents inherit the metadata from your files.

To download a file's metadata as a JSON file, go to Files, find the file whose metadata you want to download, click More Actions, and then click Download Metadata. Haystack Platform saves the metadata as <file_name>.metadata.json.

Preparing file metadata

To add metadata to your files at upload time, create one metadata file for each file you upload. The metadata file must be a JSON file with the same name as the file whose metadata it contains and the extension meta.json.

For example, if you're uploading a file called example.txt, the metadata file should be called example.txt.meta.json. If you're uploading a file called example.pdf, the metadata file should be example.pdf.meta.json.

Here's the format of metadata in your *.meta.json files: {"meta_key1": "value1", "meta_key2": "value2"}.

To learn more about adding metadata, see Add Metadata to Your Files.

To learn about how to use metadata in your search app, see Working with Metadata.

Upload Sessions​

The SDK and the Create Upload Session REST API endpoint use upload sessions. To start uploading files, you must first create a session. Sessions help you track the files you're uploading. Each session has an ID and you can check the status of your uploaded files at any time. Files start uploading only after a session is closed. If you don't close a session explicitly, it's automatically closed after 24 hours.

We recommend using upload sessions for large file uploads as they are faster and more efficient. There's no limit on the number of files per session, but we recommend keeping it to a maximum of 50,000 files. By default, a single file in a session can be up to 1 GB. If a file is larger, Haystack Platform marks it as failed with the reason FILE_TOO_LARGE and doesn't index it. The other files in the session are still processed. To index a file over 1 GB, split it into smaller files first.

After you upload files to Haystack Enterprise Platform, it may take some time before the files become visible.

This diagram explains how the upload sessions work:

First a session is opened through a request. Then, files are sent to the link from the response, the session is closed and files get uploaded to deepset.

File Updates​

To learn how enabled indexes handle updated files, see Indexes.

Choosing the Best Upload Method​

  • If you prefer to upload from Haystack Enterprise Platform and you have a limited number of files, upload from the UI.
  • If you have a lot of data and metadata to upload and are familiar with coding, use the SDK.
  • If the SDK upload doesn't work for you, use the REST API.

Prerequisites​

  • Make sure you have a workspace to upload your files to. You can easily create one if needed:

      1. In Haystack Enterprise Platform, click the workspace name in the upper left corner.

      2. Type the name of the workspace you want to create. You can create up to 100 workspaces.

        The workspace list expanded. At the bottom of the list there's an empty field where you can type the name of the new workspace and the Create button.

Upload from the UI​

  1. In Haystack Enterprise Platform, go to Files>Upload Files.
  2. Drag your files and folders to Haystack Platform. To give a file its own metadata, add its <file_name>.meta.json file to the selection too, for example report.pdf and report.pdf.meta.json. Haystack Platform uses the metadata file's content as the metadata of report.pdf and doesn't upload the metadata file itself.
  3. Optional: Expand Add Metadata (optional) and type a JSON object with the metadata you want to add to all the files in this batch, for example {"key": "value"}. The JSON must be valid. If a key appears both here and in a file's .meta.json, the value from the .meta.json file is used for that file. All other keys from both sources are kept.
  4. Click Upload. The files may take a while to be processed. When the upload finishes, the files appear on the Files page. If they don't appear after a while, check that each file is under 200 MB, then upload them again. To upload larger files, use the SDK.

If a .meta.json file has no matching file in the selection, Haystack Platform uploads it as a regular JSON file. If a .meta.json file has a matching file but doesn't contain a JSON object (for example, a list or a string), Haystack Platform uploads nothing from that selection and shows the metadata file's name. Correct the file and add it again, or upload without it.

Upload with the SDK​

This method is best if you have many files to upload. It makes it possible to add metadata to your files. You can use the command-line interface or Python methods. For details, see Upload Files in the Haystack Enterprise SDK docs.

See also our tutorials:

Upload with the REST API​

To upload files from your own code, use upload sessions or the Upload File endpoint. For step-by-step instructions, see Upload Files with the REST API.