Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Upload Files

Upload files and their metadata into a Haystack Enterprise Platform workspace. You can use the CLI or Python SDK to do it. This is the fastest route when you have many files.

About This Task​

You can upload files and folders to a Haystack Enterprise Platform workspace.

info

When using the Haystack Enterprise SDK on Windows, if haystack-enterprise isn't on your PATH, replace it with python -m haystack_enterprise_sdk.cli, for example:

haystack-enterprise validate pipeline.py

becomes

python -m haystack_enterprise_sdk.cli validate pipeline.py

Sessions​

Uploads happen through a session. The SDK opens a session, sends your files, and then closes the session. This is when the files are ingested into the workspace. A session you leave open expires after 24 hours.

A session stores the ingestion status of the files: the number of failed and successful uploads. Each session has an ID so you can check its details any time.

Folder Structure​

You don't need to follow any specific folder structure.

Duplicate File Names​

If a file you upload has the same name as a file already in the workspace, the SDK keeps both by default. To change that, use the --write-mode flag with one of the following values:

  • KEEP (default)
  • OVERWRITE
  • FAIL

For example, to overwrite the file if it already exists in the workspace, use:

haystack-enterprise upload ./my-files --write-mode OVERWRITE

Supported File Types​

Haystack Enterprise Platform supports these file types: .csv, .docx, .html, .json, .md, .pdf, .pptx, .txt, .xlsx, and .xml.

By default, upload sends every supported type in the folder. It skips unsupported types instead of failing the upload. To limit the upload to specific types, name each one with a separate --use-type flag:

haystack-enterprise upload ./my-files --use-type .md --use-type .pdf --use-type .docx

Listing a type replaces the default rather than adding to it. The example above uploads .md, .pdf, and .docx files, and skips .txt files.

Metadata​

To attach metadata to a file, add one metadata file for each file you're uploading. The name of the metadata file should match the original file's name with a meta.json suffix:

my-files/
  report.pdf
  report.pdf.meta.json
  notes.txt
  notes.txt.meta.json

Each metadata file holds a flat JSON object:

{"meta_key1": "value1", "meta_key2": "value2"}
Details

Example metadata file Here's an example of a metadata file that holds hotel reviews. The file is called hotel_reviews.txt:

Hotel reviews
=============

- The hotel was clean and comfortable.
- The staff was friendly and helpful.
- The location was convenient for exploring the city.

This is the corresponding metadata file. It's called hotel_reviews.txt.meta.json:

{
  "source": "hotel_reviews.txt",
  "reviews": 3,
  "average_score": 8.8
}

Prerequisites​

Set up the SDK to authenticate and specify the workspace you want to upload to.

Upload a Folder​

Run the following command replacing ./my-files with the path to the folder you want to upload:

haystack-enterprise upload ./my-files

By default, this uploads every supported file type in the folder, but not its subfolders. To include subfolders, add --recursive. To limit the upload to specific types, see Supported File Types.

After the upload finishes, files take a little while to appear in the platform. If you already have a pipeline deployed, it may not see the new files right away.

Upload with Python​

Upload From a Folder​

This is an example script you can use to upload a folder of files to a Haystack Enterprise Platform workspace:

from pathlib import Path

from haystack_enterprise_sdk.workflows.sync_client.files import upload

upload(
    paths=[Path("./my-files")],
    # workspace_name="my_workspace",  # defaults to DEFAULT_WORKSPACE_NAME
    blocking=True,       # wait until the files show up in the platform
    timeout_s=300,       # how long to wait when blocking
    show_progress=True,
    recursive=True,      # include subfolders
    desired_file_types=[".csv", ".docx", ".html", ".json", ".md", ".txt", ".pdf", ".pptx", ".xlsx", ".xml"],
)

Upload Raw Text​

Use this option when you process text in Python and want to upload the result rather than a file on disk. Here's an example:

from haystack_enterprise_sdk.models import HaystackEnterpriseFile
from haystack_enterprise_sdk.workflows.sync_client.files import upload_texts

upload_texts(
    files=[
        HaystackEnterpriseFile(
            name="example.txt", # the name of the file 
            text="this is text", # file contents
            meta={"key": "value"},  # optional metadata
        )
    ],
    blocking=True,
    timeout_s=300,
)

Examples​

  • Upload a folder including subfolders:
haystack-enterprise upload ./my-files --recursive
  • Upload only Markdown and CSV files from a folder:
haystack-enterprise upload ./my-files --use-type .md --use-type .csv
  • Overwrite files that already exist in the workspace:
haystack-enterprise upload ./my-files --write-mode OVERWRITE
  • Upload without waiting for the files to appear in Haystack Platform:
haystack-enterprise upload ./my-files --no-blocking
  • Upload to a specific workspace, in parallel, with no progress bar:
haystack-enterprise upload ./my-files --workspace-name my-workspace --enable-parallel-processing --no-show-progress
  • Upload with INFO and DEBUG logs turned on:
haystack-enterprise --verbose upload ./my-files