Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Run a Pipeline Larger Than 256 KB with the REST APIBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Run a pipeline whose YAML configuration is larger than 256 KB on a serverless service.

About This Task​

Saving a pipeline or index configuration larger than 256 KB fails with error 413. This limit applies when you create, validate, or update a pipeline, pipeline version, or index, in Builder or through the API. It doesn't apply to a YAML configuration pushed to a service as a revision. To run a larger pipeline, create a serverless service and push the YAML to it directly. The API calls a service a deployment.

Because the pipeline isn't saved as a pipeline version, you can't open or edit it in Builder. To change it, push and activate a new revision.

Prerequisites​

  • You need a Haystack Platform API key. For details, see Generate API Keys.
  • The pipeline YAML must pin a Haystack version, for example haystack-ai==<version> under dependencies. A revision without a pinned version is rejected.
  • jq to put the YAML file into the request body.

Run the Pipeline​

In each request, replace <YOUR_WORKSPACE> with your workspace name and <YOUR_API_KEY> with your API key.

  1. Create a serverless service with Create Deployment. Leave out origin_pipeline_id:

    curl --request POST \
         --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/deployments \
         --header 'content-type: application/json' \
         --header 'authorization: Bearer <YOUR_API_KEY>' \
         --data '{"name": "my-large-pipeline", "deployment_mode": "SERVERLESS"}'

    Copy deployment_id from the response.

  2. Push the YAML as a revision with Create Deployment Revision. Send the YAML in config_yaml, set source_type to EXTERNAL_PIPELINE, and leave out source_version_id. Replace <DEPLOYMENT_ID> with the ID from step 1 and pipeline.yaml with your file:

    jq -n --rawfile yaml pipeline.yaml \
      '{comment: "Initial version", source_type: "EXTERNAL_PIPELINE", config_yaml: $yaml}' > revision.json

    curl --request POST \
         --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/deployments/<DEPLOYMENT_ID>/revisions \
         --header 'content-type: application/json' \
         --header 'authorization: Bearer <YOUR_API_KEY>' \
         --data @revision.json

    Copy revision_id from the response.

  3. Deploy the revision with Activate Deployment Revision. Replace <REVISION_ID> with the ID from step 2:

    curl --request POST \
         --url https://api.cloud.deepset.ai/api/v1/workspaces/<YOUR_WORKSPACE>/deployments/<DEPLOYMENT_ID>/revisions/<REVISION_ID>/activate \
         --header 'authorization: Bearer <YOUR_API_KEY>'

To check that the service is running, open it in Haystack Enterprise Platform and see Monitor and Debug a Service.

What To Do Next​