Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Use Durable RunsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Use a durable run for long-running requests when you don't want to keep a connection open until the run finishes. A durable run keeps working even if you close the browser, lose your connection, or a worker restarts. The service saves the run's inputs and status, so if a worker restarts, the run begins again from the start with its original inputs instead of being lost.

Prerequisites​

  • You must have a service with a published pipeline. For details, see Publish a Pipeline to a Service.
  • You need the deployment history read permission and the search history read permission to see the Runs tab, run deep links, and the Durable option in Connect.
  • To cancel a run, you also need the services write permission.
  • The service must support durable runs. If it doesn't, the Runs tab, run deep links, and the Durable option in Connect are hidden. You don't need to turn durable runs on for each service: the first run you submit sets it up.

Submit a Durable Run​

Durable runs support valid pipelines on the supported Haystack version, including Agents, asynchronous pipelines, and Agents nested in tools or SuperComponents. Agent and asynchronous support depends on the service's runtime.

  1. In Haystack Enterprise Platform, click Gateway in the navigation and open the service.
  2. Click Connect.
  3. Under Try a durable run, switch the response mode to Durable.
  4. Edit the request body if needed, then click Start run. The run is added to the Active list on the Runs tab, where you can follow it. If it doesn't appear there, check the History list for a rejected submission and open it for details.

For services hosted on shared Fleet workers, you can submit a run even when the service is sleeping and has no worker running. The service accepts and saves the run, then starts a worker to process it. Services on dedicated hosts still need a ready host. A durable run uses the same instance size as the service's ordinary requests: the size of its fleet environment, or the service's own size if it isn't in one. A later change to the instance size applies only to durable hosting created after the change.

To submit from your own tooling instead, use the Submit with cURL example. Set HAYSTACK_API_KEY to a workspace API key, which is sent as a Bearer token. Before each new run, set DURABLE_IDEMPOTENCY_KEY to a unique value. When you retry a request, reuse the same key and the identical JSON body so the service doesn't start a duplicate run.

What Happens When You Redeploy the Service​

The first run you submit to a service sets up durable runs on the service's active revision. You can redeploy the service while durable runs are in flight. Runs already accepted finish on the revision they started on, and your next submission uses the new revision.

If the redeploy fails, the service stays open for durable runs, and your next submission starts them again on the active revision.

Pause Durable Runs and Set Limits​

To stop a service from accepting new durable runs, set durable_enabled to false on the service. The service refuses new submissions until you set it back to true, including after a redeploy. Runs already accepted keep going, and retrying an accepted idempotency key returns the original run instead of starting a new one.

Limits are settings of the service, not of a revision. When a service has reached its maximum number of unfinished runs, it refuses a new submission with a 429 error and the code durable_capacity_exhausted. Wait for runs to finish, or raise the limit. Changing a limit doesn't affect runs in flight. A new run keeps the result retention that applied when it was accepted, capped at the retention of the revision it runs on. A higher concurrency or retention limit applies fully after the next redeploy, and a lower limit applies at once.

What Happens When a Run Restarts​

If a worker or host is lost, a run submitted after this change starts again from the beginning with its original inputs. The service doesn't save intermediate progress. Model calls and Agent tool calls can run again, so an action such as posting a comment can happen twice. Keep this in mind when an Agent uses tools that change data in other systems.

To let a pipeline continue where an earlier attempt stopped, map a query_id input in the pipeline's inputs section to the component input that should receive it. Every attempt of a durable run gets the run's search-history ID in that input, and the ID is the same on each attempt. Your pipeline can use it as a key to save its own progress and look it up when it restarts. For example, an Agent tool can check whether it already posted a comment for this ID before posting again. Pipelines that don't map query_id receive their original inputs unchanged.

To find out which attempt finished a run that restarted, read the attempt field. The final result, done, and error frames of a durable run's completion stream carry it, like the streamed chunks do.

The number of times a service restarts a run is limited. By default, a run can execute up to three more times after its first attempt, and a handoff from a host that is shutting down gracefully counts toward this limit. If the limit is reached, the run fails. Your administrator can change the limit in the Fleet settings.

A run that produces no output for a long time isn't restarted, because Agent tools can be quiet while they work. Your administrator can set an idle timeout. If a run stays silent longer than the timeout, only that run fails. Other runs on the same host continue, and the failed run keeps its capacity on the host until its request ends.

If a submission with an Agent or asynchronous pipeline is rejected with a 422 error, ask your administrator to check that the service's runtime is up to date. Older runtimes reject Agent and asynchronous requests.

Run Scheduled Calls as Durable Runs​

A scheduled call on a service is a plain chat request by default. It's cut off at the request time limit, it isn't retried, and it can't resume. For a long scheduled job, such as a nightly or weekly flow, submit each scheduled call as a durable run instead.

  1. Open the service, click Settings, and go to the Scheduled Calls section.
  2. Turn on Run as a durable execution.
  3. Click Save. At the next scheduled time, the run appears in the Active list on the Runs tab.

The switch is off by default. The service must support durable runs, as described in the prerequisites. You can also set durable to true in the request body of the schedule's PUT call to the service's cron endpoint.

Each scheduled call is then submitted the same way as a run you start from Connect. It uses the schedule's query and parameters, and it runs as the user who saved the schedule.

  • One run per scheduled time: Each scheduled time gets its own fixed idempotency key, built from the schedule and the scheduled time. If a submission fails after the service accepted the run, the next attempt uses the same key and replays that run instead of starting a second one. External side effects of a run, such as posting a comment, can still happen twice.
  • Deadline for each run: A run must be accepted before the next scheduled time or before the schedule's active window closes, whichever comes first. The deadline is never longer than the service's normal admission window, so for long intervals it can fall before the next scheduled time. The service refuses a run after its deadline, so a scheduled time that the scheduler has moved on from never starts late.
  • Refused submissions: If the service refuses a submission, for example because it has reached its maximum number of unfinished runs, the schedule isn't marked as called. The scheduler retries the same scheduled time until its deadline passes. To free capacity, wait for runs to finish or cancel some. For the limits, see Pause Durable Runs and Set Limits.
  • Skipped scheduled times: If the next scheduled time arrives while an earlier one still isn't submitted, the scheduler gives up on the earlier one, logs it as skipped once, and submits the latest due time. If times are skipped often, free capacity as described in the previous item. A scheduled time that falls outside the schedule's active window was never due, so it doesn't count as skipped.
  • Last call time: The Last call value for a durable schedule is the scheduled time that was last handled, not the moment the submission succeeded.
  • Search history: A scheduled durable run doesn't use a search session. It writes its own search history entry, like any other durable run.

To follow a scheduled run, open it on the Runs tab as described in View Durable Runs.

View Durable Runs​

  1. Open the service and click Runs.
  2. Find your run in the Active list, which shows runs that are submitting, queued, running, or waiting, or in the History list, which shows finished runs and rejected submissions.
  3. Click Load older at the bottom of a list to see more runs, or Refresh to reload it.
  4. Click a run's query to open its details. To go back to the list, click All runs.

If the platform can't reach the runtime, the run is marked Stale and the page shows the last observed state until the next successful update.

Read a Run's Details​

Use run's details to check where a run stands, get its result, and collect what you need to debug a failure.

  • Header: The run name and status. Click View trace to open the run's execution trace. Use the copy run link icon to share the run's address. The ⋯ menu has Copy diagnostic details and Copy execution ID, which copy to your clipboard.
  • Summary: The facts for the run are Submitted, Last observed, Total elapsed, Kind, Attempt, and Submitted by, plus Tokens when the platform has recorded them. For a run that restarted, Attempt reads, for example, "2 · restarted". For a queued run that hasn't started, it reads "Not started".
  • IDs: Below the summary, the Execution ID, Query ID, and Pipeline revision each have a copy icon. Use them when you contact support or look the run up in search history. The pipeline revision identifies the exact pipeline configuration that the run used.
  • Output: The readable answer is shown first, followed by the Final result · JSON.

If a run fails, you can see the error summary. Click Copy diagnostic details in the summary to copy the information to send to support. The Output section then shows the error as JSON in an Error · JSON card. If the run recorded a partial result before it failed, the title is Recorded output · JSON.

You can also check the details of the submission:

  • Request: The original JSON that was submitted (Input · JSON), with a copy action. If the request wasn't retained, the tab says so.
  • Submission details: The key and value details of how the run was submitted.
  • Follow with cURL: A Stream attachment · bash command that watches this run's events without starting another run. Set HAYSTACK_API_KEY to a Personal Key with access to the service before you run it. The tab also explains how to resume from the last event you received if the connection drops.

Follow Tool Calls While an Agent Runs​

On the detail page of a run, the output shows the text the pipeline streams and, for an Agent, each tool call it makes, in the order they happen. A tool call is listed by tool name with a spinner until its result arrives. Expand it to see the arguments the Agent sent, once they're complete, and a preview of the result. Use this to check what an Agent is doing during a long run, before the final answer is ready.

When the run finishes, the final result replaces the text streamed so far. The tool calls stay above it. If the run ends before a tool returns, its spinner stops.

Token Usage of Durable Runs​

Durable runs count toward your token usage, so you can track the cost of long-running Agent runs like any other query. The platform records a run's tokens once, when the run finishes, whether it completed, failed, or was canceled. It reads them from the run's trace. Runs that finished before this recording was added aren't included.

When a run restarts, only the attempt that finishes the run is counted. The platform gets a trace only when an attempt ends, so tokens from an earlier attempt lost with its host aren't recorded.

Cancel a Run​

  1. Open the detail page of a run that's still active.
  2. Click Cancel run, then confirm in the dialog. To keep the run going, click Keep running.

Once a run reaches a terminal state, canceling is no longer available. After you confirm, check the run's status on its detail page or in the History list to see that it was canceled.

Cancellation is cooperative. The pipeline, including Agents nested in it, checks for the cancellation between steps and stops at the next check. A synchronous tool that is already running can keep going after the run is canceled, and it can keep using host capacity until the host is replaced.

Deactivate, Convert, or Delete a Service with Durable Runs​

A service that has durable run history can be deactivated or converted to serverless, and its workspace can be deleted. Each action treats the history and in-flight runs differently.

Deactivate a Service or Convert It to Serverless​

When you deactivate a service or convert it to serverless, the service stops accepting new durable runs. It keeps its run history, and runs already in flight continue. To submit durable runs again, activate the service first. A submission to a deactivated service is refused and doesn't reopen durable runs.

Delete a Workspace That Has Durable Services​

Deleting a workspace discards the durable run history of its services. Deletion can take several requests while the platform removes the durable hosts:

  1. Send the delete request. If the platform is still removing durable hosts, it returns 202 with durable_cleanup_pending set to true and a Retry-After of 5 seconds. The workspace isn't deleted yet, and its services no longer accept new durable runs.
  2. Send the same request again every five seconds until you receive 204, which means the workspace is deleted. A 404 after a 202 also means that the deletion finished.

If a service has unfinished durable runs, or runs whose state is unknown, the request returns 409 with the code durable_deletion_conflict. Every service in the workspace has already stopped accepting new durable runs, so wait for the runs to finish, or cancel them, and then send the request again.

The same applies when an organization is deleted: durable hosts are removed first, and the organization is deleted afterward.

For the request details, see Delete Workspace.

What To Do Next​