Durable RunsBeta
A durable run is an alternative way to submit a request to a service. Instead of waiting on an open connection for the pipeline to finish, you submit the request, get a receipt immediately, and the pipeline keeps working even if you close the browser, lose your connection, or the underlying worker restarts. You come back later, or watch live, to see checkpoint progress and the final result.
Use durable runs for long-running pipelines, so you don't have to hold a connection open for the whole run.
Durable runs are only available for a service when:
- The organization has the durable runs capability turned on.
- The pipeline published to the service supports durable execution.
If either condition isn't met, the durable run controls are hidden and any in-progress observation stops, but accepted work keeps running.
Permissions
To see the Runs tab, open run deep links, and connect in durable mode, you need the search-history: read permission, the same one you need for service deployment history. To cancel a durable run, you also need the services: write permission.
Submission and Idempotency
When you submit a durable run, the request includes a stable Idempotency-Key. If you retry the exact same request, the service recognizes the key and avoids starting a duplicate execution. If the service is at capacity, you see a message saying capacity is full, and you can retry later. When a submission is rejected because the service is at capacity or the request was invalid, your next submission uses a new idempotency key, so it counts as a new request, not a duplicate.
Redeploying a Service
The first run you submit to a service sets up durable runs on the service's active revision. You can redeploy the service while durable runs are in flight. Runs already accepted finish on the revision they started on, and your next submission uses the new revision.
If the redeploy fails, the service stays open for durable runs, and your next submission starts them again on the active revision.
Pausing Durable Runs and Limits
To stop a service from accepting new durable runs, set durable_enabled to false on the service. The service refuses new submissions until you set it back to true, including after a redeploy. Runs already accepted keep going, and retrying an accepted idempotency key returns the original run instead of starting a new one.
Limits are settings of the service, not of a revision. When a service has reached its maximum number of unfinished runs, it refuses a new submission with a 429 error and the code durable_capacity_exhausted. Wait for runs to finish, or raise the limit. Changing a limit doesn't affect runs in flight. A new run keeps the result retention that applied when it was accepted, capped at the retention of the revision it runs on. A higher concurrency or retention limit applies fully after the next redeploy, and a lower limit applies at once.
What Happens When a Run Restarts
If a worker or host is lost, a run starts again from the beginning with its original inputs. The service doesn't save intermediate progress. Model calls and Agent tool calls can run again, so an action such as posting a comment can happen twice. Keep this in mind when an Agent uses tools that change data in other systems.
To let a pipeline continue where an earlier attempt stopped, map a query_id input in the pipeline's inputs section to the component input that should receive it. Every attempt of a durable run gets the run's search-history ID in that input, and the ID is the same on each attempt. Your pipeline can use it as a key to save its own progress and look it up when it restarts. For example, an Agent tool can check whether it already posted a comment for this ID before posting again. Pipelines that don't map query_id receive their original inputs unchanged.
To find out which attempt finished a run that restarted, read the attempt field. The final result, done, and error frames of a durable run's completion stream carry it, like the streamed chunks do.
When a host stops, for example because of a delete, an eviction, or a node drain, each running run gets 60 seconds by default to finish there. A run still going after that moves to another host and starts again. Normal releases drain hosts first, so they wait for running runs before stopping a host. Your administrator can change this time in the Fleet settings.
Only failures use up restarts. By default, a run can restart up to three times after its host crashes, and up to three more times after its worker fails. The next crash or worker failure fails the run, and a run that fails because of its worker keeps the worker's error. A handover from a host that shuts down cleanly doesn't count toward either limit, so a long run can move between hosts through many releases and evictions. If you set the limit to 0, the first crash or worker failure fails the run, but a clean handover still runs it again on the next host. A run that hangs stays active until you cancel it. Your administrator can change the limit in the Fleet settings.
A run that produces no output for a long time isn't restarted, because Agent tools can be quiet while they work. Your administrator can set an idle timeout. If a run stays silent longer than the timeout, only that run fails. Other runs on the same host continue, and the failed run keeps its capacity on the host until its request ends.
If a submission with an Agent or asynchronous pipeline is rejected with a 422 error, ask your administrator to check that the service's runtime is up to date. Older runtimes reject Agent and asynchronous requests.
Related Information
Was this page helpful?