About This Task

By default, pipelines run synchronously, meaning they execute components one after another, waiting for each to finish before proceeding. You can modify the pipeline YAML to enable async mode, allowing components in different branches to run in parallel.

In async mode, components run in the background, so the system doesn't have to wait for one to finish before starting another. This can be useful when:

• Multiple branches can run in parallel, reducing overall execution time.
• A component needs to handle multiple requests simultaneously. (For this, the component must implement the run_async method. Otherwise, it falls back to sync.)

Enable Async Mode

1. Log in to deepset AI Platform and go to Pipelines.

2. Find the pipeline you want to run async, click More Actions next to it, and choose Edit.

   

3. When the pipeline opens in Pipeline Builder, switch to the YAML view.

   

4. Scroll down to the end of the YAML and add async_enabled: True, like this:

   YAML

   inputs:  
     files:                           
     - file_classifier.sources
   max_runs_per_component: 100
   
   metadata: {}
   async_enabled: True # enables async mode
   

5. Save your pipeline.

What To Do Next

To run your pipeline, deploy it.