Manage a ServiceBeta
Once a service is live, you keep working on it: ship a new pipeline version, wire it into your app, watch how it runs, and control its cost. Callers keep using the same endpoint while you make these changes. You find and manage all your services in Gateway.
Prerequisites
- You must have read and write permissions to services to update services. For details, see User Roles and Permissions.
- You must have a service created and a pipeline published to it. For details, see Create a Service and Publish a Pipeline to a Service.
- To understand more about services, see Gateway.
View Service Details
Every task below starts here.
- In Haystack Enterprise Platform, click Gateway in the navigation.
- Click the name of the service you want to work on.
What Do You Want to Do?
- Publish a new pipeline version
- View a past revision's pipeline
- Rename a service or edit its description
- Duplicate a service
- Let others try the service
- Deactivate a service to save resources
- Delete a service you no longer need
- Call the service from your app
- Test the service before you integrate
- Trace and debug queries
- Scale the service and control cost
- Make the service available as an MCP tool
Publish the First Pipeline to a Service
If you created a service without a pipeline, its Overview tab shows a Publish your first pipeline section instead of the service details. It offers two ways to publish: Publish from the platform, for a pipeline you built in Pipeline Builder, and Use the CLI or a coding agent, for a Haystack pipeline or agent that lives on your machine. For the steps, see Publish a Pipeline to a Service.
Publish a New Pipeline Version
When you improve a pipeline, publish the new version to the service so callers get it. The endpoint stays the same, so nothing breaks on their side.
- On Overview, in the Pipeline card, click Switch source.
- Choose the pipeline or version you want to serve.
- Describe what changed. Your note appears in the service deployment history.
- Confirm the change.
If your pipeline is in code, click Publish with the CLI instead in the Switch source wizard. You can then copy a prefilled coding agent prompt or CLI commands that publish a new version to this service. That version becomes the one the service serves once it's ready. For details, see Publish a Pipeline to a Service.
While the new version rolls out, the Pipeline card immediately shows its name and version along with a Publishing badge and a note that the service is starting. This also applies to a service's very first publish: the card shows the pipeline that's rolling out instead of a message that nothing is published yet.
To check what's running now or what you served before, use the Deployment history list on Overview. The version marked Serving is the one handling requests. To inspect the pipeline of any version in the list, see View a Past Revision's Pipeline.
View a Past Revision's Pipeline
To find out what changed between versions or which version introduced a problem, open a read-only preview of any revision's pipeline, including revisions that are no longer serving. You can inspect its diagram and YAML without affecting what's currently live.
You can start a preview from three places:
- On Overview, in the Pipeline card, click View pipeline. This opens the currently serving revision.
- In the Deployment history list, find the revision you want to inspect and click its View pipeline action.
- For a revision pushed from outside the platform, click its configuration hash.
The preview opens in a modal:
- To inspect another revision, click it in the timeline down the side. To fetch older revisions, click Load more at the end of the timeline.
- To see what a revision ran, check its pipeline diagram and YAML.
- To see where a revision stands, check its status (Current, Previous, Failed, or Publishing) and metadata, such as when it was published. To return to the revision that's serving requests, click Back to current.
- To build on a revision's pipeline, click Open in Builder. This option isn't available for revisions pushed from outside the platform, or if the underlying pipeline was deleted from the Builder.
If a revision's pipeline definition can't be loaded, the modal shows a message explaining why, for example that the revision failed to publish or that its pipeline definition is missing, along with a retry option where applicable.
Rename a Service or Edit Its Description
Keep the service easy to recognize as your setup grows.
- On Overview, in the About card, click Edit.
- Update the name and description.
- Save your changes.
Duplicate a Service
Duplicating a service creates a new service that starts from a copy of the original service's most recently published pipeline version. Use this when you want to reuse an existing service's setup as the starting point for a new environment, for example to create a staging copy of a production service, without affecting the original service.
- In the navigation, go to Gateway and open the service you want to duplicate.
- Click More Actions and select Duplicate.
- In the Duplicate service dialog, review or update the Name field. It's pre-filled with the original service's name followed by "(copy)".
- Optional: Update the Description field. It's pre-filled with the original service's description.
- Click Duplicate.
We create a new service with the name and description you provided, and publish a copy of the original service's pipeline version to it. The new pipeline version's deployment history includes a note that it was duplicated from the original service.
If the pipeline version can't be published automatically, the new service is still created. You can publish a pipeline version to it manually. For details, see Publish a Pipeline to a Service.
Let Others Try the Service
Share a link so teammates or stakeholders can try the service in a playground-style page, with your own branding and access rules.
- On Overview, in the Pipeline card, click Share.
- Configure the link settings and style.
- Click Generate Link, then copy and share it.
For every share option, see Share a Pipeline Prototype.
Deactivate a Service to Save Resources
Stop a service from handling requests when you don't need it. Deactivating stops active-hour metering. The service record and the published pipeline version stay in place, so you can bring it back anytime.
- Active: The service is deployed and handling requests.
- Standby: The service is idle after the idle timeout in hosting settings. It wakes automatically on a request, or you can wake it manually.
- Inactive: You deactivated the service. It does not handle requests.
- On Overview, in the Pipeline card, click More Actions.
- Choose Deactivate service and confirm. For a production service, type the service name to confirm.
The status changes to Inactive.
If a service is in standby, you can wake it up by clicking More Actions>Wake service. A service goes on standby after it reaches the idle timeout configured in the hosting settings.
To activate an inactive service:
- On Overview, in the Pipeline card, click More Actions.
- Choose Wake service if the service is on standby, or Activate service if it is inactive.
Delete a Service
Remove a service you don't plan to use again. This removes the endpoint
Deleting a service cannot be undone.
- On Overview, in the Pipeline card, click More Actions.
- If the service is active, click Deactivate service first.
- Open More Actions* again and choose Delete service and confirm. For a production service, type the service name to confirm.
The service is removed and you return to Gateway.
Call the Service from Your App
Point your application at the service endpoint to send it real traffic. You can call Chat, Chat Stream, or Chat Completions, or expose the service as an MCP tool. To choose the option that fits your app, see Connect to a Published Service.
- Click Connect.
- Under Add the endpoint, copy the request URL. This URL stays the same when you publish a new pipeline version.
- Authenticate with a workspace API key as a Bearer token in the
Authorizationheader. To create or manage keys, click Manage API keys.
Keep API keys secret. Store them in a secrets manager and never expose them in client-side code. For more information, see Generate API Keys.
Test the Service Before You Integrate
Send a request straight from the browser to confirm the service responds as you expect.
- Click Connect.
- Under Try it, edit the sample request if needed.
- Click Send request and check the response.
Test requests hit the live service, use the currently serving revision, and count against your quota.
Trace and Debug Queries
Find slow or failing runs and see how a query moved through the pipeline.
- Click Traces.
- Open a trace to inspect its steps.
Tracing works the same as for pipelines. To read and use traces, see Trace with Built-In Traces.
Find a Cheaper Configuration with the AI Operator
To find out whether the service can cost less without answering worse, start an optimization run on its AI Operator page. For steps, see Start an Optimization Run.
Check Hardware Metrics
If a serverless service runs slowly or fails, check whether it's running out of CPU or memory. The Metrics tab is available only for serverless services.
- On the Service Details page, click Metrics.
- Review the CPU usage, memory usage, and replica count charts, grouped by pod.
Result: You can see whether the service needs more resources. If it does, raise Min replicas or Max replicas in the Hosting section of Settings. For details, see Scale the Service and Control Cost.
Scale the Service and Control Cost
Tune how many replicas run and how long the service stays active to balance responsiveness against cost. These settings apply only to this service. If the same pipeline is also deployed on its own, configure that separately in the pipeline deployment settings.
Credits consumed ≈ min replicas × hours deployed. Services on standby or in inactive status consume no credits.
- Click Settings and go to the Hosting section.
-
For a service on serverless hosting, to give each replica more or less compute, select a size under Instance size. The list offers the sizes your organization has enabled: Small, Medium, Large, or Extra large. Organization admins choose these sizes in Organization Settings. For details, see Managing Compute Settings. If the service already runs on a size your organization has since turned off, that size stays in the list so you can see the current value. Confirm the change when prompted.
The service's replicas restart on the new size. Queries already running finish on the old replicas first. If your organization has no capacity for the size you chose, an error appears and the size stays the same. Contact your Organization Admin.
- Choose a preset, or select Custom to set your own values:
- High demand: For production traffic that needs high availability and scaling.
- Low demand: For testing or services that are used rarely.
- Custom: Set Min replicas, Max replicas, and Idle timeout yourself.
- Confirm your changes.
You can change these settings at any time.
Add Secrets to a Service
Add a secret to a service when only that service needs a credential, for example a production API key. A service secret overrides workspace and organization secrets with the same name, so you can give one service its own key without changing what other services use. The secret name must match the environment variable that the pipeline component reads.
To see how secrets at different levels relate, see Secrets and Integrations.
If the service is a managed service, the Secrets tab shows a warning that service secrets aren't applied yet. Service secrets apply only to services that run in serverless mode.
Prerequisites
You need permission to read secrets to see the Secrets tab, and permission to write secrets to create, edit, or delete them.
Create a Service Secret
- In Haystack Enterprise Platform, click Gateway and open the service.
- Click Secrets.
- Click Create Secret.
- In Key, type the secret name, for example
OPENAI_API_KEY. Use only letters, numbers, and underscores, and start with a letter or an underscore. - In Value, enter the secret value.
- Click Create.
The secret appears in the list with its Key, masked Value, Last updated date, and Created by user. The service uses it the next time it runs. The secret isn't listed in workspace or organization settings.
Edit or Delete a Service Secret
- Open the service and click Secrets.
- In the secret's row, choose the edit or delete action.
- Save your changes, or confirm the deletion.
If you delete a service secret and a workspace or organization secret with the same name exists, the service uses that secret instead.
Manage Service Secrets with the API
To add or manage service secrets from your own scripts, use the endpoints under /api/v2/workspaces/{workspace_id}/deployments/{deployment_id}/secrets, where deployment_id is the service ID:
- Create a secret:
POSTwithnameandsecretin the body. - List secrets:
GET. - Update or delete a secret:
PUTorDELETEto/secrets/{secret_id}.
For the full request and response schemas, see the API reference.
Assign the Service to a Fleet Environment
Fleet environments let your organization admin group services onto shared underlying infrastructure. Assign a service to a fleet environment when you want it to share infrastructure with other services instead of running on its own dedicated setup.
- Click Settings and go to the Hosting section.
- In the Environment list, select the fleet environment you want to use, or keep Own (isolated) to keep the service on its own dedicated infrastructure.
- Confirm your changes.
Changing the environment restarts the service, so we show a warning before you confirm. If you select an environment whose hosting settings don't match this service's current configuration, we show an inline message explaining the mismatch so you can adjust your settings before saving.
Only an organization admin can create, rename, or delete fleet environments. For details, see Managing Organization Settings. If the service uses an environment, it uses that environment's settings, and you change them on the environment. Services in the same environment share instances, so their requests add up to the environment's load. To check that load, see View Environment Metrics. To see whether the shared environment has enough capacity for its services, check its CPU, memory, and replica usage. For details, see Check an Environment's Resource Usage.
Make the Service Available as an MCP Tool
Expose this service to your workspace MCP server so AI assistants — Claude Desktop, Cursor, VS Code, and more — can call it as a tool. Using a service as an MCP tool is recommended for production use because the service endpoint stays stable when you publish a new pipeline version.
- Click Settings.
- In the MCP tool section, turn on Use as MCP tool.
The service is immediately available to AI assistants connected to the workspace MCP server.
Keep a Service Active with Scheduled Calls
Turn on scheduled calls to make the service call itself at regular intervals, so it doesn't go idle between real requests. This helps if your service gets sporadic traffic and you still want fast response times. Scheduled calls work only for services on non-serverless hosting.
- Click Settings and go to the Scheduled Calls section.
- Turn on Enabled.
- In Every (minutes), set how often the service should be called.
- Optional: In Query, type the text to send with each scheduled call. Leave it empty to dispatch an empty query.
- Optional: In Params (JSON), type extra parameters to send as valid JSON.
- Click Save.
If a schedule already exists, the section shows Last call, which is the time of the most recent scheduled call. Turn off Enabled and save to stop scheduled calls without losing your saved interval, query, and parameters.
Submit and Monitor Durable Runs
Use a durable run when you can't count on keeping a connection open until a run finishes. A durable run keeps working even if you close the browser, lose your connection, or a worker restarts. The service saves its progress at checkpoints as the run moves through the pipeline, so if anything interrupts the run, it resumes from the last checkpoint instead of starting over. You can come back anytime to check progress, see streamed output, or cancel the run.
- Viewing the Runs tab, run deep links, and the Durable option in Connect requires the same read permission as ordinary deployment history, plus the search history read permission.
- Canceling a run also requires the services write permission.
- Durable runs must also be turned on for your organization and supported by this service. If either is unavailable, the Runs tab, run deep links, and the Durable option in Connect are hidden.
Submit a Durable Run
- Click Connect.
- Under Try a durable run, switch the response mode to Durable.
- Edit the request body if needed, then click Start run. To submit from your own tooling instead, use the Submit with cURL example. Set
HAYSTACK_API_KEYto a workspace API key, which is sent as a Bearer token. Before each new run, setDURABLE_IDEMPOTENCY_KEYto a unique value. When you retry a request, reuse the same key and the identical JSON body so the service doesn't start a duplicate run.
The Submission response appears next to the request editor, along with the submission time. A successful submission opens the new run on the Runs tab. If the submission is rejected, send the request again to retry it. Durable runs support synchronous pipelines without Agents.
View Durable Runs
- Click Runs.
- Find your run in the Active list, which shows runs that are submitting, queued, running, or waiting, or in the History list, which shows finished runs and rejected submissions.
- Click Load older at the bottom of a list to see more runs, or Refresh to reload it.
- Click a run's query to open its details. To go back to the list, click All runs.
Use a run's detail page to follow its progress and debug problems. The page shows:
- Progress, a timeline of the checkpoints and other events the run has reached so far.
- Output, streamed while the run is in progress. If the connection drops, the view reconnects automatically. You can also click Reconnect.
- Final result once the run finishes, or the error if it failed.
- View trace, a link to the run's trace for deeper debugging.
If the platform can't reach the runtime, the run is marked Stale and the page shows the last observed state until the next successful update.
Cancel a Run
- Open the detail page of a run that's still active.
- Click Cancel run, then confirm in the dialog. To keep the run going, click Keep running.
Once a run reaches a terminal state, canceling is no longer available.
Set Up and Use an AI Operator
To lower this service's cost without dropping below the answer quality you need, set up an AI Operator. It measures the service's cost and answer quality and suggests cheaper configurations that are expected to keep that quality. Suggestions are advisory only: the Operator never applies a change to your service for you.
On the Service page, next to the actions menu, click Set up AI Operator if the service has no Operator yet, or AI Operator to open its existing one. For steps to add an Operator, measure a baseline, and review suggestions, see Set Up an AI Operator for a Service.
What To Do Next
- Call the endpoint from your app with an API key.
- To choose how your app calls the service (Chat, Chat Stream, Chat Completions, or MCP), see Connect to a Published Service.
- Trace with Built-In Traces when you need to debug a run.
Related Information
Was this page helpful?