Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Monitor Pipeline Performance

If you're wondering how many requests you can send from your search system to Haystack Enterprise Platform or what speed you can expect, here are your answers. Use built-in metrics, logs, and KPI dashboards to monitor your pipeline performance.

Scaling​

Unexpected surges in traffic are a challenge, but Haystack Enterprise Platform seamlessly handles scalability in response to increased demand. Haystack Enterprise Platform uses autoscaling, which automatically adjusts the infrastructure based on the usage. This ensures there are no disruptions in service, regardless of the number of concurrent requests. The system dynamically reallocates resources to maintain optimal performance. Scaling is entirely automated and doesn't require any manual adjustments from you.

Speed​

Several factors influence the speed of your search system:

  • Model size
    Pipelines with large models are slower. With LLMs, it's often a tradeoff between speed and performance. Large models, like GPT-5, may give better answers than smaller models, but they're also much slower because of their size.
  • Where your model runs
    You can run models locally, on your machine, or remotely, through API. Small models used with Retrievers or Readers are faster to run locally because they don't need much power. Large models, like ChatGPT or GPT-4, need dedicated and optimized hardware, and running them remotely is usually faster.
  • Pipeline configuration
    Some components or their settings can slow your pipeline. For example, a Ranker improves the results but also slows the system down. The same goes for a Reader with a high top_k value. It's often about finding the balance between speed and performance.
  • The length of generated responses
    For generative QA pipelines, the number of tokens you want the model to generate as an answer influences your system's speed. The longer the answer, the slower the system.

Optimizing the speed of your pipeline is all about finding the balance among these factors. It involves choosing the right-sized model, running it where it's fastest, and finding a configuration with optimal performance.

If your pipeline contains components that work better on a GPU, turn on GPU acceleration. For details, see Enable GPU Acceleration for Pipelines.

Costs​

To check how much credits and storage units your pipeline has used, check the Usage dashboard. For instructions, see Understand Your Pipeline Usage.

To check token consumption, go to the Traces tab on the Analytics page and click a trace. The trace details page shows the number of tokens used for each span, broken down by prompt, completion, reasoning, and cache tokens. For details, see Trace Your Pipelines in the UI.

Pipeline Analytics​

To check what queries were asked, the top answers, and how long it took to find them, click the pipeline's name on the Pipelines page and switch to Analytics. This brings you to the pipeline overview, where you can view all the information about your pipeline.

Overview​

The Pipeline Overview shows a KPI dashboard with key performance metrics in an easy-to-read format:

  • Total queries: Total number of queries your pipeline processed. This metric helps you understand usage patterns and traffic volume.
  • Documents: Total number of documents indexed for this pipeline.
  • Feedback coverage: Percentage of queries that received user feedback. Higher feedback coverage gives you better insights into pipeline performance. If this shows "N/A", it means no feedback was given yet.
  • Average response time: Average time it took to generate a response. Monitor this metric to identify performance issues or improvements after configuration changes.
  • Minimum inference time: The fastest response time recorded across all queries. This helps you understand your pipeline's best case performance.
  • Maximum inference time: The slowest response time recorded across all queries. Use this metric to identify potential performance bottlenecks or outliers.
  • Query volume: Number of queries your pipeline processed over the last 7 days.
  • Feedback distribution: Distribution of feedback received for the pipeline's responses. You can click on any slice of the pie chart to navigate to the Search History tab with the feedback rating filter automatically applied for that feedback category (Accurate, Inaccurate, or Wrong Document). This lets you quickly drill down into specific feedback types for detailed analysis.
  • Query Activity Heatmap: Shows when your pipeline is most active by displaying query volume across hours and days. This helps you understand usage patterns, identify peak activity period, or plan maintenance windows.

Traces​

info

Traces are collected for queries run on or after 25 June 2026. Before that date, traces are not available. To collect traces for pipelines deployed before that date, undeploy the pipeline and deploy it again.

Traces are detailed records of every step in the search process. Each trace represents the full journey of a single request as it moves through the pipeline, from the input that triggered it to the response it returned. Within each trace, there are spans, which represent individual steps. Each span is a single unit of work, like "call the retriever" or "generate the answer", with its own start and end time, and attributes, like input and output, token counts, or errors.

You can check your pipeline's traces in the Traces tab. Each trace contains component-level spans, token usage broken down by prompt, completion, reasoning, and cache tokens, inputs and outputs, log entries, run status, and failure details when a run fails, including the root-cause component. Traces also record the source that triggered the run, such as Playground, Builder, API, or a shared prototype, and the pipeline version that served it. Traces are available in the Builder on the Analytics tab, but also after running the pipeline in Builder or Playground.

Clicking a trace opens its details, where you can check the status of the run, where it was run, components that failed, duration, timeline of all spans, token counts, and more. You can view spans within a trace and the logs related to each span. Click a span to select it and view its details. You can also export the trace as JSON from the trace details page.

You can also navigate between traces and Search History: jump from a trace to its Search History entry, or open the Trace Detail page directly from a query's detailed view in Search History. To compare two pipeline versions and see exactly how a change affected latency, token consumption, and the answer itself, use Compare on the Traces page.

For the full list of what a trace captures, how to filter and search traces, and how to run a component from a trace, see Trace Your Pipelines in the UI.

In addition, Haystack Platform supports tracing with Langfuse and Weights & Biases' Weave. To learn more about using tracing, see Trace Your Pipelines.

What a Trace Captures​

A trace is the complete execution record of one pipeline run, from the input that triggered it to the response it returned. Each trace captures:

  • Spans: A hierarchical tree of every component that ran, with per-step timing, component type, and metadata tags.
  • Token usage: Prompt, completion, reasoning, and cache tokens, normalized across providers and attributed to the model that used them. This lets you compare token consumption across models even when an agent uses more than one.
  • Inputs and outputs: The data flowing into and out of every component.
  • Logs: Every structured log entry from the run, with its level and source.
  • Failures: The exception type, message, and full stack trace, with the root-cause component flagged so you know where the run failed and whether it's a configuration error you need to fix, or an external provider or infrastructure error that just needs a rerun.
  • Metadata: Status, total latency, timestamps, the source that triggered the run (Playground, Builder, API, or a shared prototype), and the pipeline version that served it.

Trace Data Retention​

Haystack traces stored with your search history are kept for a limited time. Each organization has a trace retention period, in days, that controls how long trace data stays available. You can see the current value in the trace_retention_period_days field returned by the /me endpoint. If your organization needs a different retention period, contact your deepset representative.

When a trace passes its retention period, it goes through two stages:

  • Soft delete: The trace is marked as expired. It's automatically excluded from the search history list, and requests for that query's trace detail or trace payload no longer return trace data, even though the underlying search history entry and record still exist.
  • Permanent delete: After a grace period following the soft delete, the trace artifact itself is permanently removed from storage. The search history entry stays in place, but the trace data it pointed to is deleted for good.

This process runs automatically in the background, so you don't need to take any action. Keep in mind that once a trace's retention period elapses, you can no longer view it in the search history or use it for debugging — plan on exporting or analyzing traces you need to keep before they expire.

History​

For detailed performance and feedback analysis, use the History tab to review individual queries and responses. You can generate executive summaries with AI-powered Feedback Insights. You can start with one of the suggested prompts or enter your own query. Based on the prompt, AI analyzes your pipeline activity and generates a summary.

Use Smart Filters to check if your pipeline is following patterns worth investigating. Smart filters show the analysis of the last 100 queries to surface any anomalies or patterns that might require attention.

Customize the table to show the columns you need, and view the conversation history for each query. You can choose the pipeline version whose history you want to view. Use notes and hashtags to add context to your queries and responses. Both notes and hashtags are searchable. You can view and manage them if you enable the Notes and Hashtags columns in the Search History table. To do this:

  1. Click Manage table preferences.
  2. Choose to display the Note and Hashtags columns. You can then add notes and hashtags to your queries using these columns in the table.

You can now see notes and hashtags for each query in the History table. You can also view and manage them in the detailed view for each query.

The History table includes a Status column that shows whether each run succeeded or failed. Users can submit feedback for successful runs only. Runs that ended with an error have a failed status and cannot receive new ratings. Any feedback already recorded for a run before it failed continues to display.

Use search history to identify slow queries, improve answer quality, and understand user needs. Download the search history as a CSV file for further analysis.

The exported CSV file includes the following columns:

  • query_id: The unique identifier for each search history record. Use this column to reference a specific query in your integration or downstream tooling.
  • answer: The answer text returned for the query. For retrieval and agent pipelines, this is populated from the first retrieved document's content when no explicit answer is present.
  • file_name: The name of the source file associated with the result. For retrieval and agent pipelines, this falls back to the file from the first retrieved document.
  • file_id: The unique identifier of the source file. Resolved the same way as file_name.
  • created_at: The timestamp of when the query was made, in ISO 8601 format. If your integration uses the legacy column name time, it continues to work and returns the same value.
  • feedback_created_at: The timestamp of when user feedback was submitted for the query, in ISO 8601 format. This column is empty when no feedback has been given for a query.
  • pipeline_version_id: The ID of the pipeline version that served the query. This lets you tie each query row to the exact pipeline version that served it, which is useful when you have multiple versions deployed and want to compare behavior or investigate a specific version.
info

The Status column is not included in the CSV export.

When you navigate to Search History from the Feedback Distribution chart, the feedback rating filter is automatically applied to show only queries matching the selected feedback type. You can clear this filter or add additional filters as needed.

Use filters to narrow down the results shown in the history table. Any filters you apply are automatically synced with the URL, so you can easily share the filtered view with others by just sharing the URL.

Logs​

Check pipeline logs to see what happened since it was deployed. You can view the logs on the Pipeline Details page (just click the pipeline name to get there). Expand messages for more details and possible actions.