Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Monitor Pipeline Performance

If you're wondering how many requests you can send from your search system to Haystack Enterprise Platform or what speed you can expect, here are your answers. Use built-in metrics, logs, and KPI dashboards to monitor your pipeline performance.

Scaling​

Unexpected surges in traffic are a challenge, but Haystack Enterprise Platform seamlessly handles scalability in response to increased demand. Haystack Enterprise Platform uses autoscaling, which automatically adjusts the infrastructure based on the usage. This ensures there are no disruptions in service, regardless of the number of concurrent requests. The system dynamically reallocates resources to maintain optimal performance. Scaling is entirely automated and doesn't require any manual adjustments from you.

Speed​

Several factors influence the speed of your search system:

  • Model size
    Pipelines with large models are slower. With LLMs, it's often a tradeoff between speed and performance. Large models, like GPT-5, may give better answers than smaller models, but they're also much slower because of their size.
  • Where your model runs
    You can run models locally, on your machine, or remotely, through API. Small models used with Retrievers or Readers are faster to run locally because they don't need much power. Large models, like ChatGPT or GPT-4, need dedicated and optimized hardware, and running them remotely is usually faster.
  • Pipeline configuration
    Some components or their settings can slow your pipeline. For example, a Ranker improves the results but also slows the system down. The same goes for a Reader with a high top_k value. It's often about finding the balance between speed and performance.
  • The length of generated responses
    For generative QA pipelines, the number of tokens you want the model to generate as an answer influences your system's speed. The longer the answer, the slower the system.

Optimizing the speed of your pipeline is all about finding the balance among these factors. It involves choosing the right-sized model, running it where it's fastest, and finding a configuration with optimal performance.

If your pipeline contains components that work better on a GPU, turn on GPU acceleration. For details, see Enable GPU Acceleration for Pipelines.

Pipeline Analytics​

To check what queries were asked, the top answers, and how long it took to find them, click the pipeline's name on the Pipelines page and switch to Analytics. This brings you to the pipeline overview, where you can view all the information about your pipeline.

Overview​

The Pipeline Overview shows a KPI dashboard with key performance metrics in an easy-to-read format:

  • Total queries: Total number of queries your pipeline processed. This metric helps you understand usage patterns and traffic volume.
  • Documents: Total number of documents indexed for this pipeline.
  • Feedback coverage: Percentage of queries that received user feedback. Higher feedback coverage gives you better insights into pipeline performance. If this shows "N/A", it means no feedback was given yet.
  • Average response time: Average time it took to generate a response. Monitor this metric to identify performance issues or improvements after configuration changes.
  • Minimum inference time: The fastest response time recorded across all queries. This helps you understand your pipeline's best case performance.
  • Maximum inference time: The slowest response time recorded across all queries. Use this metric to identify potential performance bottlenecks or outliers.
  • Query volume: Number of queries your pipeline processed over the last 7 days.
  • Feedback distribution: Distribution of feedback received for the pipeline's responses. You can click on any slice of the pie chart to navigate to the Search History tab with the feedback rating filter automatically applied for that feedback category (Accurate, Inaccurate, or Wrong Document). This lets you quickly drill down into specific feedback types for detailed analysis.
  • Query Activity Heatmap: Shows when your pipeline is most active by displaying query volume across hours and days. This helps you understand usage patterns, identify peak activity period, or plan maintenance windows.

History​

For detailed performance and feedback analysis, use the History tab to review individual queries and responses. You can generate executive summaries with AI-powered Feedback Insights. You can start with one of the suggested prompts or enter your own query. Based on the prompt, AI analyzes your pipeline activity and generates a summary.

Use Smart Filters to check if your pipeline is following patterns worth investigating. Smart filters show the analysis of the last 100 queries to surface any anomalies or patterns that might require attention.

Customize the table to show the columns you need, and view the conversation history for each query. You can choose the pipeline version whose history you want to view. Use notes and hashtags to add context to your queries and responses. Both notes and hashtags are searchable. You can view and manage them if you enable the Notes and Hashtags columns in the Search History table. To do this:

  1. Click Manage table preferences.
  2. Choose to display the Note and Hashtags columns. You can then add notes and hashtags to your queries using these columns in the table.

You can now see notes and hashtags for each query in the History table. You can also view and manage them in the detailed view for each query.

The History table includes a Status column that shows whether each run succeeded or failed. Users can submit feedback for successful runs only. Runs that ended with an error have a failed status and cannot receive new ratings. Any feedback already recorded for a run before it failed continues to display.

Use search history to identify slow queries, improve answer quality, and understand user needs. Download the search history as a CSV file for further analysis.

The exported CSV file includes the following columns:

  • query_id: The unique identifier for each search history record. Use this column to reference a specific query in your integration or downstream tooling.
  • answer: The answer text returned for the query. For retrieval and agent pipelines, this is populated from the first retrieved document's content when no explicit answer is present.
  • file_name: The name of the source file associated with the result. For retrieval and agent pipelines, this falls back to the file from the first retrieved document.
  • file_id: The unique identifier of the source file. Resolved the same way as file_name.
  • created_at: The timestamp of when the query was made, in ISO 8601 format. If your integration uses the legacy column name time, it continues to work and returns the same value.
  • feedback_created_at: The timestamp of when user feedback was submitted for the query, in ISO 8601 format. This column is empty when no feedback has been given for a query.
  • pipeline_version_id: The ID of the pipeline version that served the query. This lets you tie each query row to the exact pipeline version that served it, which is useful when you have multiple versions deployed and want to compare behavior or investigate a specific version.
info

The Status column is not included in the CSV export.

When you navigate to Search History from the Feedback Distribution chart, the feedback rating filter is automatically applied to show only queries matching the selected feedback type. You can clear this filter or add additional filters as needed.

Use filters to narrow down the results shown in the history table. Any filters you apply are automatically synced with the URL, so you can easily share the filtered view with others by just sharing the URL.

Logs​

Check pipeline logs to see what happened since it was deployed. You can view the logs on the Pipeline Details page (just click the pipeline name to get there). Expand messages for more details and possible actions.

Traces​

Haystack Platform supports tracing with Langfuse and Weights & Biases' Weave. To learn how to set up tracing, see Trace Your Pipelines.