Debug with Logs
Logs are enabled for every Haystack Platform pipeline and index by default.
Viewing Pipeline Logs
- In Haystack Enterprise Platform, go to Pipelines and click the name of the pipeline you want to debug. The pipeline's Analytics page opens.
- Switch to Analytics and open the Logs tab.
Viewing Index Logs
-
In Haystack Enterprise Platform, go to Indexes and click the name of the index you want to debug.
-
Switch to Logs.
- To view pipeline logs, choose Pipelines.
- To view index logs, choose Indexes.
-
Click the name of your pipeline or index to open its details.
-
Open the Logs tab.
You can filter the logs by message and pipeline type or date added. You can also search for keywords in log messages.
To download the logs, click Download CSV.
After you run a pipeline in Builder, you can see its real-time logs right there.
You can also open the trace for a query directly from Playground: click the View trace (bug) icon below the answer. For details, see Test Your Pipeline in Playground. To debug while building, use the logs in Builder. To learn how to do it, see Debug with Pipeline Builder.
To check logs for a specific trace, go to Analytics > Traces and open a trace. Then, open the span for the component you want to investigate and click View Logs. For details, see Trace Your Pipelines in the UI.
View Logs for a Specific Trace
Traces are collected for queries run on or after 25 June 2026. Before that date, traces are not available. To collect traces for pipelines deployed before that date, undeploy the pipeline and deploy it again.
You can check log entries generated during a specific pipeline run:
- In Haystack Enterprise Platform, go to Pipelines and click Analytics next to the pipeline you want to debug.
- Go to the Logs tab to view all logs generated during the trace.
Check Index Hardware Metrics
If an index runs slowly or fails, check its hardware metrics to see whether it's running out of CPU or memory.
- In Haystack Enterprise Platform, go to Indexes and click the name of the index you want to check.
- Click Metrics.
The Metrics tab shows CPU usage, memory usage, and replica count charts, grouped by pod. Use them together with the index logs to find the cause of a problem. If the index runs out of CPU or memory, give it a larger indexing pod. For details, see Set the Indexing Instance Size in Index Settings.
To confirm that indexing is running and to see how many batches the index processed over time, check the Executions chart on the same tab. Each batch of files that an index processes counts as one execution, including batches on indexes that don't pin a Haystack version. Failed batches count as errors, not as executions.
Read the Indexing Throughput Averages
To find out why indexing is slow, check how much work each indexing batch does and how long it takes. Below the hardware charts, three tiles show one average each for the time range you selected:
- Files per indexing batch: The average number of files in a batch. A low average means the index processes files in small batches.
- Processing time per batch: The average time the index takes to process a batch.
- Processing time per file: The average time the index takes to process one file.
Each tile is the mean over the whole time range, not a trend. Large batches weigh more than small ones, because the tiles divide total processing time by the total number of batches or files.
Compare the averages with the CPU and memory charts to see whether slow indexing comes from large batches or from slow processing of each file.
To see how fast the index works through your files, check the Indexing throughput chart. It shows files indexed per minute, summed over all pods. The unit stays the same for every time range, so you can compare a one-hour range with a one-week range. A drop in throughput that coincides with high CPU or memory usage can point to an undersized pod. A drop with no resource pressure can point to slow processing of each file or to small batches. Throughput also drops when fewer files are waiting to be indexed.
If memory or CPU usage is close to the limit, give the index a larger pod size. See Set the Indexing Instance Size in Index Settings.
If the index didn't process any files in the selected time range, the tiles show an em dash (—) instead of a value and the throughput chart is empty. Select a longer time range or check the index logs to see whether indexing ran.
If batches take too long, lower the maximum batch size. For steps, see Limit the Batch Size of an Index.
To tell whether an index is close to its memory limit, compare its memory usage with the limit for its pod size:
- Small: 4 GiB
- Medium: 8 GiB
- Large: 16 GiB
- XL: 32 GiB
An index without a pod size uses the default of 8 GiB. If the index has a memory limit set explicitly, that limit applies instead of the size. The CPU limit is only reported when the index has an explicit CPU limit. If memory usage stays close to the limit, set a larger pod size. For steps, see Set the Indexing Instance Size in Index Settings.
Read the Limit Lines and Size Advice
To see whether an index's pod size fits its workload, compare its usage to the pod size limit. When your organization uses indexing pod sizes, the CPU and memory charts show a dashed line at the limit of the index's pod size, and usage as a percentage of that limit. The chart caption shows the limit value. If the index has no CPU limit, the CPU chart shows raw usage in cores and no limit line. The memory chart caption also shows how close memory usage came to the limit.
Above the charts, an advice card judges your index's hardware and gives a recommendation in three areas:
- Cost: What the index cost for the selected time range.
- Fit: Whether the pod size matches the usage. For example, a Fit line reading Right size means usage stays under the limit. A Fit line reading Close to its limit means the index is close to running out of memory and needs a bigger size. If the index has no CPU limit, the card shows Cost and Scaling advice but no Fit line.
- Scaling: Whether the number of replicas fits the load. Indexes have a maximum number of replicas but no minimum, so the card judges only against the maximum.
If the Fit advice recommends a bigger size, click Change size to open the index settings, where you can choose a different size. For steps, see Set the Indexing Instance Size in Index Settings.
Read the Replicas Chart
To find out when an index was running, for example to check whether it scaled to zero between runs, read the Replicas chart. It shows one band per replica, so you can see when each replica was running. An index that's scaled to zero between runs shows empty space between its bands, and the y axis shows only 0 and 1 for a single replica. For example, an index that ran for one minute at 4:21 p.m. and one minute at 6:36 p.m. shows two short bands with nothing between them. Hover over the chart to see which replicas were running at that moment and how many there were. The legend lists the same replicas. Each replica keeps the same color in the CPU usage, memory usage, and Replicas charts, so you can match a replica's CPU and memory usage to the time it was running.
Related Information
Was this page helpful?