Navigating the Usage Dashboard

All your usage information is visible on the Usage dashboard, which you can access from your profile. Click your initials in the top right corner and choose Usage.

The top of the dashboard shows a summary of all the information. To see how the data breaks down by pipeline, check the table at the bottom of the dashboard.

1. Your Organization

Your organization name and timeframe for the usage data.

2. Credits

Credits are units that represent the number of hours your pipelines were deployed or your indexes were processing files. This includes pipelines that were deleted but contributed to credits usage during this usage cycle.

One credit is one hour of your pipelines being deployed or your indexes processing files.

You consume credits when:

• Your pipeline is deployed and active.
• Your index is actively processing files.

No credits are consumed when:

• Your pipeline is deployed but inactive, on standby.
• Your index is enabled but is not indexing any files as indexing is complete. Note that every time you upload a file, the index processes it, which counts toward credit consumption.

3. Document Storage Units

Document Storage Units (DSU) show how much vector storage you're using for your indexed documents. The DSU count on the Usage page reflect the current state. They can go up or down as you add or remove documents.

When you upload and index your files, we split them into smaller pieces called documents. These documents are stored and your DSU count depends on two key factors:

• How many documents your files are split into.
• The size (dimensionality) of the embedding model used to process them.

The more documents or the larger the model, the more storage units you use.

Calculating DSUs

To calculate DSUs, we multiply the number of stored documents by a multiplier that reflects the embedding model's dimensionality:

• A standard embedding model with 768 dimensions uses a multiplier of 1.
• A larger model with 1,536 dimensions uses a multiplier of 2, meaning it needs twice as much storage.
• If you use keyword-based search or don't use vector storage, the multiplier is just 0.2, so it uses much less storage.

Example
Say you upload 5,000 files which are split into 83,697 documents:

• With the standard embedding model (768 dimensions): 83,697 documents × 1 = 83,697 storage units
• With a larger model (1,536 dimensions): 83,697 documents × 2 = 167,394 storage units
• With keyword search only or no vector storage: 83,697 documents × 0.2 = 16,739.4 storage units

Usage Per Pipeline or Index

At the bottom of the page, you can check the detailed usage per pipeline or index:

You can check the number of credits used and the workspace it's in.

Managing Your Storage

Predicting Your Storage Needs

The number of documents you get from your files depends on how you split them during indexing and the embedding dimensions of the model you use with your Embedder to obtain vector representations of your files. The formula for calculating document storage units is: number of documents * multiplier. The vector dimension of the embedding model determines the multiplier. A dimension of 768 equates to a multiplier of 1. A dimension of 1024 equates to a multiplier of 1.3.

Let's assume you use one of the recommended embedding models, like intfloat/e5-base-v2, which has a vector dimension of 768 (equal to a multiplier of 1) and default DocumentSplitter's settings, which are:

splitter:
      type: haystack.components.preprocessors.document_splitter.DocumentSplitter
      init_parameters:
        split_by: word
        split_length: 200

This configuration splits files by word, resulting in documents of 200 words each. To calculate the anticipated number of documents, divide the total word count of your files by 200.

For example, with 5,000 files, each containing 1000 words, you have a total of 5 million words. Divide this total by 200 (the default split length), and you get 25,000 documents. If you use the intfloat/e5-base-v2 model, you must now multiply the document count by 1. You obtain 25,000 document storage units.

Optimizing Storage Usage

As you near your storage capacity, you can reduce storage consumption by:

• Deleting duplicate files.
• Reducing irrelevant files, starting from the largest ones. Detecting irrelevant files is specific to your use case. You can use the Search History endpoint to check user queries, files, and documents used.
• Adjusting settings in DocumentSplitter:
  • If your split_overlap setting is high, consider decreasing it.
  • If your split_lengthIf the setting is low, try increasing it. Remember that it's important to avoid too large values to prevent Retriever and Generator nodes from cutting the documents.