CacheChecker

Check whether values are already stored in a document store. You pick one metadata field as the cache key. Values that match existing documents come out as hits. Values with no match come out as misses, so a later component can fetch or index only the new items.

Key Features

• Looks up each incoming value in one metadata field.
• Returns matching documents on hits.
• Returns values that are not in the store on misses.
• Works with the document stores available in Pipeline Builder.

Configuration

1. Drag the CacheChecker component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   • Set Document Store to the store that already holds your documents.
   • Set Cache Field to the metadata field to match, such as url or file_id.

Connections

CacheChecker accepts items, a list of values to look up. It has two outputs:

• hits is a list of documents whose cache field matches at least one item.
• misses is the list of items that matched nothing.

Connect misses to the component that should run only for new items. misses is a plain list. When the next component expects a list of strings, such as LinkContentFetcher for URLs, add an OutputAdapter between them and set its output type to List[str].

Source Code

To check this component's source code, open cache_checker.py in the Haystack repository.

Usage Examples

Basic Configuration

  CacheChecker:
    type: haystack.components.caching.cache_checker.CacheChecker
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          - ${OPENSEARCH_HOST}
          index: documents
      cache_field: url

Using the Component in an Index

This index skips URLs that are already stored and fetches only the misses. OutputAdapter turns the plain misses list into a list of strings that LinkContentFetcher can accept.

# haystack-pipeline
components:
  CacheChecker:
    type: haystack.components.caching.cache_checker.CacheChecker
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.opensearch.document_store.OpenSearchDocumentStore
        init_parameters:
          hosts:
          - ${OPENSEARCH_HOST}
          index: documents
      cache_field: url
  misses_to_urls:
    type: haystack.components.converters.output_adapter.OutputAdapter
    init_parameters:
      template: '{{ misses }}'
      output_type: List[str]
  LinkContentFetcher:
    type: haystack.components.fetchers.link_content.LinkContentFetcher
    init_parameters:
      raise_on_failure: false
      retry_attempts: 2
      timeout: 3

connections:
- sender: CacheChecker.misses
  receiver: misses_to_urls.misses
- sender: misses_to_urls.output
  receiver: LinkContentFetcher.urls

max_runs_per_component: 100

inputs:
  query:
  - CacheChecker.items

metadata: {}

Parameters

Inputs

Parameter	Type	Description
items	List[Any]	Values to compare with the cache field.

Outputs

Parameter	Type	Description
hits	List[Document]	Documents whose cache field matches at least one item.
misses	List	Items that are not present on any document.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
document_store	DocumentStore		Document store to search for existing documents.
cache_field	str		Name of the document metadata field to match against each item.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Description
items	List[Any]	Values to compare with the cache field.

Related Information

• LinkContentFetcher
• OutputAdapter
• Pipeline Components