PineconeEmbeddingRetriever

Retrieve documents from the PineconeDocumentStore based on their dense embeddings. It compares the query and document embeddings and fetches the documents most relevant to the query based on vector similarity.

Pinecone is a managed vector database service that enables fast and scalable similarity search. It's designed for production workloads with features like automatic scaling, high availability, and real-time updates.

Key Features

• Embedding-based retrieval from a Pinecone vector database.
• Configurable number of results with top_k.
• Supports metadata filtering to narrow down the search space.
• Configurable filter policy (replace or merge) for runtime filters.
• Works with any text embedder that produces dense vector embeddings.

Configuration

1. Drag the PineconeEmbeddingRetriever component onto the canvas from the Component Library.
2. Click on the component to open the configuration panel.
3. On the General tab:
   • Configure the PineconeDocumentStore with your Pinecone index, namespace, and embedding dimensions. Make sure you have a secret called PINECONE_API_KEY in your workspace. For instructions, see Add Secrets.
   • Set top_k to control the maximum number of documents to retrieve.
4. Go to the Advanced tab to configure filter_policy and default filters.

Connections

PineconeEmbeddingRetriever receives the query embedding from a text embedder like SentenceTransformersTextEmbedder or OpenAITextEmbedder. It sends retrieved documents to PromptBuilder for use in a prompt, or to a Ranker to reorder them by relevance.

Source Code

To check this component's source code, open embedding_retriever.py in the Haystack Core Integrations repository.

Usage Examples

Basic Configuration

  PineconeEmbeddingRetriever:
    type: haystack_integrations.components.retrievers.pinecone.embedding_retriever.PineconeEmbeddingRetriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.pinecone.document_store.PineconeDocumentStore
        init_parameters:
          api_key:
            type: env_var
            env_vars:
            - PINECONE_API_KEY
            strict: true
          index: my-index
          namespace: my-namespace
          dimension: 384
          metric: cosine
      top_k: 10
      filter_policy: replace

This is an example of a semantic search pipeline where PineconeEmbeddingRetriever receives the query embedding from a text embedder and retrieves matching documents.

components:
  text_embedder:
    type: haystack.components.embedders.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
    init_parameters:
      model: sentence-transformers/all-MiniLM-L6-v2
      device:
      token:
      prefix: ''
      suffix: ''
      batch_size: 32
      progress_bar: true
      normalize_embeddings: false
      trust_remote_code: false
  PineconeEmbeddingRetriever:
    type: haystack_integrations.components.retrievers.pinecone.embedding_retriever.PineconeEmbeddingRetriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.pinecone.document_store.PineconeDocumentStore
        init_parameters:
          api_key:
            type: env_var
            env_vars:
            - PINECONE_API_KEY
            strict: true
          index: my-index
          namespace: my-namespace
          dimension: 384
          metric: cosine
          spec:
      filters:
      top_k: 10
      filter_policy: replace

connections:
- sender: text_embedder.embedding
  receiver: PineconeEmbeddingRetriever.query_embedding

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - text_embedder.text
  filters:
  - PineconeEmbeddingRetriever.filters

outputs:
  documents: PineconeEmbeddingRetriever.documents

This example shows a RAG pipeline that uses PineconeEmbeddingRetriever to find relevant documents, then passes them to a generator to answer a question.

components:
  text_embedder:
    type: haystack.components.embedders.sentence_transformers_text_embedder.SentenceTransformersTextEmbedder
    init_parameters:
      model: sentence-transformers/all-MiniLM-L6-v2
      device:
      token:
      prefix: ''
      suffix: ''
      batch_size: 32
      progress_bar: true
      normalize_embeddings: false
      trust_remote_code: false
  retriever:
    type: haystack_integrations.components.retrievers.pinecone.embedding_retriever.PineconeEmbeddingRetriever
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.pinecone.document_store.PineconeDocumentStore
        init_parameters:
          api_key:
            type: env_var
            env_vars:
            - PINECONE_API_KEY
            strict: true
          index: my-index
          namespace: my-namespace
          dimension: 384
          metric: cosine
          spec:
      filters:
      top_k: 10
      filter_policy: replace
  prompt_builder:
    type: haystack.components.builders.prompt_builder.PromptBuilder
    init_parameters:
      required_variables: "*"
      template: |-
        Given the following documents, answer the question.

        Documents:
        {% for document in documents %}
        {{ document.content }}
        {% endfor %}

        Question: {{ question }}
        Answer:
  generator:
    type: haystack.components.generators.openai.OpenAIGenerator
    init_parameters:
      api_key:
        type: env_var
        env_vars:
        - OPENAI_API_KEY
        strict: true
      model: gpt-4o-mini
      generation_kwargs:
  answer_builder:
    type: deepset_cloud_custom_nodes.augmenters.deepset_answer_builder.DeepsetAnswerBuilder
    init_parameters:
      reference_pattern: acm

connections:
- sender: text_embedder.embedding
  receiver: retriever.query_embedding
- sender: retriever.documents
  receiver: prompt_builder.documents
- sender: prompt_builder.prompt
  receiver: generator.prompt
- sender: generator.replies
  receiver: answer_builder.replies
- sender: retriever.documents
  receiver: answer_builder.documents
- sender: prompt_builder.prompt
  receiver: answer_builder.prompt

max_runs_per_component: 100

metadata: {}

inputs:
  query:
  - text_embedder.text
  - prompt_builder.question
  - answer_builder.query
  filters:
  - retriever.filters

outputs:
  documents: retriever.documents
  answers: answer_builder.answers

Parameters

Inputs

Parameter	Type	Description
query_embedding	List[float]	Embedding of the query.
filters	Optional[Dict[str, Any]]	Filters applied to the retrieved Documents. The way runtime filters are applied depends on the filter_policy chosen at retriever initialization.
top_k	Optional[int]	Maximum number of Documents to return.

Outputs

Parameter	Type	Description
documents	List[Document]	List of documents similar to the query embedding.

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
document_store	PineconeDocumentStore		The Pinecone Document Store.
filters	Optional[Dict[str, Any]]	None	Filters applied to the retrieved Documents.
top_k	int	10	Maximum number of Documents to return.
filter_policy	Union[str, FilterPolicy]	FilterPolicy.REPLACE	Policy to determine how filters are applied.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
query_embedding	List[float]		Embedding of the query.
filters	Optional[Dict[str, Any]]	None	Filters applied to the retrieved Documents. The way runtime filters are applied depends on the filter_policy chosen at retriever initialization. See init method docstring for more details.
top_k	Optional[int]	None	Maximum number of Documents to return.

Related Information

• Add Secrets