Skip to main content

Connect to an External Document Store

Run your query pipelines on data stored in an external database, such as Pinecone, Weaviate, Qdrant, or others.

About this Task​

Currently, the deepset AI Platform supports the following document stores:

  • OpenSearch (the core document store; unless you're using your own OpenSearch cluster, you don't need to provide credentials—just use it out of the box)
  • Elasticsearch
  • MongoDB
  • Pinecone
  • Qdrant
  • Snowflake
  • Weaviate

These databases act as document stores in deepset AI Platform. For more information, see Document Stores.

You can also add an integration with any other database through a custom component. For details, see Custom Components.

Prerequisites​

  • You need an active API key to the database you want to use.
  • Basic knowledge of document stores in deepset AI Platform. For details, see Document Stores.
  • Understanding of secrets in deepset AI Platform. For more information, see Add Secrets to Connect to Third Party Providers.
  • Check the parameters you can configure for your document store, especially the name of the parameter for passing the API key. See the documentation for your document store in the Document Stores section.

Run Queries on Data in Your Document Store​

This task involves three steps:

  1. Create a secret to securely store your API key to the document store.
  2. Write your documents to the document store.
  3. Retrieve documents from the document store.

Create a Secret for Your Document Store​

MongoDB

To use MongoDBAtlasDocumentStore, you can simply paste your API key on the Integrations page:

  1. Click your profile icon in the top right corner and choose Connections.
  2. Find MongoDB, click Connect, and paste the API key.

This step is needed to enable a connection with the database you want to use as the document store without adding the API key explicitly in the configuration. You can add a secret only for a single workspace or for the whole organization:

Add Workspace-Level Secret​

  1. Click your profile icon and choose Settings.
  2. Go to Workspace>Secrets.
  3. Click Create Secret.
  4. Enter a secret name that matches the environment variable to store the API key.
  5. Enter the API key and any other required details.
  6. Click Create. You can use this connection in pipelines and indexes in the current workspace.

Add Organization-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Organization>Secrets.
  3. Enter a secret name that matches the environment variable to store the API key.
  4. Enter the API key and any other required details.
  5. Click Create. You can use this connection in pipelines and indexes in all workspaces in the current organization.

You'll then use the secret name as the API key for components that need to connect to the document store.

Write Documents into the Document Store​

DocumentWriter is the component that writers preprocessed documents into the document store. You must add it at the end of your index.

  1. Build your indexing pipeline and add DocumentWriter as its last component.

    Index from a template

    If you're' using a template, DocumentWriter is already there connected to an OpenSearchDocumentStore. You can delete the document store and replace it with another one.

  2. Choose a document store you want to use from the Document Stores group in the Component Library and drag it onto the canvas.

  3. Connect the document store to DocumentWriter and configure it on the component card.

  4. Save your index.

Retrieve Documents From the Document Store​

Each document store has dedicated retrievers that are called the same as the document store. For example, MongoDBAtlasDocumentStore has two compatible retrievers: MongoDBAtlasEmbeddingRetriever and MongoDBAtlasFullTextRetriever.

Add the retrievers to your query pipeline and configure the document store they should connect to in the same way you configured DocumentWriter. Pass the secret's name as the API key for the document store.