More Document Stores

Document store is where your query pipelines access data. The indexing pipeline converts your files into documents and stores them in a document store of your choice. Until now, Deepset Cloud has supported the OpenSearch document store, but the list has grown. Starting with this release, you can use:

• Elasticsearch
• Pinecone
• Qdrant
• Weaviate

Each document store comes with dedicated Retrievers to make retrieval make the best use of the document store's technology.

Check Document Stores for more details.

Sample Datasets

Want to try a pipeline quickly but have no data? Use the sample files we prepared on healthcare, law, and tech and finance. They're available on the Files page:

Export Your Pipelines as Python

Save your pipelines locally as Python files. Pipelines that include deepset Cloud custom components (like our templates) may need additional adjustments and won't work out of the box after export. Export is available in Pipeline Builder - open any pipeline for editing and you'll find it there.

Support For More Models and Integrations

💡

Secrets

Create secrets to securely manage connections to model providers or integrations. For more information, see Add Secrets to Connect to Third Party Providers.

We've added support for the following model providers:

• Fastembed: Models for generating embeddings and ranking documents. We've added the following components through which you can use Fastembed models:
  • Embedders:
    • FastembedTextEmbedder
    • FastembedDocumentEmbedder
    • FastembedSparseTextEmbedder
    • FastembedSparseDocumentEmbedder
  • Rankers:
    • FastembedRanker
• Google Vertex AI available through the following Generators:
  • VertexAIImageQA
  • VertexAICodeGenerator
  • VertexAITextGenerator
  • VertexAIImageCaptioner
  • VertexAIImageGenerator
  • VertexAIGeminiGenerator
  • VertexAIGeminiChatGenerator
• Jina AI embedding and ranking models available through:
  • Embedders:
    • JinaTextEmbedder
    • JinaDocumentEmbedder
  • Rankers:
    • JinaRanker
• Llama.cpp library for efficient inference of large language models, available through:
  • Generators:
    • LlamaCppGenerator
    • LlamaCppChatGenerator
• Mistral AI models available through an API with a pay-as-you-go access to the latest Mistral models. You can use it through:
  • Embedders:
    • MistralTextEmbedder
    • MistralDocumentEmbedder
  • Generators:
    • MistralChatGenerator
• Ollama models available through:
  • Embedders:
    • OllamaTextEmbedder
    • OllamaDocumentEmbedder
  • Generators:
    • OllamaGenerator
    • OllamaChatGenerator

And the following integrations:

• Langfuse available through LangfuseConnector
• Ragas available through RagasEvaluator

Naming Changes

To make things clearer and better reflect what they’re for, we’ve updated a couple of names:

• Studio is now Pipeline Builder
• Prompt Studio is now Prompt Explorer

The names have changed, but the functionality remains the same.

Indexing Pipelines No Longer Required

The range of scenarios deepset Cloud covers is growing, so we're adjusting to make workflows smooth and convenient. That's why indexing pipelines are no longer mandatory. You can have a pipeline with just the query part, and that's fine. This should make use cases like summarization or querying data in an external database, like Snowflake, smoother.

For other use cases where you query data in one of the document stores integrated with deepset Cloud, we still recommend using an indexing pipeline to prepare your data.

Labelling and V1 Templates Are Going Away

Having analyzed the data, we've decided to deprecate the Labelling feature. You can use Feedback to label your document search pipelines.

We're also deprecating version 1 templates as we prepare to fully switch to version 2. Your version 1 pipelines will continue to work, but you won't be able to create them.