Basic Information

• Pipeline type: Used in indexing and pipelines for connecting to a remote database.
• Type: deepset_cloud_custom_nodes.auth.cloud_sql_auth.CloudSQLAuthProxy
• Components it can connect with:
  • CloudSQLAuthProxy doesn't connect to any other components. Add it directly to your pipeline to connect deepset Cloud with your GCP-hosted Cloud SQL for PostgreSQL database.

Inputs

This component doesn't require any inputs as it doesn't connect to any other component in the pipeline. Its sole task is to connect to a remote database

Outputs

Overview

deepset Cloud can connect to your Cloud SQL for PostgreSQL with the pgvector extension.

Cloud SQL for PostgreSQL is a fully managed relational database service from Google Cloud. It streamlines database management by handling backups, updates, and scaling so you can focus on your applications. For details, see Google Cloud documentation.

The pgvector extension enables vector similarity search for Postgres. For details, see the pgvector GitHub repository. In deepset Cloud, it's represented as the PgvectorDocumentStore where your pipelines can access your data.

To enable deepset Cloud pipelines to query data in your Cloud SQL for PostgreSQL database, you connect deepset Cloud to your database using the CloudSQLAuthProxy component. When included in your pipeline, CloudSQLAuthProxy creates a secure connection to your Google Cloud SQL database. You only need to add it to the pipeline—no additional connections to other components are required, as its sole role is establishing the database connection.

You then use a Pgvector retriever that accesses the PgvectorDocumentStore and fetches the relevant documents.

CloudSQLAuthProxy downloads the Cloud SQL for PostgreSQL proxy binary and saves it to a local file. It then uses it to open a server port to connect to the database. The default binary is for Linux. You can specify both the binary's URL and the filename for storage. Afterward, it uses the credentials you provide to initialize the connection. By default, it reads the credentials from the CLOUD_SQL_CREDENTIALS environment variable.

Limitations

The following features don't work in this setup, as its an external database and deepset Cloud doesn't have access to this information:

• Pipeline indexing status. The pipeline will show as partially indexed.
• The number of indexed documents. They'll show as skipped.
• Automatic index creation or deletion when deploying or undeploying pipelines. You'll need to manage your index in Cloud SQL. For guidance, refer to Cloud SQL documentation.

Usage Example

This is an example of an indexing pipeline with CloudSQLAuthProxy at the beginning of the pipeline. As you can see, it's not listed in the connections section as it doesn't require to be connected to other components.

components:
  cloud_sql:
    type: deepset_cloud_custom_nodes.auth.cloud_sql_auth.CloudSQLAuthProxy
    init_parameters:
      url: ""https://storage.googleapis.com/cloud-sql-connectors/cloud-sql-proxy/v2.13.0/cloud-sql-proxy.darwin.arm64"
      instance_connection_name: ""careful-time-421813:us-central1:myinstance"
      # we store json_credentials in the CLOUD_SQL_DATABASE env variable from which they're read by default, so we don't specify them here
      
  FileTypeRouter:
    type: haystack.components.routers.file_type_router.FileTypeRouter
    init_parameters:
      mime_types:
        - text/plain
        - text/markdown
  TextFileToDocument:
    type: haystack.components.converters.txt.TextFileToDocument
    init_parameters:
      encoding: utf-8
  MarkdownToDocument:
    type: haystack.components.converters.markdown.MarkdownToDocument
    init_parameters:
      table_to_single_line: false
      progress_bar: true
  DocumentJoiner:
    type: haystack.components.joiners.document_joiner.DocumentJoiner
    init_parameters:
      join_mode: concatenate
      weights: null
      top_k: null
      sort_by_score: true
  DocumentSplitter:
    type: haystack.components.preprocessors.document_splitter.DocumentSplitter
    init_parameters:
      split_by: word
      split_length: 200
      split_overlap: 0
      split_threshold: 0
      splitting_function: null
  DocumentWriter:
    type: haystack.components.writers.document_writer.DocumentWriter
    init_parameters:
      document_store:
        type: haystack_integrations.document_stores.pgvector.document_store.PgvectorDocumentStore
        init_parameters:
          table_name: deepset_test
          embedding_dimension: 768
          vector_function: cosine_similarity
          recreate_table: True,
          search_strategy: hnsw
      policy: NONE
connections:
  - sender: FileTypeRouter.text/plain
    receiver: TextFileToDocument.sources
  - sender: FileTypeRouter.text/markdown
    receiver: MarkdownToDocument.sources
  - sender: TextFileToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: MarkdownToDocument.documents
    receiver: DocumentJoiner.documents
  - sender: DocumentJoiner.documents
    receiver: DocumentSplitter.documents
  - sender: DocumentSplitter.documents
    receiver: DocumentWriter.documents
max_loops_allowed: 100
metadata: {}
inputs:
  files:
    - FileTypeRouter.sources

Init Parameters