Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

AmazonS3VectorsDocumentStore

A document store that keeps Haystack documents as vectors, with their metadata, in an Amazon S3 Vectors vector bucket and index. Use it with AmazonS3VectorsEmbeddingRetriever to run similarity search on serverless vector storage in Amazon S3.

Configuration​

Add Workspace-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Workspace>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in the current workspace.

Add Organization-Level Integration​

  1. Click your profile icon and choose Settings.
  2. Go to Organization>Integrations.
  3. Find the provider you want to connect and click Connect next to them.
  4. Enter the API key and any other required details.
  5. Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.

Set vector_bucket_name, index_name, and dimension. By default, the store creates the bucket and index if they don't exist, so the AWS credentials need create permissions, or set create_bucket_and_index to False. Pass credentials with the AWS credential parameters. If you don't set them, the store uses the default AWS credential chain.

Usage Example​

AmazonS3VectorsDocumentStore:
type: haystack_integrations.document_stores.amazon_s3_vectors.document_store.AmazonS3VectorsDocumentStore
init_parameters:
vector_bucket_name: my-vector-bucket
index_name: my-index
dimension: 1024
distance_metric: cosine

Service Limits​

  • Maximum top_k is 10,000 results per query. Search is approximate, so a top_k close to the number of stored or matching documents can return fewer results.
  • Maximum vector dimension is 4,096.
  • Metadata per vector is 40 KB in total, and 2 KB of it can be filterable.
  • A metadata value must be a string, a boolean, a finite number, or a non-empty array of only strings or only non-boolean numbers. The store drops other values with a warning instead of failing the write.
  • Vectors are float32 only. Documents written without an embedding get a placeholder vector and are only meaningfully retrievable through filter_documents().
  • Only the OVERWRITE duplicate policy is supported, because writes are upserts.
  • The distance metric is cosine or euclidean. You set it when you create the index, and you can't change it later.
  • filter_documents() filters locally. To filter large stores, use AmazonS3VectorsEmbeddingRetriever with filters.
  • Document.blob isn't stored.

Parameters​

Init Parameters​

ParameterTypeDefaultDescription
vector_bucket_namestrName of the S3 vector bucket. Required.
index_namestrName of the vector index within the bucket. Required.
dimensionintDimensionality of the embeddings when creating an index, for example 768 or 1536. When you connect to an existing index, its actual dimension is used. Required.
distance_metricstrcosineDistance metric for similarity search: cosine or euclidean.
region_nameOptional[str]NoneAWS region. If not set, the store uses the default from the environment or configuration.
aws_access_key_idOptional[Secret]NoneAWS access key ID. If not set, the store uses the default credential chain.
aws_secret_access_keyOptional[Secret]NoneAWS secret access key.
aws_session_tokenOptional[Secret]NoneAWS session token for temporary credentials.
create_bucket_and_indexboolTrueWhether to create the vector bucket and index if they don't exist.
non_filterable_metadata_keysOptional[List[str]]NoneExtra metadata keys to mark as non-filterable on the index, beyond the internal key for document content.