AmazonS3VectorsDocumentStore
A document store that keeps Haystack documents as vectors, with their metadata, in an Amazon S3 Vectors vector bucket and index. Use it with AmazonS3VectorsEmbeddingRetriever to run similarity search on serverless vector storage in Amazon S3.
Configuration
Add Workspace-Level Integration
- Click your profile icon and choose Settings.
- Go to Workspace>Integrations.
- Find the provider you want to connect and click Connect next to them.
- Enter the API key and any other required details.
- Click Connect. You can use this integration in pipelines and indexes in the current workspace.
Add Organization-Level Integration
- Click your profile icon and choose Settings.
- Go to Organization>Integrations.
- Find the provider you want to connect and click Connect next to them.
- Enter the API key and any other required details.
- Click Connect. You can use this integration in pipelines and indexes in all workspaces in the current organization.
Set vector_bucket_name, index_name, and dimension. By default, the store creates the bucket and index if they don't exist, so the AWS credentials need create permissions, or set create_bucket_and_index to False. Pass credentials with the AWS credential parameters. If you don't set them, the store uses the default AWS credential chain.
Usage Example
AmazonS3VectorsDocumentStore:
type: haystack_integrations.document_stores.amazon_s3_vectors.document_store.AmazonS3VectorsDocumentStore
init_parameters:
vector_bucket_name: my-vector-bucket
index_name: my-index
dimension: 1024
distance_metric: cosine
Service Limits
- Maximum
top_kis 10,000 results per query. Search is approximate, so atop_kclose to the number of stored or matching documents can return fewer results. - Maximum vector dimension is 4,096.
- Metadata per vector is 40 KB in total, and 2 KB of it can be filterable.
- A metadata value must be a string, a boolean, a finite number, or a non-empty array of only strings or only non-boolean numbers. The store drops other values with a warning instead of failing the write.
- Vectors are
float32only. Documents written without an embedding get a placeholder vector and are only meaningfully retrievable throughfilter_documents(). - Only the
OVERWRITEduplicate policy is supported, because writes are upserts. - The distance metric is
cosineoreuclidean. You set it when you create the index, and you can't change it later. filter_documents()filters locally. To filter large stores, useAmazonS3VectorsEmbeddingRetrieverwith filters.Document.blobisn't stored.
Parameters
Init Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
vector_bucket_name | str | Name of the S3 vector bucket. Required. | |
index_name | str | Name of the vector index within the bucket. Required. | |
dimension | int | Dimensionality of the embeddings when creating an index, for example 768 or 1536. When you connect to an existing index, its actual dimension is used. Required. | |
distance_metric | str | cosine | Distance metric for similarity search: cosine or euclidean. |
region_name | Optional[str] | None | AWS region. If not set, the store uses the default from the environment or configuration. |
aws_access_key_id | Optional[Secret] | None | AWS access key ID. If not set, the store uses the default credential chain. |
aws_secret_access_key | Optional[Secret] | None | AWS secret access key. |
aws_session_token | Optional[Secret] | None | AWS session token for temporary credentials. |
create_bucket_and_index | bool | True | Whether to create the vector bucket and index if they don't exist. |
non_filterable_metadata_keys | Optional[List[str]] | None | Extra metadata keys to mark as non-filterable on the index, beyond the internal key for document content. |
Related Information
Was this page helpful?