based on the documentation here – https://aws.amazon.com/blogs/big-data/build-multimodal-search-with-amazon-opensearch-service/, it states that open search can call amazon bedrock to generate embeddings. it uses special ml connector to ingest data.
I have set up an open search serverless collection via terraform , see sample code below . once the open search collection/cluster is created . how can i set up a pipeline , preferably in terraform, to ingest data from s3 and call bedrock service and push it to my open search database, if possible? or do i have to manually or programmatically read data from s3 , call bedrock via api and then do a push to open search .
provider "aws" {
region = local.region
}
module "opensearch_collection_private" {
source = "../../modules/collection"
name = "${local.name}-private"
description = "Example private OpenSearch Serverless collection"
type = "SEARCH"
create_access_policy = true
create_network_policy = true
network_policy = {
AllowFromPublic = false
SourceVPCEs = [
aws_opensearchserverless_vpc_endpoint.example.id
]
}
tags = local.tags
}