I’m trying to connect from Lambda to S3 within a VPC. Sometimes it works, sometimes it doesn’t. An S3 endpoint is made within the VPC and it’s working correctly as it sometimes demonstrates.
The weird thing is that if I run a Lambda and let it timeout, the next one will run most of the times. So it can be cold start, but that would be weird as it times out no matter the memory + runtime of the Lambda. The full process looks like this:
-
Glue job normalizes data and writes partitions to S3
-
S3 triggers Lambda to read the key of the partition and pushes it to SQS to have more control of the workflow
-
SQS triggers a Lambda that reads the partition and ingests it into OpenSearch
Point 3 is where it goes wrong. Sometimes it works, sometimes it doesn’t.
Anyone has any clue?
I want a stable connection between S3 and Lambda.
Ruben is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
1
This behaviour often happens when an AWS Lambda function has been configured to use both Public & Private subnets. It likely works on the Private subnets, but not the Public subnets.
You should modify the Lambda network configuration to only use Private subnets.