I am trying to create a connection between AWS Glue and the Redshift database in Glue. Currently, I am getting an error:
Connection creation is failed.
Create connection failed during validating credentials. Please validate connection inputs and VPC connectivity to Security Token Service, Secrets Manager and REDSHIFT.
IAM Role used during connection creation has these permissions:
- AdministratorAccess
- AmazonRedshiftFullAccess
- AmazonS3FullAccess
- AmazonVPCFullAccess
- AWSGlueConsoleFullAccess
- AWSGlueServiceRole
- CloudWatchFullAccess
- CloudWatchFullAccessV2
IAM role trusted entities:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"glue.amazonaws.com",
"redshift-serverless.amazonaws.com",
"redshift.amazonaws.com",
"lambda.amazonaws.com",
"secretsmanager.amazonaws.com",
"s3.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
The Security Group attached to RedShift Serverless has these out and in rules:
I think Glue in the west region can connect to any region. In my case, my RedShift Serverless was created in us-east-1.
Thank you
After getting some help from AWS, I was able to create a connection. Here is what was recommended for the above setup. Add SecretsManagerReadWrite to IAM role.
Add the following VPC endpoints to the VPC and subnet where your Redshift cluster is configured:
- S3 (Gateway type)
- STS (Interface type)
- Secrets Manager (Interface type)
- Redshift (Interface type)
For me to make this work, I had to:
Configure the IAM Role used for the connection to include following policies:
- AmazonRedshiftFullAccess
- AmazonS3FullAccess
- AmazonVPCFullAccess
- AWSGlueConsoleFullAccess
- AWSKeyManagementServicePowerUser
- SecretsManagerReadWrite
Add VPC endpoints for the following services (with the subnet used in the GLUE connection toggled on – find this by creating a errornous connection in Glue, and you can see it there – dno if you can find it other places)
- com.amazonaws.eu-central-1.redshift
- com.amazonaws.eu-central-1.sts
- com.amazonaws.eu-central-1.secretsmanager
Configure the Amazon Redshift Inbound and Outbound rules according to: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-redshift-home.html – the Set up Amazon VPC section.
Since I’ve played around with this for almost the entire day, I might have toggled other stuff too, but at least I needed these settings 🙂
And for the life of me, I don’t get why permissions in cloud providers have to be this complicated – I get they need to stay secure, but I have probably today introduced multiple vulnerabilities in my cloud setup, just playing around with stuff I had no clue about what actually did, just trying to get this connection to work.
1