Elasticsearch Version
7.17.12

Installed Plugins
repository-s3

Java Version
JAVA_RUNTIME_VERSION=”20.0.2+9-78

OS Version
Linux elasticsearch-data-0 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

environnement
I’m running Elasticsearch on an Amazon EKS cluster, 2 data nodes and a master node

Problem Description
I created an IAm ROLE for accessing the s3 bucket to store snapshots , on my eks I created a kubernets serviceaccount ( IRSA) that assumes the role IAM created.
I installed also the repository-s3 plugin inside the image elasticsearch 7.17.12.

I’m having a problem with Elasticsearch 7.17.12. I’m trying to set a ‘repository’ where Elasticsearch can store snapshots.
But, it seems like Elasticsearch can’t access the S3 bucket I’ve set up, even though everything looks set up correctly.

Steps to Reproduce

I’m encountering an issue when trying to create an S3 snapshot repository in Elasticsearch 7.17.12. I’m using the S3 repository plugin of the same version.

Here’s the command I’m using to create the repository:

curl -X PUT "http://localhost:9200/_snapshot/elasticsearch-data-backup" -H 'Content-Type: application/json' -u 'user:password' -k  -d' 
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-storage-backup-dev",
    "endpoint": "s3.eu-west-3.amazonaws.com",
    "protocol": "https",
    "compress": true,
    "chunk_size": "100mb",
    "server_side_encryption": true 
    }
}'

output:

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node"}],"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-B77q-Ol-Qz6u5jNS1YR69A/master.dat] using a single upload","caused_by":{"type":"amazon_s3_exception","reason":"amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: REZJX25X7E2TJWCZ; S3 Extended Request ID: 4e``

I’ve set up an IAM role with the necessary permissions and attached it to the Elasticsearch service. The IAM role has the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.eu-west-3.amazonaws.com/id/<ID>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-west-3.amazonaws.com/id/<ID>:sub": [
                        "system:serviceaccount:default:elasticsearch-data",
                        "system:serviceaccount:default:elasticsearch-master"
                    ]
                }
            }
        }
    ]
}

The S3 bucket policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-id>role/elastic-dev"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::elastic-storage-backup-dev",
                "arn:aws:s3:::elastic-storage-backup-dev/*"
            ]
        }
    ]
}

The Elasticsearch service and the S3 bucket are both in the eu-west-3 region. Public access is blocked for the S3 bucket.

Despite this, I’m still encountering the “Access Denied” error when trying to create the S3 snapshot repository. Any help would be greatly appreciated.