I have an EKS cluster and I’m trying to set up Cloudwatch Agent as explained here (Option 2):

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Observability-EKS-addon.html#install-CloudWatch-Observability-EKS-addon-serviceaccountrole

However, looking at the cloudwatch agent logs, I see:

2024-08-07T21:17:56Z E! {"caller":"exporterhelper/common.go:292","msg":"Exporting failed. Rejecting data.","kind":"exporter","data_type":"metrics","name":"awsemf/containerinsights","error":"SharedCredsLoad: failed to load shared credentials filencaused by: FailedRead: unable to open filencaused by: open /root/.aws/credentials: no such file or directory","rejected_items":1170,"stacktrace":"go.opentelemetry.io/collector/exporter/exporterhelper.(*baseExporter).sendntgo.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:292ngo.opentelemetry.io/collector/exporter/exporterhelper.NewMetricsRequestExporter.func1ntgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:134ngo.opentelemetry.io/collector/consumer.ConsumeMetricsFunc.ConsumeMetricsntgo.opentelemetry.io/collector/[email protected]/metrics.go:25ngithub.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourcetotelemetry.(*wrapperMetricsExporter).ConsumeMetricsntgithub.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/resource_to_telemetry.go:32ngo.opentelemetry.io/collector/internal/fanoutconsumer.(*metricsConsumer).ConsumeMetricsntgo.opentelemetry.io/[email protected]/internal/fanoutconsumer/metrics.go:60ngo.opentelemetry.io/collector/processor/batchprocessor.(*batchMetrics).exportntgo.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:449ngo.opentelemetry.io/collector/processor/batchprocessor.(*shard).sendItemsntgo.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:261ngo.opentelemetry.io/collector/processor/batchprocessor.(*shard).startLoopntgo.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:223"}
2024-08-07T21:17:56Z W! {"caller":"[email protected]/batch_processor.go:263","msg":"Sender failed","kind":"processor","name":"batch/containerinsights","pipeline":"metrics/containerinsights","error":"SharedCredsLoad: failed to load shared credentials filencaused by: FailedRead: unable to open filencaused by: open /root/.aws/credentials: no such file or directory"}

So it seems that it’s looking for a credentials file.

If I describe the addon, I see it is using the right role:

{
    "addon": {
        "addonName": "amazon-cloudwatch-observability",
        "clusterName": "eks-cluster",
        "status": "ACTIVE",
        "addonVersion": "v1.10.0-eksbuild.2",
        "health": {
            "issues": []
        },
        "addonArn": "arn:aws:eks:eu-west-2:xxx:addon/eks-cluster/amazon-cloudwatch-observability/b2c89759-7791-6f15-67ca-6fcb0510652d",
        "createdAt": "2024-08-07T22:16:04.440000+01:00",
        "modifiedAt": "2024-08-07T22:16:45.187000+01:00",
        "serviceAccountRoleArn": "arn:aws:iam::xxx:role/cloudwatch_agent",
        "tags": {}
    }
}

The role:

{
    "Role": {
        "Path": "/",
        "RoleName": "cloudwatch_agent",
        "RoleId": "AROA4YYE6JCUP3VG2DE54",
        "Arn": "arn:aws:iam::xxx:role/cloudwatch_agent",
        "CreateDate": "2024-08-07T21:14:37+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Federated": "arn:aws:iam::xxx:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/4BAA6DC8AAD40A9660EE4B17AFFF2926"
                    },
                    "Action": "sts:AssumeRoleWithWebIdentity",
                    "Condition": {
                        "StringEquals": {
                            "oidc.eks.eu-west-2.amazonaws.com/id/4BAA6DC8AAD40A9660EE4B17AFFF2926:aud": "sts.amazonaws.com",
                            "oidc.eks.eu-west-2.amazonaws.com/id/4BAA6DC8AAD40A9660EE4B17AFFF2926:sub": "system:serviceaccount:amazon-cloudwatch:cloudwatch-agent"
                        }
                    }
                }
            ]
        },
        "Description": "",
        "MaxSessionDuration": 3600,
        "Tags": [
            {
                "Key": "alpha.eksctl.io/cluster-name",
                "Value": "eks-cluster"
            },
            {
                "Key": "alpha.eksctl.io/iamserviceaccount-name",
                "Value": "amazon-cloudwatch/cloudwatch-agent"
            },
            {
                "Key": "alpha.eksctl.io/eksctl-version",
                "Value": "0.188.0"
            },
            {
                "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name",
                "Value": "eks-cluster"
            }
        ],
        "RoleLastUsed": {}
    }
}

Any tips how to debug / fix this ?