I am following the guide to override the number of maximum pods per node from https://docs.aws.amazon.com/eks/latest/userguide/cni-increase-ip-addresses.html

The guide is quite straightforward if I were to use eksctl to create my managed EKS node group. However, I am using Terraform, and I cannot seem to reach a deterministic and reproducible state for my cluster.

I am using a custom launch template with custom user data. However, after the first terraform apply (when none of the infrastructure is provisioned), the maximum number of pods per node remains at the default value of 17 for a t3.medium instance. Making any small unrelated change to the node group terraform configuration recycles the node group, and the maximum number of pods suddenly becomes 110, even though I have set a value of 109 in my custom launch template configuration.

I have tried two approaches, to create a .conf file in the systemd folder to override the kubelet configuration as seen in the following template:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -o xtrace
echo "KUBELET_EXTRA_ARGS=--max-pods=${max_pods}" >> /etc/systemd/system/kubelet.service.d/30-extra-args.conf
systemctl daemon-reload
systemctl restart kubelet
--//--

With the following custom aws_launch_template:

resource "aws_launch_template" "eks" {
  name_prefix   = "${local.env}-${local.eks_name}-eks-nodes"
  instance_type = "t3.medium"
  user_data     = base64encode(templatefile("${path.module}/user_data.sh.tpl", {
    max_pods = 109, # to make sure that the actual max pods value is sourced from here
  }))

  tags = {
    Name = "${local.env}-${local.eks_name}-eks-nodes"
  }

  lifecycle {
    create_before_destroy = true
  }
}

output "launch_template_latest_version" {
  value = aws_launch_template.eks.latest_version
}

I’ve also tried running the bootstrap command on the node with the necessary parameters:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash

/etc/eks/bootstrap.sh ${cluster_name} 
  --use-max-pods false 
  --kubelet-extra-args '--max-pods=${max_pods}'
--//

This fails to create the node group altogether with a timeout error.

The environment variable on the aws-node DaemonSet is applied using the kubectl_manifest resource from a kubectl provider:

resource "kubectl_manifest" "aws_node_patch" {
  yaml_body = <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aws-node
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: aws-node
        env:
        - name: ENABLE_PREFIX_DELEGATION
          value: "true"
EOF

  depends_on = [
    aws_eks_addon.vpc_cni
  ]

}

I am wondering whether I am missing something or applying the right parameters in the wrong spot?