I am new to EKS and Terraform. I am using Terraform script to create an EKS cluster using GPU nodes. The script eventually throws an error after 20 minutes stating that last error: i-******: NodeCreationFailure: Instances failed to join the kubernetes cluster.

Logged in to the node to see what is going on:

• systemctl status kubelet => kubelet.service - Kubernetes Kubelet. Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; preset: disabled) Active: inactive (dead)
• systemctl restart kubelet => Job for kubelet.service failed because of unavailable resources or another system error. See "systemctl status kubelet.service" and "journalctl -xeu kubelet.service" for details.
• journalctl -xeu kubelet.service => ...kubelet.service: Failed to load environment files: No such file or directory ...kubelet.service: Failed to run 'start-pre' task: No such file or directory ...kubelet.service: Failed with result 'resources'.

I am using the latest version of this AMI: amazon-eks-node-al2023-x86_64-nvidia-1.31-* as the Kubernetes version is 1.31 and my instance type: g4dn.2xlarge.

I tried many different combinations, but no luck. Any help is appreciated. Here is the relevant portion of my Terraform script:

resource "aws_eks_cluster" "eks_cluster" {
  name     = "${var.branch_prefix}eks_cluster"
  role_arn = module.iam.eks_execution_role_arn

  access_config {
    authentication_mode                         = "API_AND_CONFIG_MAP"
    bootstrap_cluster_creator_admin_permissions = true
  }

  vpc_config {
    subnet_ids = var.eks_subnets
  }

  tags = var.app_tags
}

resource "aws_launch_template" "eks_launch_template" {
  name          = "${var.branch_prefix}eks_lt"
  instance_type = var.eks_instance_type
  image_id      = data.aws_ami.eks_gpu_optimized_worker.id 

  block_device_mappings {
    device_name = "/dev/sda1"

    ebs {
      encrypted   = false
      volume_size = var.eks_volume_size_gb
      volume_type = "gp3"
    }
  }

  network_interfaces {
    associate_public_ip_address = false
    security_groups             = module.secgroup.eks_security_group_ids
  }

  user_data = filebase64("${path.module}/userdata.sh")
  key_name  = "${var.branch_prefix}eks_deployer_ssh_key"

  tags = {
    "kubernetes.io/cluster/${aws_eks_cluster.eks_cluster.name}" = "owned"
  }
}

resource "aws_eks_node_group" "eks_private-nodes" {
  cluster_name    = aws_eks_cluster.eks_cluster.name
  node_group_name = "${var.branch_prefix}eks_cluster_private_nodes"
  node_role_arn   = module.iam.eks_nodes_group_execution_role_arn
  subnet_ids      = var.eks_subnets

  capacity_type  = "ON_DEMAND"

  scaling_config {
    desired_size = var.eks_desired_instances
    max_size     = var.eks_max_instances
    min_size     = var.eks_min_instances
  }

  update_config {
    max_unavailable = 1
  }

  launch_template {
    name    = aws_launch_template.eks_launch_template.name
    version = aws_launch_template.eks_launch_template.latest_version
  }

  tags = {
    "kubernetes.io/cluster/${aws_eks_cluster.eks_cluster.name}" = "owned"
  }
}