I was following the ECK installation guide on Elastic site https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-elasticsearch.html

I am applying this to DigitalOceans managed Kubernetes cluster. I’m using ECK 2.13 and Kubernetes version 1.29.

That was the highest supported Kubernetes version according to https://www.elastic.co/support/matrix#matrix_kubernetes

But I now see that Kubernetes 1.30 is also supported, maybe something is fixed in 1.30 regarding this, but I doubt it.

The first version of my applied yaml:

apiVersion: elasticsearch.k8s.elastic.co/v1

kind: Elasticsearch

metadata:

  name: elasticsearch

spec:

  version: 8.14.0

  nodeSets:

  - name: default

    count: 1

    # Only uncomment the below section if you are not using the previous Daemonset to set max_map_count.

    # config:

    #  node.store.allow_mmap: false

    podTemplate:

      spec:

        # This init container ensures that the max_map_count setting has been applied before starting Elasticsearch.

        # This is not required, but is encouraged when using the previous Daemonset to set max_map_count.

        # Do not use this if setting config.node.store.allow_mmap: false

        initContainers:

        - name: max-map-count-check

          command: ['sh', '-c', "while true; do mmc=$(cat /proc/sys/vm/max_map_count); if [ ${mmc} -eq 262144 ]; then exit 0; fi; sleep 1; done"]

(I also have a DaemonSet applied which raises the max_map_count and Kibana applied, following the guide)

By default the ECK creates a PVC with 1Gi.

The default PVC that was created:

kubectl get pvc -o yaml

apiVersion: v1

items:

- apiVersion: v1

  kind: PersistentVolumeClaim

  metadata:

    annotations:

      pv.kubernetes.io/bind-completed: "yes"

      pv.kubernetes.io/bound-by-controller: "yes"

      volume.beta.kubernetes.io/storage-provisioner: dobs.csi.digitalocean.com

      volume.kubernetes.io/storage-provisioner: dobs.csi.digitalocean.com

    creationTimestamp: "2024-06-12T06:32:15Z"

    finalizers:

    - kubernetes.io/pvc-protection

    labels:

      common.k8s.elastic.co/type: elasticsearch

      elasticsearch.k8s.elastic.co/cluster-name: elasticsearch

      elasticsearch.k8s.elastic.co/statefulset-name: elasticsearch-es-default

    name: elasticsearch-data-elasticsearch-es-default-0

    namespace: default

    ownerReferences:

    - apiVersion: elasticsearch.k8s.elastic.co/v1

      kind: Elasticsearch

      name: elasticsearch

      uid: 787b64f8-de1f-479b-807a-5018dbb88ebf

    resourceVersion: "231899"

    uid: 407e2462-f9b3-4489-9588-4d8139e8a1ac

  spec:

    accessModes:

    - ReadWriteOnce

    resources:

      requests:

        storage: 1Gi

    storageClassName: do-block-storage

    volumeMode: Filesystem

    volumeName: pvc-407e2462-f9b3-4489-9588-4d8139e8a1ac

  status:

    accessModes:

    - ReadWriteOnce

    capacity:

      storage: 1Gi

    phase: Bound

kind: List

metadata:

  resourceVersion: ""

Now I just add the volumeTemplates under my nodeSet as stated here https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html

Note that I’m not even changing the storage value, but I intend to raise it after I figure this problem out.

apiVersion: elasticsearch.k8s.elastic.co/v1

kind: Elasticsearch

metadata:

  name: elasticsearch

spec:

  version: 8.14.0

  nodeSets:

  - name: default

    count: 1

    volumeClaimTemplates:

    - metadata:

        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.

      spec:

        accessModes:

        - ReadWriteOnce

        resources:

          requests:

            storage: 1Gi

        storageClassName: do-block-storage

    # Only uncomment the below section if you are not using the previous Daemonset to set max_map_count.

    # config:

    #  node.store.allow_mmap: false

    podTemplate:

      spec:

        # This init container ensures that the max_map_count setting has been applied before starting Elasticsearch.

        # This is not required, but is encouraged when using the previous Daemonset to set max_map_count.

        # Do not use this if setting config.node.store.allow_mmap: false

        initContainers:

        - name: max-map-count-check

          command: ['sh', '-c', "while true; do mmc=$(cat /proc/sys/vm/max_map_count); if [ ${mmc} -eq 262144 ]; then exit 0; fi; sleep 1; done"]

When I apply this yaml, I get the following error message:

Error from server (Forbidden): error when applying patch:

{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"elasticsearch.k8s.elastic.co/v1","kind":"Elasticsearch","metadata":{"annotations":{},"name":"elasticsearch","namespace":"default"},"spec":{"nodeSets":[{"count":1,"name":"default","podTemplate":{"spec":{"initContainers":[{"command":["sh","-c","while true; do mmc=$(cat /proc/sys/vm/max_map_count); if [ ${mmc} -eq 262144 ]; then exit 0; fi; sleep 1; done"],"name":"max-map-count-check"}]}},"volumeClaimTemplates":[{"metadata":{"name":"elasticsearch-data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"do-block-storage"}}]}],"version":"8.14.0"}}n"}},"spec":{"nodeSets":[{"count":1,"name":"default","podTemplate":{"spec":{"initContainers":[{"command":["sh","-c","while true; do mmc=$(cat /proc/sys/vm/max_map_count); if [ ${mmc} -eq 262144 ]; then exit 0; fi; sleep 1; done"],"name":"max-map-count-check"}]}},"volumeClaimTemplates":[{"metadata":{"name":"elasticsearch-data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"do-block-storage"}}]}]}}

to:

Resource: "elasticsearch.k8s.elastic.co/v1, Resource=elasticsearches", GroupVersionKind: "elasticsearch.k8s.elastic.co/v1, Kind=Elasticsearch"

Name: "elasticsearch", Namespace: "default"

for: "02ElasticsearchCluster.yml": error when patching "02ElasticsearchCluster.yml": admission webhook "elastic-es-validation-v1.k8s.elastic.co" denied the request: Elasticsearch.elasticsearch.k8s.elastic.co "elasticsearch" is invalid: spec.nodeSet[0].volumeClaimTemplates: Invalid value: []v1.PersistentVolumeClaim{v1.PersistentVolumeClaim{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"elasticsearch-data", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PersistentVolumeClaimSpec{AccessModes:[]v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, Selector:(*v1.LabelSelector)(nil), Resources:v1.VolumeResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList{"storage":resource.Quantity{i:resource.int64Amount{value:1073741824, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"1Gi", Format:"BinarySI"}}}, VolumeName:"", StorageClassName:(*string)(0xc00134ba40), VolumeMode:(*v1.PersistentVolumeMode)(nil), DataSource:(*v1.TypedLocalObjectReference)(nil), DataSourceRef:(*v1.TypedObjectReference)(nil), VolumeAttributesClassName:(*string)(nil)}, Status:v1.PersistentVolumeClaimStatus{Phase:"", AccessModes:[]v1.PersistentVolumeAccessMode(nil), Capacity:v1.ResourceList(nil), Conditions:[]v1.PersistentVolumeClaimCondition(nil), AllocatedResources:v1.ResourceList(nil), AllocatedResourceStatuses:map[v1.ResourceName]v1.ClaimResourceStatus(nil), CurrentVolumeAttributesClassName:(*string)(nil), ModifyVolumeStatus:(*v1.ModifyVolumeStatus)(nil)}}}: volume claim templates can only have their storage requests increased, if the storage class allows volume expansion. Any other change is forbidden

The request gets denied with error “volume claim templates can only have their storage requests increased, if the storage class allows volume expansion. Any other change is forbidden”.

I wish it says what change I did that is forbidden. I have also tried “storageClassName: standard” and no storageClassName at all with the same problem.

I read somewhere else that I can check the StatefulSets PVC to see what ECK actually set. I copied only the PVC part of the output.

kubectl get statefulset elasticsearch-es-default -o yaml

  volumeClaimTemplates:

  - apiVersion: v1

    kind: PersistentVolumeClaim

    metadata:

      creationTimestamp: null

      name: elasticsearch-data

    spec:

      accessModes:

      - ReadWriteOnce

      resources:

        requests:

          storage: 1Gi

      volumeMode: Filesystem

    status:

      phase: Pending

Can anyone see the issue here? Is there something wrong with my file? Or some issue with ECK?

I have tried setting different settings in volumeTemplates to figure out what’s wrong. But no matter what I put I get the same error.

Tried “storageClassName: standard” and no storageClassName. And also tried setting the same volumeMode and volumeName that is set in the PVC spec I posted above.