Currently, I have set up an Elasticsearch in a StatefulSet with 3 replicas and Kibana with 1 replica in a deployment in OpenShift. Now, I am facing persistence problems in both Kibana and Elasticsearch. When I restart my pods, I lose all the data. When I deploy my StatefulSet, it automatically creates PVCs based on my replicas. If I change the password of the Kibana system and restart the pod, it gives me an error and I need to reset it also. Even if I added the Elasticsearch_username as kibana_system and Elasticsearch_password, it doesn’t work.

            - name: ELASTICSEARCH_USERNAME
              value: kibana_system
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: kibana-system
                  key: kibana-system

I’m using this redhat image :
https://catalog.redhat.com/software/containers/elastic/elasticsearch/5fac2d6dac3db90370a224c7

this is what i got :
My elasticsearch statefullSet :

kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: elasticsearch
  namespace: newelk-uat
  uid: 25b6c912-b4a7-4c2e-82a5-324147c367c4
  resourceVersion: '372112555'
  generation: 374
  creationTimestamp: '2024-05-23T09:21:56Z'
spec:
  replicas: 2
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch
    spec:
      volumes:
        - name: elasticsearch-certs-p12
          secret:
            secretName: elasticsearch-certs-p12
            defaultMode: 420
        - name: elasticsearch-user-password
          secret:
            secretName: elasticsearch-user-password
            defaultMode: 420
      containers:
        - resources:
            limits:
              cpu: '1'
              memory: 3Gi
            requests:
              cpu: 500m
              memory: 1Gi
          terminationMessagePath: /dev/termination-log
          name: elasticsearch
          env:
            - name: ingest.geoip.downloader.enabled
              value: 'false'
            - name: xpack.security.enabled
              value: 'true'
            - name: elasticsearch-xpack
              value: disabled
            - name: cluster.name
              value: Cluster-Uat-Openshift
            - name: discovery.seed_hosts
              value: >-
                elasticsearch-0.elasticsearch-discovery,elasticsearch-1.elasticsearch-discovery
            - name: cluster.initial_master_nodes
              value: 'elasticsearch-0,elasticsearch-1'
            - name: xpack.security.transport.ssl.enabled
              value: 'true'
            - name: xpack.security.transport.ssl.verification_mode
              value: certificate
            - name: xpack.security.transport.ssl.client_authentication
              value: required
            - name: xpack.security.transport.ssl.keystore.path
              value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
            - name: xpack.security.transport.ssl.truststore.path
              value: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
            - name: ELASTIC_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-user-password
                  key: elasticsearch-user-password
          ports:
            - name: rest
              containerPort: 9200
              protocol: TCP
            - name: inter-node
              containerPort: 9300
              protocol: TCP
          imagePullPolicy: Always
          volumeMounts:
            - name: elasticsearch-certs-p12
              mountPath: /usr/share/elasticsearch/config/certs
          terminationMessagePolicy: File
          image: >-
            openshift.artifactory.com/elastic/elasticsearch:8.14.0
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  volumeClaimTemplates:
    - kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: elasticsearch-storage
        creationTimestamp: null
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        volumeMode: Filesystem
      status:
        phase: Pending
  serviceName: elasticsearch-discovery
  podManagementPolicy: OrderedReady
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 0
  revisionHistoryLimit: 10
status:
  observedGeneration: 374
  availableReplicas: 2
  updateRevision: elasticsearch-7dd6ccd4f8
  currentRevision: elasticsearch-7dd6ccd4f8
  currentReplicas: 2
  updatedReplicas: 2
  replicas: 2
  collisionCount: 0
  readyReplicas: 2

and this is my kibana deployment :

kind: Deployment
apiVersion: apps/v1
metadata:
  name: kibana
  namespace: newelk-uat
  uid: 01b46a0b-b87b-4e06-b517-40602bd32fd1
  resourceVersion: '372138708'
  generation: 308
  creationTimestamp: '2024-02-22T12:47:26Z'
  labels:
    app: kibana
    app.kubernetes.io/component: kibana
    app.kubernetes.io/instance: kibana
    app.kubernetes.io/name: kibana
    app.kubernetes.io/part-of: kibana
    app.openshift.io/runtime: elastic
  annotations:
    deployment.kubernetes.io/revision: '126'
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: kibana
        app.kubernetes.io/component: kibana
        app.kubernetes.io/name: kibana
        deployment: kibana
      annotations:
        openshift.io/generated-by: OpenShiftWebConsole
    spec:
      volumes:
        - name: kibana-storage
          persistentVolumeClaim:
            claimName: kibana-storage
        - name: elasticsearch-certs-p12
          secret:
            secretName: elasticsearch-certs-p12
            defaultMode: 420
      containers:
        - resources:
            limits:
              cpu: 500m
              memory: 1Gi
            requests:
              cpu: 250m
              memory: 512Mi
          terminationMessagePath: /dev/termination-log
          name: kibana
          env:
            - name: openssl.legacy.provider
              value: 'false'
            - name: ELASTICSEARCH_URL
              value: >-
                http://elasticsearch-discovery.svc.cluster.local:9200
            - name: ELASTICSEARCH_HOSTS
              value: >-
                http://elasticsearch-discovery.svc.cluster.local:9200
            - name: elasticsearch.ssl.verificationMode
              value: none
            - name: SERVER_PUBLICBASEURL
              value: >-
                https://kibana-elk-uat.fr/
            - name: xpack.security.enrollment.enabled
              value: 'true'
            - name: ELASTICSEARCH_USERNAME
              value: kibana_system
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: kibana-system
                  key: kibana-system
          ports:
            - containerPort: 5601
              protocol: TCP
          imagePullPolicy: Always
          volumeMounts:
            - name: kibana-storage
              mountPath: /usr/share/kibana/data
            - name: elasticsearch-certs-p12
              mountPath: /usr/share/kibana/config/certs
          terminationMessagePolicy: File
          image: >-
            openshift.artifactory.com/elastic/kibana@sha256:061305bf6358a6e97f00bbff65409dd073fb6505a0d7fc77e12b3b6fb9c99dd3
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
status:
  observedGeneration: 308
  replicas: 1
  updatedReplicas: 1
  readyReplicas: 1
  availableReplicas: 1
  conditions:
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2024-07-15T15:31:30Z'
      lastTransitionTime: '2024-06-18T15:42:28Z'
      reason: NewReplicaSetAvailable
      message: ReplicaSet "kibana-6cdcf679f5" has successfully progressed.
    - type: Available
      status: 'True'
      lastUpdateTime: '2024-07-16T09:24:48Z'
      lastTransitionTime: '2024-07-16T09:24:48Z'
      reason: MinimumReplicasAvailable
      message: Deployment has minimum availability.```

Thank you !