I attempted to configure the xpack.security.enabled property in order to add authentication to ELK. Before setting xpack security, APM logs were functional as expected. However, no new APM logs are generated after it is enabled.

I have enabled xpack security for the ELK stack by adding the xpack.security.enabled property in elasticsearch.yml, executed the Elasticsearch pod, and used elasticsearch-setup-passwords to get autogenerated passwords for each role. I added the following configs to the kibana.yml and apm-server.yml files.

apm-server.yml

 containers:
      - env:
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearch:9200
        - name: ELASTICSEARCH_USERNAME
          value: apm_system
        - name: ELASTICSEARCH_PASSWORD
          value: yyyyyyyyy

kibana.yml

containers:
      - env:
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearch:9200
        - name: ELASTICSEARCH_USERNAME
          value: kibana
        - name: ELASTICSEARCH_PASSWORD
          value: xxxxxxxxx

Elasticsearch.yml

 containers:
      - env:
        - name: xpack.monitoring.enabled
          value: "true"
        - name: xpack.watcher.enabled
          value: "false"
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        - name: discovery.type
          value: single-node
        - name: cluster.routing.allocation.disk.threshold_enabled
          value: "false"
        - name: xpack.security.enabled
          value: "true"

I restarted the pods in the following order: Elasticsearch, Kibana, and APM server after adding the above configurations.
Then used the elastic login and password that was generated automatically to log into Kibana.
After navigating to the discover menu, I noticed that there are no APM logs collected after adding the xpack security. Also found that no log data (the data that saved into Elasticsearch from a backend microservice) saved in the index patterns.
When checking the logs in that service and APM server, found the below authentication issue.

-backend service log

[10:04:03 ERR] Failed to index audit log entry: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. Call: Status code 401 from: PUT /audit_logs/_doc/173f0-def1-4552-9401-719729. ServerError: Type: security_exception Reason: “missing authentication credentials for REST request [/audit_logs/_doc/173f0-def1-4552-9401-719729]

-apm-server log

“message”:”Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): 401 Unauthorized: {“error”:{“root_cause”:[{“type”:”security_exception”,”reason”:”missing authentication credentials for REST request [/]”,”header”:{“WWW-Authenticate”:”Basic realm=”security” charset=”UTF-8″”}}],”type”:”security_exception”,”reason”:”missing authentication credentials for REST request [/]”,”header”:{“WWW-Authenticate”:”Basic realm=”security” charset=”UTF-8″”}},”status”:401}”,”service.name”:”apm-server”,”ecs.version”:”1.6.0″}

Any guidance or advice to fix this issue would be greatly appreciated. Thank you!