I have an existing ELK stack with logs, metrics and APM all reporting in successfully for a couple of years now.

We recently added a Cassandra cluster to our application, and I’d like to start getting logs and metrics to it. I could set up Filebeat/Metricbeat the way we have for other services, but I see there’s an official Cassandra integration available (via Jolokia) – but I’m having trouble getting it to work.

In cassandra-env.sh I’ve added

JVM_OPTS="$JVM_OPTS -javaagent:/usr/share/cassandra/lib/jolokia-jvm-1.7.2.jar=port=9000,host=0.0.0.0"

I can curl that from the local host, and I get a message back, e.g.

curl 'localhost:9000/jolokia/?ignoreErrors=true&canonicalNaming=false'

{
  "request": {
    "type": "version"
  },
  "value": {
    "agent": "1.7.1",
    "protocol": "7.2",
    "config": {
      "listenForHttpService": "true",
      "maxCollectionSize": "0",
      "authIgnoreCerts": "false",
...

I’ve also set up the APM agent, yaml:

id: 2016d7cc-135e-5583-9758-3ba01f5a06e5
revision: 3
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://...:9200'
    username: 'elastic'
    password: '...'
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
...

Everything seems like its working. There are log files getting generated on the Cassandra instance in /opt/Elastic/Agent/data/elastic-agent-bc2cbf/logs for filebeat and metricbeat.

Switching back over to Kibana, I added the Cassandra integration. Some of the input options are non-obvious to me, but essentially:

Integration name: cassandra-1
Collect System Logs from Cassandra (switch on)
  Cassandra logs (switch on)
    Paths /var/log/cassandra/system.log
Collect Cassandra metrics (switch on)
  Settings
    hosts: ....
    Cassandra metrics (switched on)
      Period 10s

Everything seems fine – ELK stack gets some log and metric indexes added (logs-* and metrics-*), but they are completely empty. Nothing populates. Nothing populates in the APM observability either.

How can I try and debug this? I don’t see any obvious errors in the logstash logs. I’m not sure where else to look – are the logs getting to ELK and then failing? Is the APM server not successfully reaching out to Jolokia to get metrics?