I faced with problem due to connect two cassandra clusters placed in different datacenters. Each datacenter have public ip – lets assume there are 111.111.111.111 and 222.222.222.222. Cassandra nodes starts in docker contaners – 3 nodes in each cluster. The problem is clusters don’t see each other.
Here are docker-compose files for first cluster on host with ip 111.111.111.111
version: '3.3'
services:
cass1:
image: cassandra
container_name: cass1
hostname: cass1
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 5s
timeout: 5s
retries: 60
networks:
- cassandra
ports:
- "9042:9042"
- "7000:7000"
environment: &environment
HEAP_NEWSIZE: 128M
MAX_HEAP_SIZE: 2048M
CASSANDRA_SEEDS: "cass1, cass2, 222.222.222.222”
CASSANDRA_CLUSTER_NAME: ks
CASSANDRA_DC: DC1
CASSANDRA_RACK: West
CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
CASSANDRA_NUM_TOKENS: 256
cass2:
image: cassandra
container_name: cass2
hostname: cass2
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 5s
timeout: 5s
retries: 60
networks:
- cassandra
ports:
- "9043:9042"
- "7001:7000"
environment: *environment
depends_on:
- cass1
cass3:
image: cassandra
container_name: cass3
hostname: cass3
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 5s
timeout: 5s
retries: 60
networks:
- cassandra
ports:
- "9044:9042"
- "7002:7000"
environment: *environment
depends_on:
- cass2
networks:
cassandra:
And the second cluster on host with ip 222.222.222.222
version: '3.3'
services:
cass1:
image: cassandra
container_name: cass1
hostname: cass1
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 5s
timeout: 5s
retries: 60
networks:
- cassandra
ports:
- "9042:9042"
- "7000:7000"
environment: &environment
HEAP_NEWSIZE: 128M
MAX_HEAP_SIZE: 2048M
CASSANDRA_SEEDS: "cass1, cass2, 111.111.111.111”
CASSANDRA_CLUSTER_NAME: ks
CASSANDRA_DC: DC2
CASSANDRA_RACK: East
CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
CASSANDRA_NUM_TOKENS: 256
cass2:
image: cassandra
container_name: cass2
hostname: cass2
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 5s
timeout: 5s
retries: 60
networks:
- cassandra
ports:
- "9043:9042"
- "7001:7000"
environment: *environment
depends_on:
- cass1
cass3:
image: cassandra
container_name: cass3
hostname: cass3
healthcheck:
test: ["CMD", "cqlsh", "-e", "describe keyspaces" ]
interval: 5s
timeout: 5s
retries: 60
networks:
- cassandra
ports:
- "9044:9042"
- "7002:7000"
environment: *environment
depends_on:
- cass2
networks:
cassandra:
After startup each cluster is seeing only itself and doesn’t see other cluster. For example nodetool status output for 111.111.111.111
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.18.0.4 108.81 KiB 256 60.1% 3ceecd38-0c30-4189-9a52-cb23465ca0ca West
UN 172.18.0.3 113.77 KiB 256 70.4% f1f6f63a-7cc5-4883-8b90-5159c6b6613c West
UN 172.18.0.2 108.74 KiB 256 69.5% 94395942-8efa-447b-9961-2d80375bce35 West
netcat show that another host is available
nc -v 222.222.222.222 7000
Connection to 222.222.222.222 7000 port [tcp/*]
succeeded!
No errors in cassandra logs.
Maybe is some missconfiguration happened? I tried to run 2 cluster on same host and it worked properly with intercommunication between clusters.
TY for answers.
stroiker is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.