Problem Description:

We have a table in Cassandra version 3.11.7. After migrating three nodes to another availability zone, I encountered an issue where I cannot retrieve data by the primary key, despite the data being successfully inserted.

Table schema:

CREATE TABLE m_operational.app_settings (
    endpoint_id text PRIMARY KEY,
    format text,
    settings_json text,
    platform text
) WITH bloom_filter_fp_chance = 0.01
    AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
    AND comment = ''
    AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
    AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND crc_check_chance = 1.0
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99PERCENTILE';

Select query that i trying to execute:

select * from m_operational.app_settings where endpoint_id='endpoint-i-need';

And I get nothing even after insert query:

INSERT INTO m_operational.app_settings (endpoint_id, format, settings_json, platform) VALUES( 'endpoint-i-need', 'free', '{""settings"": ""my-settings""}', 'PC');

tried to create a similar record next to it but with a different key – everything worked:

INSERT INTO m_operational.app_settings (endpoint_id, format, settings_json, platform) VALUES( 'endpoint-i-need1', 'free', '{""settings"": ""my-settings""}', 'PC');
select * from m_operational.app_settings where endpoint_id='endpoint-i-need1';

This problem started to arise during the migration of 3 Cassandra nodes from one availability zone to another and remains to this day (There are 9 nodes in the cluster).

The migration was carried out as follows:

• Creation of new nodes and adding to the existing cluster
• Reconfiguration of services to use new nodes
• Decommission and removal of old nodes

After migration I execute nodetool cleanup on each of the nodes and then ran a repair with the command nodetool repair --full -pr -j 2 on each of the nodes. Nothing has changed and the select still doesn’t work. Then I tried to run repair for a specific table on all nodes with nodetool repair --full -pr -j 2 m_operational app_settings – that didn’t help either.

What can I look at in Cassandra to understand the cause of the problem? or how can i fix this problem?