I am running Ignite in a Kubernetes cluster with the following configuration:

• persistence enabled
• Caches are partitioned
• Backups enabled

For the K8s configuration I have a service enabled to expose the cluster and its ports. I am using a stateful set to add new pods to the cluster and using the “TcpDiscoveryKubernetesIpFinder” to add the new pods to the cluster.

I am noticing strange behavior whenever I attempt to scale my cluster by N amount. I scale the stateful set and once the nodes are all created and ready, I manually add the new pods to the cluster baseline. I have clients (mostly thin clients) connected to the Ignite cluster and they each have a Continuous Query per cache that I am reading from. I attempted to scale my cluster while the clients are still connected to the cluster and noticed that Continuous Queries on every client was beginning to lose some data. It looked like any data that was coming from the new nodes in the cluster were lost. I tested this by connecting multiple clients to an Ignite cluster and logging a count of all the Continuous Query entries it gets. For the Ignite cluster I had it add 10000 records to a cache every 5 minutes. The counts of the writes and reads fully matched on all clients until I scaled the cluster. Once scaled I would get varied results but I never got all 100000 records that were added. This continues until I restart the client connections and restart the Continuous Queries. In that event all the reads matched the writes again.

My question is: is this a known issue or is there already some workaround for this? I see no information regarding it in the Ignite documentation.

Here is my data storage configuration:

<property name="dataStorageConfiguration">
    <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
        <property name="walPath" value="/ignite/wal"/>
        <property name="walArchivePath" value="/ignite/walarchive"/>
        <property name="maxWalArchiveSize" value="12gb"/>
        <property name="metricsEnabled" value="true"/>
        <property name="walSegmentSize" value="128mb"/>

        <property name="defaultDataRegionConfiguration">
            <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                <!-- enable mertrics for the default data region -->
                <property name="metricsEnabled" value="true"/>
                <property name="persistenceEnabled" value="true"/>
                <property name="name" value="Default_Region"/>
                <property name="maxSize" value="8gb"/>

            </bean>
        </property>
    </bean>
</property>