I’ve set up replication between 2 artifactory instances (both version 7.77.x) with the following settings 0 16 * * * ? * cron-trigger-docu
So i expect the replication happening every hour at minute 16.
It’s skipping some hours randomly and I want to know why.
I’ve looked into the logs, here is a an example output.
grep: grep -B 3 "remote folder replication for" console.log | egrep -v "Removing the properties of| download|Deploy to|Executing file list request| Revoked "
2024-07-01T21:16:00.751Z [jfrt ] [INFO ] [19499ffc69a0ead0] [o.a.a.c.BasicStatusHolder:216 ] [9a0ead0|art-exec-130] - Completed remote folder replication for my-remote-repo/ with 2 deployed files, 0 deleted files, 2 properties change, 0 statistics change, 0 mkDirs... average events per second 5.35
--
2024-07-02T05:15:34.733Z [jfac ] [INFO ] [34acfe578aecfd20] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task4 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T05:16:00.001Z [jfrt ] [INFO ] [b50bc45dcaf59fb ] [o.a.s.TaskServiceImpl:441 ] [caf59fb|art-exec-159] - IMPOSSIBLE task process for RemoteReplicationJob is active: RemoteReplicationJob
2024-07-02T05:16:00.002Z [jfrt ] [INFO ] [7d63138c117989ee] [o.a.a.c.BasicStatusHolder:216 ] [17989ee|art-exec-173] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T05:16:18.746Z [jfrt ] [INFO ] [7d63138c117989ee] [o.a.a.c.BasicStatusHolder:216 ] [17989ee|art-exec-173] - Completed remote folder replication for my-remote-repo/ with 16 deployed files, 0 deleted files, 16 properties change, 0 statistics change, 0 mkDirs... average events per second 1.71
--
2024-07-02T06:15:34.747Z [jfac ] [INFO ] [75df1dec0acbb85] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task1 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T06:16:00.002Z [jfrt ] [INFO ] [869b36de6bd8da73] [o.a.a.c.BasicStatusHolder:216 ] [bd8da73|art-exec-146] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T06:16:01.012Z [jfrt ] [INFO ] [869b36de6bd8da73] [o.a.a.c.BasicStatusHolder:216 ] [bd8da73|art-exec-146] - Completed remote folder replication for my-remote-repo/ with 3 deployed files, 0 deleted files, 3 properties change, 0 statistics change, 0 mkDirs... average events per second 5.94
--
2024-07-02T09:15:34.784Z [jfac ] [INFO ] [684ee05b4cc50c5c] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task9 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T09:16:00.002Z [jfrt ] [INFO ] [45f495b847ae10fc] [o.a.a.c.BasicStatusHolder:216 ] [7ae10fc|art-exec-183] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T09:16:00.770Z [jfrt ] [INFO ] [45f495b847ae10fc] [o.a.a.c.BasicStatusHolder:216 ] [7ae10fc|art-exec-183] - Completed remote folder replication for my-remote-repo/ with 2 deployed files, 0 deleted files, 2 properties change, 0 statistics change, 0 mkDirs... average events per second 5.21
--
2024-07-02T11:15:34.809Z [jfac ] [INFO ] [1af984378372a938] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task4 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T11:16:00.001Z [jfrt ] [INFO ] [73289f664342424d] [o.a.s.TaskServiceImpl:441 ] [342424d|art-exec-159] - IMPOSSIBLE task process for RemoteReplicationJob is active: RemoteReplicationJob
2024-07-02T11:16:00.002Z [jfrt ] [INFO ] [8c622373eaf7b4e0] [o.a.a.c.BasicStatusHolder:216 ] [af7b4e0|art-exec-178] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T11:16:00.819Z [jfrt ] [INFO ] [8c622373eaf7b4e0] [o.a.a.c.BasicStatusHolder:216 ] [af7b4e0|art-exec-178] - Completed remote folder replication for my-remote-repo/ with 2 deployed files, 0 deleted files, 2 properties change, 0 statistics change, 0 mkDirs... average events per second 4.9
--
2024-07-02T14:12:52.092Z [jfcon] [INFO ] [488d272b4f97708e] [entitlements_storage.go:86 ] [main ] - Received entitlements update with no changes [entitlements_storage]
2024-07-02T14:12:52.105Z [jfcon] [INFO ] [488d272b4f97708e] [entitlements_service.go:260 ] [main ] - Synced action 'fetch entitlements' - finished execution [entitlement_service]
2024-07-02T14:15:34.836Z [jfac ] [INFO ] [4264874affb31844] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task4 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T14:16:00.003Z [jfrt ] [INFO ] [fb6f3f96f1873b24] [o.a.a.c.BasicStatusHolder:216 ] [1873b24|art-exec-182] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T14:16:00.871Z [jfrt ] [INFO ] [fb6f3f96f1873b24] [o.a.a.c.BasicStatusHolder:216 ] [1873b24|art-exec-182] - Completed remote folder replication for my-remote-repo/ with 2 deployed files, 0 deleted files, 2 properties change, 0 statistics change, 0 mkDirs... average events per second 4.61
--
2024-07-02T17:15:34.902Z [jfac ] [INFO ] [7115ab7692da64b1] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task1 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T17:16:00.001Z [jfrt ] [INFO ] [3bae37a43d8afb7f] [o.a.s.TaskServiceImpl:441 ] [d8afb7f|art-exec-191] - IMPOSSIBLE task process for RemoteReplicationJob is active: RemoteReplicationJob
2024-07-02T17:16:00.002Z [jfrt ] [INFO ] [383e70045e44a596] [o.a.a.c.BasicStatusHolder:216 ] [e44a596|art-exec-201] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T17:16:00.850Z [jfrt ] [INFO ] [383e70045e44a596] [o.a.a.c.BasicStatusHolder:216 ] [e44a596|art-exec-201] - Completed remote folder replication for my-remote-repo/ with 2 deployed files, 0 deleted files, 2 properties change, 0 statistics change, 0 mkDirs... average events per second 4.72
--
2024-07-02T18:15:34.906Z [jfac ] [INFO ] [266cac0605dada] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task3 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T18:16:00.002Z [jfrt ] [INFO ] [a358ddde89e96528] [o.a.a.c.BasicStatusHolder:216 ] [9e96528|art-exec-201] - Starting remote folder replication for 'my-remote-repo'.
2024-07-02T18:16:00.359Z [jfrt ] [INFO ] [a358ddde89e96528] [o.a.a.c.BasicStatusHolder:216 ] [9e96528|art-exec-201] - Completed remote folder replication for my-remote-repo/ with 0 deployed files, 0 deleted files, 0 properties change, 0 statistics change, 0 mkDirs... average events per second 0
--
2024-07-02T21:15:34.944Z [jfac ] [INFO ] [4a655af76167fc62] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task2 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T21:16:00.002Z [jfrt ] [INFO ] [e909f95dea67749a] [o.a.a.c.BasicStatusHolder:216 ] [a67749a|art-exec-192] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T21:16:00.716Z [jfrt ] [INFO ] [e909f95dea67749a] [o.a.a.c.BasicStatusHolder:216 ] [a67749a|art-exec-192] - Completed remote folder replication for my-remote-repo/ with 2 deployed files, 0 deleted files, 2 properties change, 0 statistics change, 0 mkDirs... average events per second 5.6
--
2024-07-02T22:15:34.965Z [jfac ] [INFO ] [db82462ec2dd3a6e] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task5 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-02T22:16:00.002Z [jfrt ] [INFO ] [9413c382cf727e53] [o.a.a.c.BasicStatusHolder:216 ] [f727e53|art-exec-190] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-02T22:16:00.799Z [jfrt ] [INFO ] [9413c382cf727e53] [o.a.a.c.BasicStatusHolder:216 ] [f727e53|art-exec-190] - Completed remote folder replication for my-remote-repo/ with 2 deployed files, 0 deleted files, 2 properties change, 0 statistics change, 0 mkDirs... average events per second 5.02
--
2024-07-03T00:13:48.364Z [jfcon] [INFO ] [488d272b4f97708e] [entitlements_storage.go:86 ] [main ] - Received entitlements update with no changes [entitlements_storage]
2024-07-03T00:13:48.376Z [jfcon] [INFO ] [488d272b4f97708e] [entitlements_service.go:260 ] [main ] - Synced action 'fetch entitlements' - finished execution [entitlement_service]
2024-07-03T00:15:34.982Z [jfac ] [INFO ] [bd662b4fd3e0409] [.s.s.r.StaleTokenCleanupJob:66] [jf-access-task2 ] - Starting Access job to remove stale nodes and join-admin-tokens on server: artifactory-001
2024-07-03T00:16:00.003Z [jfrt ] [INFO ] [2b82ec7c1811dfd0] [o.a.a.c.BasicStatusHolder:216 ] [811dfd0|art-exec-199] - Starting remote folder replication for 'my-remote-repo'.
--
2024-07-03T00:16:00.924Z [jfrt ] [INFO ] [2b82ec7c1811dfd0] [o.a.a.c.BasicStatusHolder:216 ] [811dfd0|art-exec-199] - Completed remote folder replication for my-remote-repo/ with 3 deployed files, 0 deleted files, 3 properties change, 0 statistics change, 0 mkDirs... average events per second 6.52
missing the following hours on 2nd of July:
01:16
02:16
03:16
04:16
07:16
08:16
10:16
12:16
13:16
15:16
16:16
19:16
20:16
23:16
If i look/grep around for 2024-07-02T01:16 i get the following lines, not showing any hints why it won’t run the replication
2024-07-02T01:16:03.277Z 35[jfob ] [INFO ] [1eaa77bc4c50f81e] [shipping_service.go:626 ] [main ] [] - Uploading 1 consumption file/s to artifactory/api/v1/system/logs/usageartifactory/2024-07-02 with params map[nodeId:949589f3ab0f productName:artifactory serviceId:01hv6nhk9v4e770se4n45z0j4v], filelist: [/opt/jfrog/artifactory/var/data/metadata/usage/archived/metadata-consumption-usage-2024-07-02T01-00-37.432.log.gz] [shipping_service]
2024-07-02T01:16:03.325Z [jfrt ] [INFO ] [1eaa77bc4c50f81e] [o.a.e.UploadServiceImpl:594 ] [ttp-nio-8081-exec-28] - Deploy to 'jfrog-usage-logs:New/artifactory/2024-07-02/949589f3ab0f-metadata-consumption-usage-2024-07-02T01-00-37.432.log.gz' Content-Length: 0 (estimation) artificial: false
2024-07-02T01:16:10.918Z [jfrt ] [INFO ] [e52c136cfd194e79] [aseBundleCleanupServiceImpl:84] [d194e79|art-exec-163] - Starting to cleanup incomplete Release Bundles
2024-07-02T01:16:10.940Z [jfrt ] [INFO ] [e52c136cfd194e79] [b.ReleaseBundleServiceImpl:485] [d194e79|art-exec-163] - Finished deleting orphan/unidentified items from _intransit repository
2024-07-02T01:16:10.940Z [jfrt ] [INFO ] [e52c136cfd194e79] [aseBundleCleanupServiceImpl:90] [d194e79|art-exec-163] - Finished incomplete Release Bundles cleanup