I am nab , a new of Talend. We are operatiing with an on-premises of 2 Tac servers (active-active) and 2 jobservers . Currently 25 tasks are running simultaneously on each job server , but occasionally a job server goes down because of its out of memory error on jvm.
When the workload becomes too concentrated on one job server, it may cause it to go down.
Therefore, I want to limit the number of concurrent executions on each job server to around 20. Is it possible to set this limit value somewhere in the conf settings?
The memory of this job server is 32 GB, but it seems to momentarily exceed this limit. Since increasing memory is difficult due to hardware constraints, we are considering if it’s possible to control the number of concurrent executions through configuration settings.
If you know of an effective way to configure this, please let me know. Thanks
/jobserver status when down/
systemctl status talend-rjs-8.0.1
Loaded: loaded (/etc/systemd/system/talend-rjs-8.0.1.service; enabled; preset: disabled)
Active: failed (Result: oom-kill) since Wed 2024-12-11 10:45:38 JST; 21min ago
Duration: 1w 19h 18min 42.874s
,,,
systemd[1]: talend-rjs-8.0.1.service: A process of this unit has been killed by the OOM killer.
systemd[1]: talend-rjs-8.0.1.service: Main process exited, code=exited, status=137/n/a
systemd[1]: talend-rjs-8.0.1.service: Failed with result ‘oom-kill’.
systemd[1]: talend-rjs-8.0.1.service: Consumed 2month 2w 6d 17min 21.810s CPU time.
/ccurrent setting/
which configuration should be change? please give me your advice.
[ conf]# cat TalendJobServer.properties |grep MAX
org.talend.remote.server.MultiSocketServer.MAX_CONCURRENT_CONNECTIONS=1000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_NORMAL_CASE=300000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_ABNORMAL_CASE=86400000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_DURATION_BEFORE_CLEANING_OLD_EXECUTIONS_LOGS=7776000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_OLD_EXECUTIONS_LOGS=1000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_FILE_LISTENERS=6000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_LIBRARY_DESC_NB=1000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_LIBRARY_DESC_SIZE=100KB
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_DURATION_BEFORE_CLEANING_OLD_JOBS=7776000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_NB=6000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_OLD_JOBS=200
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_FILE_SIZE=1G
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_SIZE=1G
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIPPED_ENTRIES=2048
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ARCHIVES_DIR_SIZE=100G
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIP_NAME_LENGTH=240
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_FOLDER_NAME_LENGTH=240
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_UNZIPPED_FILE_NAME_LENGTH=240
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_ZIP_DEPTH=64
[ conf]# cat TalendJobServer.properties |grep JOB
org.talend.remote.jobserver.server.TalendJobServer.EXECUTE_UNSIGNED_JOB_TOKEN=
org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_ARCHIVE_SIGNATURE_CHECK=ON_UPLOAD
org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH=
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MIN_NUMBER_JOB_EXECUTIONS_BEFORE_RELEASE=50
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_NORMAL_CASE=300000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_DURATION_BEFORE_JOB_EXECUTION_RELEASE_ABNORMAL_CASE=86400000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_DURATION_BEFORE_CLEANING_OLD_JOBS=7776000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_NB=6000
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_OLD_JOBS=200
org.talend.remote.jobserver.commons.config.JobServerConfiguration.RESTORE_JOBTABLE=true
org.talend.remote.jobserver.commons.config.JobServerConfiguration.MAX_JOB_FILE_SIZE=1G
our versions are..
Tac version : Patch_20230915_TPS-5495_v1-8.0.1
Jobserver : 8.0.1.20230928_1351_patch
Go into your talend job server folder and find the properties file.
WEB-INF/classes/quartz.properties
You need to either add / modify this :
org.quartz.threadPool.threadCount
And set it to 20.
Restart your job server and you’re good to go.
Additionnaly you can also search to optimize RAM usage using the Xms and Xmx parameters of the JVM for each job and also in the TAC configuration check that the task distribution is optimized and distributed evenly between the 2 jobs servers
source : https://help.qlik.com/talend/en-US/installation-guide-linux/8.0/recommended-configuration