when I submitted the first job to yarn cluster , the job is running ,and then submitted the second job, the second job state never run until the first job completed either success or fail. means not able to run 5 jobs concurrently and jobs are accepted mode.
Below is the issue comes in running jobs.
diagnostics: [Mon May 20 03:25:09 -0400 2024] Application is added to the scheduler and is not yet activated. Queue’s AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; AM Resource Request = <memory:8192, vCores:1>; Queue Resource Limit for AM = <memory:8192, vCores:1>; User AM Resource Limit of the queue = <memory:8192, vCores:1>; Queue AM Resource Usage = <memory:8192, vCores:1>
Is there any way to run multiple spark jobs parallel on yarn with cluster mode?
skzaid ahmad is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.