I’m trying to train a Hugging Face (HF) LLM locally in a SLURM environment. My parallelization choice is ZeRo (stage 1 or 2) with DeepSpeed. I have 8 GPUs and want to test with 1 node initially.

• Part of my SLURM (sbatch) file:

#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=12
#SBATCH --gpus=8
#SBATCH --gres=gpu:8

srun --ntasks=8 --cpus-per-task=12 --gpus-per-task=1 --gpu-bind=closest python3 test.py

• Training Arguments:

training_args = TrainingArguments(
    output_dir='./results',
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    num_train_epochs=1,
    logging_steps=1,
    save_steps=1,
    local_rank=int(os.getenv('LOCAL_RANK', '-1')),
    deepspeed="ds_config.json",
    fp16=False,
    dataloader_num_workers=4,
    report_to="none"
)

• The ds_config.json:

{
  "fp16": {
    "enabled": false
  },
  "optimizer": {
    "type": "AdamW",
    "params": {
      "lr": "auto",
      "betas": "auto",
      "eps": "auto",
      "weight_decay": "auto"
    }
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto",
      "warmup_num_steps": "auto"
    }
  },
  "zero_optimization": {
    "stage": 2,
    "allgather_partitions": true,
    "allgather_bucket_size": "auto",
    "reduce_scatter": true,
    "reduce_bucket_size": "auto",
    "overlap_comm": true,
    "contiguous_gradients": true,
    "cpu_offload": false
  },
  "gradient_accumulation_steps": "auto",
  "gradient_clipping": "auto",
  "steps_per_print": 2000,
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "wall_clock_breakdown": false
}

I have verified that all 8 GPUs are accessible when submitting a job. However, I am encountering an out-of-memory error because the system fails to detect the GPUs and defaults to using the CPU :

[WARNING] [real_accelerator.py:162:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.

How can I resolve this issue to ensure the GPUs are correctly detected and utilized during training?
Thank you in advance.