Train model in slurm cluster.

[mrbeann]

I try to use colossalAI in a cluster managed by slurm. I first open a shell through a command like srun --pty /bin/bash. Then I try the starter example. However it raises the following error,

INFO     colossalai - paramiko.transport - INFO: Connected (version 2.0, client OpenSSH_7.4)
Error: failed to run torchrun --nproc_per_node=1 --nnodes=1 --node_rank=0 --rdzv_backend=c10d --rdzv_endpoint=127.0.0.1:29500 --rdzv_id=colossalai-default-job train.py on 127.0.0.1

Is there any idea about this?

[kurisusnowdeng]

Hi @mrbeann , I don't think torchrun can work well with slurm. To deal with this kind of issues, we provide our own launcher for the slurm platform. Please refer to https://colossalai.org/docs/basics/launch_colossalai#launch-with-slurm and have a try.