Coati SFT Training Process stack in "[extension] Compiling or loading the JIT-built cpu_adam kernel during runtime now" #3468
Unanswered
linmou
asked this question in
Community | Q&A
Replies: 1 comment 1 reply
-
|
My problem has been resolved. The solution is to change the ColossalAI installation method from the original 'CUDA_EXT=1 pip install colossalai 'Replace with Download From Source,' CUDA '_ EXT=1 pip install .’. I hope it can help everyone |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thanks for your help. But I tried 'CUDA_EXT=1 pip install git+https://github.com/hpcaitech/ColossalAI.git', and another error occurs 'cannot import name 'ZeroDDP' from 'colossalai.nn.parallel', which colossal version did you install ? |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
When I run Chat/examples/train_sft.sh several times, the process seems stacked somewhere.
No error reports, it just stops after showing "[extension] Compiling or loading the JIT-built cpu_adam kernel during runtime now". Any idea about why and where it stacks?
Beta Was this translation helpful? Give feedback.
All reactions