Loaded weights with error during training

[t626644510]

Self Checks

• This template is only for bug reports. For questions, please visit Discussions.
• I have thoroughly reviewed the project documentation (installation, training, inference) but couldn't find information to solve my problem. English 中文 日本語 Portuguese (Brazil)
• I have searched for existing issues, including closed ones. Search issues
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• Please do not modify this template and fill in all required fields.

Cloud or Self Hosted

Self Hosted (Source)

Environment Details

WSL2, python3.10, all envs build just follow the introduction from document.

Steps to Reproduce

The strange warning displayed every time when running the train.py(or webui from manage.py) either 1.5 or 1.4.3 here.
I'm nor sure is it common situation during train process and some friends said that's definitely abnormal.
Fig1 from 1.5 and Fig2 from 1.4.3.

✔️ Expected Behavior

everthing goes well

❌ Actual Behavior

The model file can be created (normally?) and merged.
But the train speed is quite slow and the warning will be displayed again when merging.
Furthermore, the accuracy from Top5 hardly increase slightly and the inferenced voice by this strange model is also weird.

[t626644510]

some issue said that's the problem from GPU but is there anyway to avoid or bypass it? How did this problem arise?QAQ