You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After update 100, the model begin to predict, while as shown above, the prediction has been running for over 24 hours without progress to new model update, is that normal? If not, what maybe the main cause of this problem? I'm using your code of issue #6 to do the data preprocessing, with train/validate/test 6:2:2, thanks!
The text was updated successfully, but these errors were encountered:
Hi @ae86208 the PD time (which is the data loading time) is quite large (~640s it should generally be < 5s) in your experiments. While testing this is going to take a huge amount of time since it will go over all batches. You will need to investigate why this is happening.
Maybe the hard drive you are using for the dataset has a lot of latency? In any case you can either
rewrite the data handler code for loading batches
modify it to load all the batches at once provided you have enough memory
After update 100, the model begin to predict, while as shown above, the prediction has been running for over 24 hours without progress to new model update, is that normal? If not, what maybe the main cause of this problem? I'm using your code of issue #6 to do the data preprocessing, with train/validate/test 6:2:2, thanks!
The text was updated successfully, but these errors were encountered: