-
|
Hi！ |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
not offensive at all. the matbench discovery test set (we call it WBM) is public so everybody can test their model on it. this was done internally by DeepMind as we do not have access to GNoME. in case you're asking if there's any overlap between the GNoME training dataset and our test dataset, the answer is likely yes given that both datasets were generated with element substitution from MP (and in the case of GNoME also OQMD) parent structures. the substitution protocols differ in that GNoME uses a more general partial Wyckoff site substitution which can create new prototypes (see SAPS in the GNoME paper). to somewhat control for the likely overlap with WBM GNoME was evaluated twice: once on the complete test set and once only on WBM compositions that do not appear in the GNoME training set. the performance in both cases was similar, which suggests data leakage is not the main reason for GNoME's good performance. if you're wondering what influence overlapping structure prototypes have, that's more difficult to assess as the SAPS substitution could mean GNoME training data has complete coverage of the prototypes in WBM. but a recent re-analysis of all open models in matbench discovery that excluded structures from the test set with a matching prototype in MP showed little difference in the model metrics (except for DAF). this suggests prototype overlap is not a significant factor for thermodynamic stability prediction. see the relevant section in the paper. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your comprehensive and detailed reply! |
Beta Was this translation helpful? Give feedback.
not offensive at all. the matbench discovery test set (we call it WBM) is public so everybody can test their model on it. this was done internally by DeepMind as we do not have access to GNoME.
in case you're asking if there's any overlap between the GNoME training dataset and our test dataset, the answer is likely yes given that both datasets were generated with element substitution from MP (and in the case of GNoME also OQMD) parent structures. the substitution protocols differ in that GNoME uses a more general partial Wyckoff site substitution which can create new prototypes (see SAPS in the GNoME paper).
to somewhat control for the likely overlap with WBM GNoME was evaluated twice: on…