Can't create tfds dataset #2

kr3kls · 2024-05-29T21:42:08Z

Running into a ValueError when I try to convert the dataset to tfrecord format. The error says In with name "event_id": Dtype int32 do not match <class 'numpy.int64'>

The example below uses event_id, but I also get the error with ef_number, and category.

Full stack trace and error dump

2024-05-29 17:07:57,579 - WARNING - From C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.

2024-05-29 17:07:57,706 - INFO - Generating dataset tornet (C:\Users\bduser\project_group_14\tornet-main\data\working\tornet\1.0.0)
Downloading and preparing dataset Unknown size (download: Unknown size, generated: Unknown size, total: Unknown size) to C:\Users\bduser\project_group_14\tornet-main\data\working\tornet\1.0.0...
Traceback (most recent call last):
File "C:\Users\bduser\project_group_14\tfds_preprocess.py", line 19, in
builder.download_and_prepare(**{'download_config':dl_config})
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\logging\init.py", line 168, in call
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\dataset_builder.py", line 691, in download_and_prepare
self._download_and_prepare(
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\dataset_builder.py", line 1584, in _download_and_prepare
future = split_builder.submit_split_generation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\split_builder.py", line 341, in submit_split_generation
return self._build_from_generator(**build_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\split_builder.py", line 417, in _build_from_generator
utils.reraise(e, prefix=f'Failed to encode example:\n{example}\n')
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\split_builder.py", line 415, in _build_from_generator
example = self._features.encode_example(example)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\features\features_dict.py", line 243, in encode_example
utils.reraise(
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\features\features_dict.py", line 241, in encode_example
example[k] = feature.encode_example(example_value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\bduser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow_datasets\core\features\tensor_feature.py", line 178, in encode_example
raise ValueError(
ValueError: Failed to encode example:
{'DBZ': array([[[[27.5, 27.5],
[24.5, 20.5],
[21. , 29. ],
...,
[38.5, 32. ],
[35. , 28.5],
[34.5, 31. ]],

[[25. , 26. ],
[28. , 29. ],
[23. , 31. ],
...,
[32. , 29.5],
[33.5, 27. ],
[35. , 29.5]],

[[24.5, 29. ],
[26.5, 32. ],
[28.5, 26. ],
...,
[30.5, 27. ],
[30.5, 26. ],
[30. , 23. ]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[27. , 32. ],
[25. , 32. ],
[26. , 32. ],
...,
[21. , 22. ],
[19.5, 18. ],
[23. , 20.5]],

[[30. , 29. ],
[27. , 31.5],
[27.5, 29. ],
...,
[21.5, 23. ],
[21.5, 21.5],
[25.5, 19. ]],

[[30.5, 27. ],
[26. , 27.5],
[26.5, 31. ],
...,
[23.5, 19. ],
[23. , 19. ],
[25.5, 18.5]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[29. , 24.5],
[27.5, 28.5],
[25.5, 30. ],
...,
[28.5, 19. ],
[29.5, 23.5],
[27. , 22.5]],

[[29.5, 28. ],
[23.5, 27.5],
[26. , 28.5],
...,
[20.5, 18.5],
[22. , 19. ],
[25.5, 21.5]],

[[24.5, 30.5],
[23. , 26. ],
[24.5, 27.5],
...,
[22. , 16. ],
[19.5, 19. ],
[21. , 14.5]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[22. , 28.5],
[27. , 19. ],
[23. , 20. ],
...,
[25.5, 15. ],
[26.5, 13.5],
[27. , 15.5]],

[[24. , 24. ],
[21.5, 24.5],
[25. , 21. ],
...,
[26. , 14. ],
[29.5, 16.5],
[28.5, 15. ]],

[[24.5, 23.5],
[25.5, 22.5],
[20.5, 26.5],
...,
[27. , 19. ],
[26. , 19. ],
[26. , 18.5]],

...,

[[ nan, nan],
[ nan, 4.5],
[ nan, 8. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 2. ],
[ nan, 6. ],
[ nan, 7. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 2. ],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]]], dtype=float32), 'VEL': array([[[[ 6.5, 13.5],
[10. , 13. ],
[10.5, 14. ],
...,
[18.5, 18.5],
[18.5, 19.5],
[19.5, 20.5]],

[[ 6.5, 11.5],
[ 9.5, 11. ],
[ 9. , 12.5],
...,
[17. , 21. ],
[16.5, 19. ],
[18.5, 21.5]],

[[ 6.5, 11.5],
[10. , 13. ],
[ 7.5, 13. ],
...,
[17. , 20.5],
[18.5, 21.5],
[17.5, 23. ]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[ 8. , 16.5],
[10. , 16. ],
[12. , 13.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[10. , 13.5],
[ 9.5, 14.5],
[11.5, 15. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 4.5, 13. ],
[ 7. , 12.5],
[ 9. , 13.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[ 6.5, 14. ],
[ 3.5, 14.5],
[ 9. , 10.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 7. , 11. ],
[ 5.5, 13. ],
[ 7.5, 13. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 8.5, 13.5],
[ 6. , 12. ],
[11. , 13. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[ 7.5, 15. ],
[ 4. , 17. ],
[ 7.5, 12.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 8. , 18. ],
[ 7. , 16.5],
[ 4. , 15.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[10.5, 15.5],
[ 6.5, 15.5],
[ 5.5, 15.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 33.5],
[ nan, 32. ],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 33. ],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]]], dtype=float32), 'KDP': array([[[[-0.1484375, -0.046875 ],
[-0.203125 , -0.046875 ],
[-0.1484375, -0.1015625],
...,
[ 0.296875 , 0.046875 ],
[ 0.296875 , 0.046875 ],
[ 0.296875 , 0.046875 ]],

[[-0.1484375, -0.046875 ],
[-0.203125 , -0.046875 ],
[-0.1484375, -0.1015625],
...,
[ 0.296875 , 0.046875 ],
[ 0.296875 , 0.046875 ],
[ 0.296875 , 0.046875 ]],

[[-0.1484375, -0.046875 ],
[-0.203125 , 0. ],
[-0.203125 , 0.046875 ],
...,
[ 0.046875 , 0.3984375],
[ 0.046875 , 0.3984375],
[ 0.046875 , 0.3984375]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[ 0. , -0.046875 ],
[ 0. , -0.046875 ],
[ 0. , -0.046875 ],
...,
[ 0.1015625, -0.1484375],
[ 0.1015625, -0.1484375],
[ 0.1015625, -0.1015625]],

[[ 0. , -0.046875 ],
[ 0. , -0.046875 ],
[ 0. , -0.046875 ],
...,
[ 0.1015625, -0.1484375],
[ 0.1015625, -0.1484375],
[ 0.1015625, -0.1015625]],

[[ 0.046875 , 0. ],
[ 0.046875 , 0. ],
[ 0.046875 , 0. ],
...,
[-0.046875 , 0.3515625],
[-0.046875 , 0.3515625],
[-0.046875 , 0.296875 ]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
...,
[ 0.25 , -0.203125 ],
[ 0.25 , -0.203125 ],
[ 0.25 , -0.203125 ]],

[[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
...,
[ 0.25 , -0.203125 ],
[ 0.25 , -0.203125 ],
[ 0.25 , -0.203125 ]],

[[-0.046875 , 0.1015625],
[-0.046875 , 0.1015625],
[-0.046875 , 0.1015625],
...,
[ 0. , -0.1015625],
[ 0. , -0.1015625],
[ 0. , -0.1484375]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
...,
[ 0. , -0.3515625],
[ 0. , -0.3515625],
[ 0. , -0.3515625]],

[[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
[-0.046875 , 0.046875 ],
...,
[ 0. , -0.3515625],
[ 0. , -0.3515625],
[ 0. , -0.3515625]],

[[ 0.1015625, 0.1015625],
[ 0.1015625, 0.1015625],
[ 0.1015625, 0.1015625],
...,
[-0.1015625, -0.046875 ],
[-0.1015625, -0.046875 ],
[-0.1484375, -0.046875 ]],

...,

[[ nan, 0. ],
[ nan, 0. ],
[ nan, 0. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]]], dtype=float32), 'RHOHV': array([[[[0.9951172 , 0.98828125],
[0.9980469 , 0.95214844],
[0.96875 , 0.99121094],
...,
[0.99121094, 0.9716797 ],
[0.96191406, 0.9785156 ],
[0.96484375, 0.9814453 ]],

[[0.9814453 , 0.9746094 ],
[0.9980469 , 0.99121094],
[0.9814453 , 0.99121094],
...,
[0.9814453 , 0.9785156 ],
[0.98828125, 0.98535156],
[0.98828125, 0.98828125]],

[[0.98535156, 0.9785156 ],
[0.9951172 , 0.9951172 ],
[0.9951172 , 0.98535156],
...,
[0.98828125, 0.9550781 ],
[0.99121094, 0.9716797 ],
[0.9716797 , 0.9550781 ]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[0.9951172 , 0.99121094],
[0.9951172 , 0.9746094 ],
[0.99121094, 0.9580078 ],
...,
[0.9716797 , 0.9951172 ],
[0.90527344, 0.9980469 ],
[0.96191406, 0.9785156 ]],

[[0.9951172 , 0.9814453 ],
[0.99121094, 0.98828125],
[0.9980469 , 0.96484375],
...,
[0.9785156 , 1.0048828 ],
[0.99121094, 0.9980469 ],
[0.9580078 , 0.99121094]],

[[0.9951172 , 0.9746094 ],
[0.98828125, 0.9746094 ],
[0.9814453 , 0.9746094 ],
...,
[0.9013672 , 0.98535156],
[0.9785156 , 0.98828125],
[0.9716797 , 0.9785156 ]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[0.9951172 , 0.9453125 ],
[0.9951172 , 0.95214844],
[0.99121094, 0.9716797 ],
...,
[0.98535156, 0.99121094],
[0.98535156, 0.96875 ],
[0.9785156 , 0.95214844]],

[[0.9951172 , 0.99121094],
[0.9951172 , 0.9814453 ],
[0.9951172 , 0.9716797 ],
...,
[0.9716797 , 0.9580078 ],
[1.0019531 , 0.9951172 ],
[0.99121094, 0.96191406]],

[[0.98828125, 0.9980469 ],
[0.98828125, 0.9345703 ],
[0.9980469 , 0.9121094 ],
...,
[0.99121094, 0.9746094 ],
[0.9716797 , 1.0048828 ],
[0.9814453 , 0.9453125 ]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[0.99121094, 0.9980469 ],
[1.0019531 , 0.98535156],
[0.9951172 , 0.98535156],
...,
[0.99121094, 0.9814453 ],
[0.98828125, 0.98535156],
[0.9482422 , 0.98535156]],

[[0.99121094, 0.99121094],
[0.99121094, 0.98828125],
[0.9951172 , 0.99121094],
...,
[0.9580078 , 0.98828125],
[0.9785156 , 0.9453125 ],
[0.9453125 , 0.82128906]],

[[0.9951172 , 0.9580078 ],
[0.9980469 , 0.9951172 ],
[0.98535156, 0.9951172 ],
...,
[0.91796875, 0.9716797 ],
[0.98828125, 0.9550781 ],
[0.96875 , 0.88183594]],

...,

[[ nan, nan],
[ nan, 0.9951172 ],
[ nan, 1.0517578 ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 0.921875 ],
[ nan, 1.0517578 ],
[ nan, 1.0517578 ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 1.0517578 ],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]]], dtype=float32), 'ZDR': array([[[[ 0.375 , 0.75 ],
[ 0.5625, 0.75 ],
[-0.125 , 0.375 ],
...,
[ 0.3125, -0.5 ],
[ 0.6875, 0.3125],
[-0.375 , -0.5 ]],

[[-0.375 , 1.9375],
[ 0.3125, 1.125 ],
[ 0.4375, -0.1875],
...,
[ 1.625 , 0.25 ],
[ 2.3125, 0. ],
[ 0.5625, -1.8125]],

[[-0.1875, 0.5 ],
[ 0.5 , 0.375 ],
[ 0.125 , 0.3125],
...,
[ 0.25 , 1.375 ],
[ 1.375 , -0.25 ],
[ 0.75 , -1.8125]],

...,

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]]],

[[[ 0. , 0.3125],
[ 0.3125, 0.5 ],
[-0.125 , 0.1875],
...,
[-0.375 , 0.5625],
[ 0.5625, -0.125 ],
[ 0.3125, 0.4375]],

[[ 0. , 1.8125],
[-0.125 , 1.125 ],
[-0.0625, 0.6875],
...,
[ 0. , 0.3125],
[ 0.25 , 0.3125],
[-0.125 , 0.75 ]],

[[ 0.375 , -0.75 ],
[ 0.125 , -0.9375],
[-0.625 , 0.9375],
...,
[-0.3125, -0.4375],
[-0.125 , 0.1875],
[ 0.25 , -0.375 ]],

...,

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]]],

[[[ 0.1875, 1.375 ],
[ 0.5625, 1.5625],
[ 0.5625, 1.5 ],
...,
[-0.0625, 1. ],
[ 0.5625, -0.75 ],
[ 0.8125, 2.4375]],

[[ 0.0625, 0.1875],
[ 0.375 , 0.6875],
[ 0.1875, 0. ],
...,
[ 0.0625, 0.875 ],
[ 0.6875, -0.6875],
[ 0.1875, -0.1875]],

[[ 0.5625, 0.0625],
[ 0.25 , 1. ],
[-0.4375, -0.875 ],
...,
[-0.375 , -1.4375],
[-0.375 , 0.9375],
[ 0.4375, 0.875 ]],

...,

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]]],

[[[ 0.8125, 0.125 ],
[ 0. , -0.875 ],
[ 0.0625, 0.25 ],
...,
[ 0. , -1.625 ],
[-0.4375, -0.1875],
[ 0.3125, -1.9375]],

[[-0.375 , 0.125 ],
[-0.0625, 1.4375],
[ 0.6875, -0.3125],
...,
[ 1.8125, 0. ],
[ 0.25 , -1.8125],
[-1.3125, 0. ]],

[[ 0.375 , 1.5 ],
[-0.1875, -0.5 ],
[ 0. , 0.125 ],
...,
[ 1.75 , -0.625 ],
[ 0. , 0.125 ],
[ 0.8125, 3.0625]],

...,

[[-8. , -8. ],
[-8. , 0.75 ],
[-8. , -0.4375],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -1.875 ],
[-8. , 2. ],
[-8. , 0.75 ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]],

[[-8. , -2.25 ],
[-8. , -8. ],
[-8. , -8. ],
...,
[-8. , -8. ],
[-8. , -8. ],
[-8. , -8. ]]]], dtype=float32), 'WIDTH': array([[[[ 4. , 2. ],
[ 3.5, 2. ],
[ 5. , 2.5],
...,
[ 2. , 3. ],
[ 1.5, 2. ],
[ 2.5, 2.5]],

[[ 2.5, 3. ],
[ 4. , 3.5],
[ 4. , 3.5],
...,
[ 3. , 2.5],
[ 2.5, 1.5],
[ 3.5, 3. ]],

[[ 4. , 3.5],
[ 4. , 4. ],
[ 4. , 3. ],
...,
[ 3.5, 1.5],
[ 3.5, 2.5],
[10.5, 2.5]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[ 5. , 2. ],
[ 4.5, 3.5],
[ 5. , 5. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 4.5, 5. ],
[ 5. , 3. ],
[ 4. , 3.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 4.5, 5.5],
[ 3.5, 3.5],
[ 4.5, 5. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[ 5. , 5. ],
[ 5.5, 4.5],
[ 4. , 5. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 4. , 5. ],
[ 5. , 4.5],
[ 4. , 3.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 4.5, 4.5],
[ 4.5, 5.5],
[ 5.5, 3. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]],

[[[ 5.5, 4.5],
[ 3. , 5.5],
[ 6. , 6.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 5.5, 5.5],
[ 5.5, 5. ],
[ 5. , 6. ],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ 5. , 6. ],
[ 6. , 6. ],
[ 4.5, 3.5],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

...,

[[ nan, nan],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 1. ],
[ nan, 1.5],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]],

[[ nan, 0.5],
[ nan, nan],
[ nan, nan],
...,
[ nan, nan],
[ nan, nan],
[ nan, nan]]]], dtype=float32), 'range_folded_mask': array([[[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

...,

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]]],

[[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

...,

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]]],

[[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

...,

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]]],

[[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[1., 1.],
[1., 1.],
[1., 1.]],

...,

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]],

[[0., 0.],
[0., 0.],
[0., 0.],
...,
[0., 0.],
[0., 0.],
[0., 0.]]]], dtype=float32), 'label': array([0, 0, 0, 0], dtype=uint8), 'category': array([1]), 'event_id': array([480440]), 'ef_number': array([-1]), 'az_lower': array([330.], dtype=float32), 'az_upper': array([390.], dtype=float32), 'rng_lower': array([118160.], dtype=float32), 'rng_upper': array([178160.], dtype=float32), 'time': array([1381187670, 1381187970, 1381188270, 1381188570], dtype=int64)}
In with name "event_id":
Dtype int32 do not match <class 'numpy.int64'>}

Steps to reproduce:

I downloaded all of the dataset with zenodo.
All of the hashes match.
My environment variables are set.
I copied the code from the /tornet-main/tornet/data/tfds/tornet/README.md for the tfrecord format and created tfds_preprocess.py file.
Run python .\tfds_preprocess.py

Installed Software/Packages:
Python 3.11.9 on Windows 11

Basic requirements
numpy 1.26.4
matplotlib 3.9.0
pandas 2.2.2
xarray 2024.5.0
netCDF4 1.6.5
tqdm 4.66.4
glob2 0.7
jupyter 1.0.0

Tensorflow requirements
Tensorflow 2.15.0
Keras 2.15.0
tensorflow_datasets 4.9.4

Any ideas on how to get past this?

markveilletteLL · 2024-06-06T19:38:03Z

Hello,

@kr3kls If you change lines 53-55 in tornet.data.loader here

tornet/tornet/data/loader.py

Line 53 in e3505af

    
           data['category']=np.array([{'TOR':0,'NUL':1,'WRN':2}[ds.attrs['category']]]) # tornadic, null (random), or warning

to the following:

data['category']=np.array([{'TOR':0,'NUL':1,'WRN':2}[ds.attrs['category']]],dtype=np.int64) # tornadic, null (random), or warning
data['event_id']=np.array([int(ds.attrs['event_id'])],dtype=np.int64)
data['ef_number']=np.array([int(ds.attrs['ef_number'])],dtype=np.int64)

does it help?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't create tfds dataset #2

Can't create tfds dataset #2

kr3kls commented May 29, 2024

markveilletteLL commented Jun 6, 2024 •

edited

Loading

Can't create tfds dataset #2

Can't create tfds dataset #2

Comments

kr3kls commented May 29, 2024

markveilletteLL commented Jun 6, 2024 • edited Loading

markveilletteLL commented Jun 6, 2024 •

edited

Loading