Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests and ci #2

Merged
merged 23 commits into from
Oct 11, 2023
Merged

Tests and ci #2

merged 23 commits into from
Oct 11, 2023

Conversation

trey-stafford
Copy link
Contributor

@trey-stafford trey-stafford commented Oct 4, 2023

  • Setup pre-commit to do linting/formatting
  • Move relevant tool/project config to pyproject.toml
  • Setup GitHub Actions workflow for running typechecker (mypy), unit tests, and pre-commit hooks.
  • Format existing source files, clean up unused imports, fix missing imports, etc.

Resolved the issue below by merging in Scott's latest changes!


Getting a failure in an integration test designed to ensure that this library's implementation of compute_initial_daily_ecdr_dataset return the same result as pm_icecon's:

$ PYTHONPATH=/home/trst2284/code/pm_tb_data/:/home/trst2284/code/pm_icecon python -m pytest -s
================================ test session starts ================================
platform linux -- Python 3.10.12, pytest-7.4.2, pluggy-1.3.0
rootdir: /home/trst2284/code/seaice_ecdr
configfile: pytest.ini
collected 7 items

seaice_ecdr/tests/integration/test_initial_daily_ecdr_generation.py ...F
seaice_ecdr/tests/unit/test_initial_daily_ecdr.py ...

===================================== FAILURES ======================================
__________________ test_seaice_idecdr_and_pmicecon_conc_identical ___________________

sample_pmicecon_dataset = <xarray.Dataset>
Dimensions:  (y: 896, x: 608)
Dimensions without coordinates: y, x
Data variables:
    conc     (y, x) float64 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0
sample_idecdr_dataset = <xarray.Dataset>
Dimensions:                (time: 1, y: 896, x: 608)
Coordinates:
  * time                   (time) i...  missing_value:        255
    time_coverage_start:  2021-04-05 00:00:00
    time_coverage_end:    2021-04-05 23:59:59

    def test_seaice_idecdr_and_pmicecon_conc_identical(
        sample_pmicecon_dataset, sample_idecdr_dataset
    ):
        """Test that pm_icecon yields a 'conc' field."""
        pmicecon_conc_ds = sample_pmicecon_dataset
        pmi_conc_field = np.array(pmicecon_conc_ds.variables['conc'])

        ide_conc_ds = sample_idecdr_dataset
        ide_conc_field = np.squeeze(np.array(ide_conc_ds.variables['conc']))

        # We know that the original conc field has zeros where TBs were not
        # available, so only check where idecdr is not nan
        indexes_to_check = ~np.isnan(ide_conc_field)
>       assert_equal(
            pmi_conc_field[indexes_to_check],
            ide_conc_field[indexes_to_check],
        )
E       AssertionError:
E       Arrays are not equal
E
E       Mismatched elements: 28097 / 544712 (5.16%)
E       Max absolute difference: 94.64499699
E       Max relative difference: 1.
E        x: array([0., 0., 0., ..., 0., 0., 0.])
E        y: array([0., 0., 0., ..., 0., 0., 0.])

seaice_ecdr/tests/integration/test_initial_daily_ecdr_generation.py:98: AssertionError
============================== short test summary info ==============================
FAILED seaice_ecdr/tests/integration/test_initial_daily_ecdr_generation.py::test_seaice_idecdr_and_pmicecon_conc_identical - AssertionError:
============================ 1 failed, 6 passed in 3.96s ============================

This error appears when using this repo's add_qa_field branch and pm_icecon's update_for_nise_cdr_cetb, so I don't think this PR is to blame. Just recording this here until we track down the source of the issue.

Make more consistent with the `pm_tb_data` repo. We should probably think about
re-ogranizing some of these docs into individual files. The README should really
just cover front-matter that people need at a glace. A `doc/` directory with
individual sections for e.g., development/contributing, testing, etc. (ideally
served by e.g., RTD) would be ideal.
* Line length too long
* Add missing import
And move config from `mypy.ini` to `pyproject.toml` - consistent with other tool configuration
Base automatically changed from use-pm-tb-data to main October 6, 2023 22:16
Ignore some things. This code will likely be refactored greatly over the coming
weeks anyway.
@trey-stafford trey-stafford requested a review from sc0tts October 6, 2023 23:00
No unit tests defined yet.
Eventually this will get set to `main` and we'll have development of
`seaice_ecdr` track along with the main branches of `pm_tb_data` and `pm_icecon`.
Experiment with checking out multiple repos in GHA
Removed in favor of using `pyproject.toml`
Updates for the `update_for_nise_cdr_cetb` branch of `pm_icecon`
@trey-stafford trey-stafford merged commit 01e94f8 into main Oct 11, 2023
1 check passed
@trey-stafford trey-stafford deleted the tests-and-ci branch October 11, 2023 22:38
nsidc-service-admin pushed a commit that referenced this pull request Jan 9, 2024
Not sure what the hdf5 library error is, but it appears to have to do with
`dask` and using `open_mfdataset`.

This gets repeated:

```
HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 4:
  #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute
    major: Attribute
    minor: Can't open object
  #1: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #2: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed
    major: Virtual Object Layer
    minor: Can't open object
  #3: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute
    major: Attribute
    minor: Can't open object
  #4: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header
    major: Attribute
    minor: Unable to initialize object
  #5: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'
    major: Attribute
    minor: Object not found
```

By telling dask not to use more than one thread per worker, these errors go
away. See https://stackoverflow.com/questions/72821108/hdf5-warnings-when-accessing-xarray-dataset
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant