upgrade Finch to 0.8.0 #231

huard · 2022-01-13T21:37:05Z

Overview

Please include a summary of the changes and which issues are fixed.

Please also include relevant motivation and context.

List any dependencies that are required for this change.

Changes

Non-breaking changes

Updated changelog by @matprov
Add hourly_to_daily process by @huard
Avoid annoying warnings by updating birdy (environment-docs) by @huard
Upgrade to clisops 0.8.0 to accelerate spatial averages over regions.
Upgrade to xesmf 0.6.2 to fix spatial averaging bug not weighing correctly cells with varing areas.
Update to PyWPS 4.5.1 to allow the creation of recursive directories for outputs.

Breaking changes

Related Issue / Discussion

Should allow custom nested directories to be configured for outputs (see Finch and Raven not creating their wps_output under /bird_name #11 & Fix birds not creating their wps output under each bird name #203)
Corresponding notebook changes: Use catalog from prod and fix output url format change from Finch 0.8.0 Ouranosinc/pavics-sdi#244

Additional Information

crim-jenkins-bot · 2022-01-13T22:09:55Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/817/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : finch-v0.8.0
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-8.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/697/

NOTEBOOK TEST RESULTS

    
[2022-01-13T21:51:10.447Z] ============================= test session starts ==============================
[2022-01-13T21:51:10.447Z] platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
[2022-01-13T21:51:10.447Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2022-01-13T21:51:10.447Z] plugins: anyio-3.3.4, dash-2.0.0, nbval-0.9.6, tornasync-0.6.0.post2
[2022-01-13T21:51:10.447Z] collected 238 items
[2022-01-13T21:51:10.447Z] 
[2022-01-13T21:51:17.253Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2022-01-13T21:51:50.740Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2022-01-13T21:52:01.913Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2022-01-13T21:52:12.829Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 13%]
[2022-01-13T21:52:15.575Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 17%]
[2022-01-13T21:52:30.498Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb FFFFFFFFF.          [ 21%]
[2022-01-13T21:52:48.799Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 22%]
[2022-01-13T21:53:00.231Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb FFFFFF           [ 24%]
[2022-01-13T21:53:01.174Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 25%]
[2022-01-13T21:53:07.452Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 28%]
[2022-01-13T21:53:15.014Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 30%]
[2022-01-13T21:57:14.199Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 36%]
[2022-01-13T21:58:17.505Z] ..............                                                           [ 42%]
[2022-01-13T21:58:25.633Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 44%]
[2022-01-13T21:58:28.242Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 47%]
[2022-01-13T21:58:55.392Z] .......F.F.......                                                        [ 54%]
[2022-01-13T21:59:02.102Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 56%]
[2022-01-13T21:59:04.022Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 57%]
[2022-01-13T21:59:04.550Z] .FFFFFFFF.                                                               [ 61%]
[2022-01-13T21:59:16.689Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 65%]
[2022-01-13T21:59:26.366Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 68%]
[2022-01-13T22:00:19.026Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 76%]
[2022-01-13T22:00:20.420Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2022-01-13T22:00:25.828Z] ......                                                                   [ 79%]
[2022-01-13T22:00:43.939Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2022-01-13T22:01:00.330Z] .............                                                            [ 85%]
[2022-01-13T22:01:12.568Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2022-01-13T22:01:53.259Z] ....s.                                                                   [ 88%]
[2022-01-13T22:02:08.273Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2022-01-13T22:02:12.978Z] ...                                                                      [ 89%]
[2022-01-13T22:02:39.538Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2022-01-13T22:04:54.885Z] ......                                                                   [ 92%]
[2022-01-13T22:04:54.886Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2022-01-13T22:08:18.498Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2022-01-13T22:08:18.499Z] 
[2022-01-13T22:08:18.499Z] =================================== FAILURES ===================================

tlvu · 2022-01-14T17:03:42Z

@tlogan2000 You've asked to keep PAVICS unchanged before your demo in a week. Are you only using xclim in the Jupyter env or you'll also use Finch. If yes, do you want to audit this Finch upgrade?

tlvu

climex, forecasts and subset-user-input notebook failure fixed in PR Ouranosinc/pavics-sdi#244. The weaver one is a known failure.

PR good to go. Don't forget the usual release note update, bump version process and tagging after merge.

I'd also wait for an OK from Travis. He has requested for the env not to change for his upcoming demo.

tlvu · 2022-01-14T17:13:07Z

run tests

crim-jenkins-bot · 2022-01-14T17:45:35Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/818/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : finch-v0.8.0
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-8.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/698/

NOTEBOOK TEST RESULTS

    
[2022-01-14T17:26:31.126Z] ============================= test session starts ==============================
[2022-01-14T17:26:31.126Z] platform linux -- Python 3.7.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
[2022-01-14T17:26:31.126Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2022-01-14T17:26:31.126Z] plugins: anyio-3.3.4, dash-2.0.0, nbval-0.9.6, tornasync-0.6.0.post2
[2022-01-14T17:26:31.126Z] collected 238 items
[2022-01-14T17:26:31.126Z] 
[2022-01-14T17:26:37.341Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2022-01-14T17:27:05.919Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  7%]
[2022-01-14T17:27:17.455Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 10%]
[2022-01-14T17:27:28.382Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 13%]
[2022-01-14T17:27:30.421Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 17%]
[2022-01-14T17:29:07.087Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ..........          [ 21%]
[2022-01-14T17:29:25.268Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 22%]
[2022-01-14T17:29:53.645Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 24%]
[2022-01-14T17:29:55.032Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 25%]
[2022-01-14T17:30:01.426Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 28%]
[2022-01-14T17:30:09.720Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 30%]
[2022-01-14T17:33:38.671Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 36%]
[2022-01-14T17:34:36.545Z] ..............                                                           [ 42%]
[2022-01-14T17:34:44.300Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 44%]
[2022-01-14T17:34:46.522Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 47%]
[2022-01-14T17:35:09.665Z] .................                                                        [ 54%]
[2022-01-14T17:35:17.130Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 56%]
[2022-01-14T17:35:18.523Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 57%]
[2022-01-14T17:35:19.307Z] .FFFFFFFF.                                                               [ 61%]
[2022-01-14T17:35:32.222Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 65%]
[2022-01-14T17:35:41.963Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 68%]
[2022-01-14T17:36:34.389Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [ 76%]
[2022-01-14T17:36:35.335Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 76%]
[2022-01-14T17:36:39.543Z] ......                                                                   [ 79%]
[2022-01-14T17:37:01.585Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 79%]
[2022-01-14T17:37:14.196Z] .............                                                            [ 85%]
[2022-01-14T17:37:24.184Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 85%]
[2022-01-14T17:38:03.172Z] ....s.                                                                   [ 88%]
[2022-01-14T17:38:15.404Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 88%]
[2022-01-14T17:38:20.926Z] ...                                                                      [ 89%]
[2022-01-14T17:38:42.891Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 90%]
[2022-01-14T17:40:49.751Z] ......                                                                   [ 92%]
[2022-01-14T17:40:51.026Z] notebooks/hummingbird.ipynb ............                                 [ 97%]
[2022-01-14T17:44:16.124Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2022-01-14T17:44:16.124Z] 
[2022-01-14T17:44:16.125Z] =================================== FAILURES ===================================

tlvu · 2022-01-31T15:54:37Z

Talked to @aulemahal this morning. He's thinking about releasing a newer Finch so maybe we don't merge this PR but wait for his newer Finch.

Do we have a need for this Finch right now or it can wait.

huard · 2022-01-31T18:14:30Z

My main concern is allowing progress on the related issues outlined in the description. If you're not going to work on them before Pascal's next release, then the delay doesn't matter.

tlvu · 2022-01-31T23:48:09Z

My main concern is allowing progress on the related issues outlined in the description. If you're not going to work on them before Pascal's next release, then the delay doesn't matter.

Yes I have a few other things on my plate but I do not need a real go-live to be able to test the other related issues.

crim-jenkins-bot · 2022-02-02T16:30:02Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/826/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : finch-v0.8.0
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-91.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/707/

NOTEBOOK TEST RESULTS

    
</code>

tlvu · 2022-02-03T15:54:46Z

I tested this new Finch with new wps config to create the different dirs:

birdhouse-deploy/birdhouse/config/finch/wps.cfg.template

Lines 2 to 8 in a4ba10e

    
           outputurl = https://${PAVICS_FQDN_PUBLIC}/wpsoutputs/finch 
        
           outputpath = /data/wpsoutputs/finch 
        
           # link means hardlink so both temp_path, workdir and outputpath should be on the same partition for hardlink to work 
        
           storage_copy_function = link 
        
           temp_path = /data/wpsoutputs/finch/tmp 
        
           workdir = /data/wpsoutputs/finch/workdir

Unfortunately I am getting the following errors:

$ docker logs finch                                                                                                           
[2022-02-03 15:37:30 +0000] [1] [INFO] Starting gunicorn 20.1.0                                                                                       
[2022-02-03 15:37:30 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)                                                                          
[2022-02-03 15:37:30 +0000] [1] [INFO] Using worker: sync                                                                                             
[2022-02-03 15:37:30 +0000] [6] [INFO] Booting worker with pid: 6                                                                                     
server->workdir configuration value /data/wpsoutputs/finch/workdir is not directory                                                                   
server->outputpath configuration value /data/wpsoutputs/finch is not directory                                                                        
[2022-02-03 15:38:00 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:6)                                                                                     
[2022-02-03 15:38:00 +0000] [6] [INFO] Worker exiting (pid: 6)                                                                                        
[2022-02-03 15:38:01 +0000] [1] [WARNING] Worker with pid 6 was terminated due to signal 9

Will diagnose and report further findings.

tlvu · 2022-02-03T16:20:37Z

Tried a notebook on Finch just in case the error about dir not created are false positive, confirmed the error about dir creation is real.

2022-02-03 16:18:29,720 - PYWPS - ERROR - Exception: code: 500, description: No applicable error code, please check error log., locator:
Traceback (most recent call last):
  File "/opt/conda/envs/finch/lib/python3.9/site-packages/pywps/app/Service.py", line 314, in call
    raise e
  File "/opt/conda/envs/finch/lib/python3.9/site-packages/pywps/app/Service.py", line 304, in call
    response = self.execute(
  File "/opt/conda/envs/finch/lib/python3.9/site-packages/pywps/app/Service.py", line 81, in execute
    process = self.prepare_process_for_execution(identifier)
  File "/opt/conda/envs/finch/lib/python3.9/site-packages/pywps/app/Service.py", line 97, in prepare_process_for_execution
    tempdir = tempfile.mkdtemp(prefix='pywps_process_', dir=workdir)
  File "/opt/conda/envs/finch/lib/python3.9/tempfile.py", line 363, in mkdtemp
    _os.mkdir(file, 0o700)
FileNotFoundError: [Errno 2] No such file or directory: '/data/wpsoutputs/finch/workdir/pywps_process_8z895epm'

Zeitsperre · 2022-03-16T21:45:31Z

Closing. Superseded by #235

upgrade Finch to 0.8.0

58217a4

huard requested a review from tlvu January 13, 2022 21:37

tlvu mentioned this pull request Jan 14, 2022

Use catalog from prod and fix output url format change from Finch 0.8.0 Ouranosinc/pavics-sdi#244

Merged

tlvu approved these changes Jan 14, 2022

View reviewed changes

Merge remote-tracking branch 'origin/master' into finch-v0.8.0

d4779ea

tlvu mentioned this pull request Feb 14, 2022

Finch and Raven not creating their wps_output under /bird_name #11

Closed

Zeitsperre closed this Mar 16, 2022

tlvu deleted the finch-v0.8.0 branch April 6, 2022 15:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade Finch to 0.8.0 #231

upgrade Finch to 0.8.0 #231

huard commented Jan 13, 2022 •

edited by tlvu

Loading

crim-jenkins-bot commented Jan 13, 2022

tlvu commented Jan 14, 2022

tlvu left a comment

tlvu commented Jan 14, 2022

crim-jenkins-bot commented Jan 14, 2022

tlvu commented Jan 31, 2022

huard commented Jan 31, 2022

tlvu commented Jan 31, 2022

crim-jenkins-bot commented Feb 2, 2022

tlvu commented Feb 3, 2022

tlvu commented Feb 3, 2022

Zeitsperre commented Mar 16, 2022

upgrade Finch to 0.8.0 #231

upgrade Finch to 0.8.0 #231

Conversation

huard commented Jan 13, 2022 • edited by tlvu Loading

Overview

Changes

Related Issue / Discussion

Additional Information

crim-jenkins-bot commented Jan 13, 2022

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu commented Jan 14, 2022

tlvu left a comment

Choose a reason for hiding this comment

tlvu commented Jan 14, 2022

crim-jenkins-bot commented Jan 14, 2022

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu commented Jan 31, 2022

huard commented Jan 31, 2022

tlvu commented Jan 31, 2022

crim-jenkins-bot commented Feb 2, 2022

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

tlvu commented Feb 3, 2022

tlvu commented Feb 3, 2022

Zeitsperre commented Mar 16, 2022

huard commented Jan 13, 2022 •

edited by tlvu

Loading