Spawners crash when deploying after cluster reboot

[mkossatz]

We set up this workshop via RHPDS for a customer, then put the cluster to sleep so when can start it for the meeting the next day. Upon restarting the cluster, the Homeroom was available but upon clicking the individual language-specific lab links, these crashed. We manually re-created the project to fix it.

Having restarted it a second time so that the customer can continue to work on it, most labs worked and just the Java one didn't. We didn't look further into it the first time it happened and the details below are specific to the Java lab crashing upon cluster restart but we assume it's the same problem.

This is the Pod that crashes:
lab-getting-started-java-spawner-2-6jfk4

Logs:

+ . /opt/app-root/src/jupyterhub_config.sh
++ CONFIGURATION_TYPE=terminal-server
++ '[' -f /opt/app-root/src/configs/terminal-server.sh ']'
++ . /opt/app-root/src/configs/terminal-server.sh
+++ KUBERNETES_SERVER_URL=https://172.30.0.1:443
+++ OAUTH_METADATA_URL=https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ curl -ks https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ jq -r .issuer
+++ OAUTH_ISSUER_ADDRESS=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ export OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
++ '[' -f /opt/app-root/configs/jupyterhub_config.sh ']'
++ . /opt/app-root/configs/jupyterhub_config.sh
+ trap 'kill -TERM $PID' TERM INT
+ PID=32
+ wait 32
+ jupyterhub -f /opt/app-root/src/jupyterhub_config.py
INFO: Workshop name is 'lab-getting-started-java'.
INFO: Application name is 'lab-getting-started-java'.
INFO: Configuration type is 'terminal-server'.
INFO: Homeroom link is ''.
INFO: Homeroom name is 'homeroom'.
INFO: Service account name is 'lab-getting-started-java-spawner'.
INFO: Namespace is 'lab'.
INFO: Full service account name is 'system:serviceaccount:lab:lab-getting-started-java-spawner'.
/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.30.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
INFO: Homeroom link set to http://homeroom-lab.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com.
[I 2021-05-12 10:49:19.642 JupyterHub app:2120] Using Authenticator: oauthenticator.openshift.OpenShiftOAuthenticator-0.9.0
[I 2021-05-12 10:49:19.643 JupyterHub app:2120] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2021-05-12 10:49:19.644 JupyterHub app:1257] Loading cookie_secret from /opt/app-root/data/cookie_secret
[I 2021-05-12 10:49:19.660 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-05-12 10:49:19.662 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2021-05-12 10:49:19.662 JupyterHub app:1534] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-12 10:49:19.662 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[I 2021-05-12 10:49:19.729 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace lab
[I 2021-05-12 10:49:19.788 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace lab
[W 2021-05-12 10:49:22.852 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:25.924 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:28.996 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:32.068 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:35.140 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:38.212 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
[W 2021-05-12 10:49:41.284 JupyterHub utils:226] Failed to connect to http://10.131.1.78:10080/user/user11/ ([Errno 113] No route to host)
++ kill -TERM 32
+ trap - TERM INT
+ wait 32
/opt/app-root/etc/run.sh: line 18:    32 Terminated              jupyterhub -f /opt/app-root/src/jupyterhub_config.py
+ STATUS=143
+ exit 143

[mkossatz]

We have that problem continuously after cluster reboots. This is now always with the java lab spawner. Oddly, the log lacks the failed connection errors that I thought were the reason for the crash.

+ . /opt/app-root/src/jupyterhub_config.sh
++ CONFIGURATION_TYPE=terminal-server
++ '[' -f /opt/app-root/src/configs/terminal-server.sh ']'
++ . /opt/app-root/src/configs/terminal-server.sh
+++ KUBERNETES_SERVER_URL=https://172.30.0.1:443
+++ OAUTH_METADATA_URL=https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ curl -ks https://172.30.0.1:443/.well-known/oauth-authorization-server
++++ jq -r .issuer
+++ OAUTH_ISSUER_ADDRESS=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ export OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ OPENSHIFT_REST_API_URL=https://172.30.0.1:443
+++ export OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
+++ OPENSHIFT_AUTH_API_URL=https://oauth-openshift.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com
++ '[' -f /opt/app-root/configs/jupyterhub_config.sh ']'
++ . /opt/app-root/configs/jupyterhub_config.sh
+ trap 'kill -TERM $PID' TERM INT
+ PID=32
+ wait 32
+ jupyterhub -f /opt/app-root/src/jupyterhub_config.py
INFO: Workshop name is 'lab-getting-started-java'.
INFO: Application name is 'lab-getting-started-java'.
INFO: Configuration type is 'terminal-server'.
INFO: Homeroom link is ''.
INFO: Homeroom name is 'homeroom'.
INFO: Service account name is 'lab-getting-started-java-spawner'.
INFO: Namespace is 'lab'.
INFO: Full service account name is 'system:serviceaccount:lab:lab-getting-started-java-spawner'.
/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.30.0.1'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
INFO: Homeroom link set to http://homeroom-lab.apps.cluster-aocpw-19c9.aocpw-19c9.example.opentlc.com.
[I 2021-05-14 08:07:44.204 JupyterHub app:2120] Using Authenticator: oauthenticator.openshift.OpenShiftOAuthenticator-0.9.0
[I 2021-05-14 08:07:44.204 JupyterHub app:2120] Using Spawner: kubespawner.spawner.KubeSpawner
[I 2021-05-14 08:07:44.205 JupyterHub app:1257] Loading cookie_secret from /opt/app-root/data/cookie_secret
[I 2021-05-14 08:07:44.222 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-05-14 08:07:44.223 JupyterHub app:1532] No admin users, admin interface will be unavailable.
[W 2021-05-14 08:07:44.223 JupyterHub app:1534] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-05-14 08:07:44.223 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[I 2021-05-14 08:07:44.302 JupyterHub reflector:199] watching for pods with label selector='component=singleuser-server' in namespace lab
[I 2021-05-14 08:07:44.531 JupyterHub reflector:199] watching for events with field selector='involvedObject.kind=Pod' in namespace lab
++ kill -TERM 32
+ trap - TERM INT
+ wait 32
/opt/app-root/etc/run.sh: line 18:    32 Terminated              jupyterhub -f /opt/app-root/src/jupyterhub_config.py
+ STATUS=143
+ exit 143

[blues-man]

See this #193