Upgrade k8s test.arcticdata.io from 3.0.0 to 3.1.0 and run tests

[artntek]

See upgrade checklist, below, created from this template

...and test
• metacatui submissions
• R client tests see steps here
• load testing (@taojing2002?)

[artntek]

Checklist: Metacat K8s Upgrade Steps

1. Values: Edit your existing values override file

• Remove any uid or gid overrides, since we're now adopting the defaults of 59996 for postgres
  and 59997 for metacat
• Temporarily disable probes until hashstore conversion is done

2. Prep Before Upgrading

• set storage.hashstore.disableConversion: true, so the hashstore converter won't run yet
• In the metacat database, verify that all the systemmetadata.checksum_algorithm entries are
  on the list of supported
  algorithms
  (NOTE: syntax matters! E.g. sha-1 is OK, but sha1 isn't):

  kubectl exec ${RELEASE_NAME}-postgresql-0 -- bash -c "psql -U metacat << EOF
    SELECT DISTINCT checksum_algorithm FROM systemmetadata WHERE checksum_algorithm NOT IN
          ('MD2','MD5','SHA-1','SHA-256','SHA-384','SHA-512','SHA-512/224','SHA-512/256');
  EOF"
  
  # then manually update each to the correct syntax; e.g:
  kubectl exec ${RELEASE_NAME}-postgresql-0 -- bash -c "psql -U metacat << EOF
    UPDATE systemmetadata SET checksum_algorithm='SHA-1' WHERE checksum_algorithm='SHA1';
  EOF"
  # ...etc

3. Upgrading - NOTE THERE WILL BE DOWNTIME!

= = = = = = = = Downtime starts here = = = = = = = =

• Change ownership ON CEPHFS as follows:

  ## postgres (59996:59996) in postgresql data directory
  sudo chown -R 59996:59996 /mnt/ceph/repos/REPO-NAME/postgresql
  
  ## tomcat (59997:59997) in metacat directory
  sudo chown -R 59997:59997 data dataone documents logs

• ...then ensure all metacat data and documents files have g+rw permissions, otherwise,
  hashstore converter can't create hard links:

  sudo chmod -R g+rw data documents dataone

• helm upgrade, debug any startup and configuration issues, and allow hashstore upgrade to
  finish.

  NOTE: while hashstore conversion is still in progress, it is expected for metacatUI to
  display Oops! It looks like there was a problem retrieving your search results., and for
  /metacat/d1/mn/v2/ api calls to display Metacat has not been configured

  See Tips, below for
  how to detect when hashstore conversion finishes

= = = = = = = = Downtime ends here = = = = = = = =

• When hashstore conversion has finished, re-enable probes and helm upgrade to apply changes