HiQGA Documentation
This is a general purpose package for spatial statistical inference, geophysical forward modeling, Bayesian inference and inversion (both determinstic and probabilistic).
Readily usable geophysical forward operators are to do with AEM, CSEM and MT physics (references underneath), for which the time domain AEM codes are fairly production-ready. The current EM modeling is in 1D, but the inversion framework is dimensionally agnostic (e.g., you e regress images). Adding your own geophysical operators is easy, keep reading down here.
This package implements both the nested (2-layer) and vanilla trans-dimensional Gaussian process algorithm as published in
- An information theoretic Bayesian uncertainty analysis of AEM systems over Menindee Lakes, Australia. A.Ray, A-Y. Ley-Cooper, R.C. Brodie, R. Taylor, N. Symington, N. F. Moghaddam, Geophysical Journal International, 235(2)., 2023.
- Bayesian inversion using nested trans-dimensional Gaussian processes, A. Ray, Geophysical Journal International, 226(1), 2021.
- Bayesian geophysical inversion with trans-dimensional Gaussian process machine learning, A. Ray and D. Myer, Geophysical Journal International 217(3), 2019.
- There is also a flavour of within-bounds Gauss-Newton/Occam's inversion implemented. For SkyTEM, TEMPEST and VTEM (all AEM), this is fully functional, but for other forward propagators you will have to provide a Jacobian (the linearization of the forward operator).
Installation
To install, in a perfect world we'd use Julia's Pkg REPL by hitting ] to enter pkg> mode. Then enter the following, at the pkg> prompt:
pkg> add HiQGA If installing to follow along for the 2022 AEM workshop branch use instead of the above pkg> add HiQGA@0.2.2
Usage
Examples of how to use the package can be found in the examples directory. Simply cd to the relevant example directory and include the .jl files in the order they are named. If using VSCode make sure to do Julia: Change to this Directory from the three dots menu on the top right. The Markov Chain Monte Carlo sampler is configured to support parallel tempering on multiple CPUs - some of the examples accomplish this with Julia's built-in multiprocessing, and others use MPI in order to support inversions on HPC clusters that don't work with Julia's default SSH-based multiprocessing. The MPI examples require MPI.jl and MPIClusterManagers.jl, which are not installed as dependencies for this package, so you will need to ensure they are installed and configured correctly to run these examples. See here for MPI on the NCI.
Some example scripts have as a dependency Revise.jl as we're still actively developing this package, so you may need to install Revise if not already installed. All Julia users should be developing with Revise! After installation, to run the examples, simply clone the package separately (or download as a ZIP), navigate to the examples folder and run the scripts in their numerical order.
Updating the package
pkg> update HiQGADeveloping HiQGA or modifying it for your own special forward physics
After you have ]added HiQGA you can simply do:
pkg>dev HiQGAIf you haven't added it already, you can do:
pkg>dev https://github.com/GeoscienceAustralia/HiQGA.jl.gitIt will download to your JULIA_PKG_DEVDIR.
Another way is to simply clone or download this repository to your JULIA_PKG_DEVDIR, rename the cloned directory HiQGA removing the .jl bit and do
pkg>dev HiQGAHere's a gist on adding your own module if you want to modify the source code. Alternatively, if you only want to use the sampling methods in HiQGA.transD_GP without contributing to the source (boo! j/k) here's another gist which is more appropriate. These gists were written originally for a package called transD_GP so you will have to modify using transD_GP to using HiQGA.transD_GP. Documentation is important and we're working on improving it before a full-release.
Development setup on NCI
If you don't already have access to a julia binary, download the appropriate version .tar.gz from here and then untar it in a location you have write access to. Then, in your $HOME/bin directory make a symlink to the julia binary like so:
cd ~/bin
-ln -s /somwehere/home/me/julia-x.x.x/bin/julia .The preferred development and usage environment for HiQGA is Visual Studio Code, which provides interactive execution of Julia code through the VSCode Julia extension. To install VSCode on the National Computational Infrastructure (NCI), you need to extract the VSCode rpm package using the steps in this gist, to a location where your account has write access. You will NOT be using vscode on a gadi login node, but on OOD.
Get Julia language support from VSCode after launching the VSCode binary by going to File->Extensions by searching for Julia. If after installation it doesn't find the Julia binary, go to File->Extensions->Julia->Manage(the little gear icon) and manually type in /home/yourusername/bin/julia in the "Executable Path" field.
It is also useful to use Revise.jl to ensure changes to the package are immediately reflected in a running Julia REPL (this is the reason that Revise is a dependency on some example scripts as noted above). More information on a workflow to use Revise during development can be found here.
Installing MPI.jl and MPIClusterManagers.jl on NCI
We have found that the safest bet for MPI.jl to work without UCX issues on NCI is to use intel-mpi. In order to install MPI.jl and configure it to use the intel-mpi provided by the module intel-mpi/2021.10.0, following the example below.
$ module load intel-mpi/2021.10.0
+HiQGA Documentation · HiQGA - High Quality Geophysical Analysis HiQGA Documentation
This is a general purpose package for spatial statistical inference, geophysical forward modeling, Bayesian inference and inversion (both determinstic and probabilistic).
Readily usable geophysical forward operators are to do with AEM, CSEM and MT physics (references underneath), for which the time domain AEM codes are fairly production-ready. The current EM modeling is in 1D, but the inversion framework is dimensionally agnostic (e.g., you e regress images). Adding your own geophysical operators is easy, keep reading down here.
This package implements both the nested (2-layer) and vanilla trans-dimensional Gaussian process algorithm as published in
- An information theoretic Bayesian uncertainty analysis of AEM systems over Menindee Lakes, Australia. A.Ray, A-Y. Ley-Cooper, R.C. Brodie, R. Taylor, N. Symington, N. F. Moghaddam, Geophysical Journal International, 235(2)., 2023.
- Bayesian inversion using nested trans-dimensional Gaussian processes, A. Ray, Geophysical Journal International, 226(1), 2021.
- Bayesian geophysical inversion with trans-dimensional Gaussian process machine learning, A. Ray and D. Myer, Geophysical Journal International 217(3), 2019.
- There is also a flavour of within-bounds Gauss-Newton/Occam's inversion implemented. For SkyTEM, TEMPEST and VTEM (all AEM), this is fully functional, but for other forward propagators you will have to provide a Jacobian (the linearization of the forward operator).
Installation
HPC users look here to set up MPI for large jobs, or if you do not have Julia installed at all.
To install HiQGA, in a perfect world we'd use Julia's Pkg REPL by hitting ] to enter pkg> mode. Then enter the following, at the pkg> prompt:
pkg> add HiQGA
If installing to follow along for the 2022 AEM workshop branch use instead of the above pkg> add HiQGA@0.2.2
Usage
Examples of how to use the package can be found in the examples directory. Simply cd to the relevant example directory and include the .jl files in the order they are named. If using VSCode make sure to do Julia: Change to this Directory from the three dots menu on the top right. The Markov Chain Monte Carlo sampler is configured to support parallel tempering on multiple CPUs - some of the examples accomplish this with Julia's built-in multiprocessing, and others use MPI in order to support inversions on HPC clusters that don't work with Julia's default SSH-based multiprocessing. The MPI examples require MPI.jl and MPIClusterManagers.jl, which are not installed as dependencies for this package, so you will need to ensure they are installed and configured correctly to run these examples. See here for MPI on the NCI.
Some example scripts have as a dependency Revise.jl as we're still actively developing this package, so you may need to install Revise if not already installed. All Julia users should be developing with Revise! After installation, to run the examples, simply clone the package separately (or download as a ZIP), navigate to the examples folder and run the scripts in their numerical order.
Updating the package
pkg> update HiQGA
Developing HiQGA or modifying it for your own special forward physics
After you have ]added HiQGA you can simply do:
pkg>dev HiQGA
If you haven't added it already, you can do:
pkg>dev https://github.com/GeoscienceAustralia/HiQGA.jl.git
It will download to your JULIA_PKG_DEVDIR.
Another way is to simply clone or download this repository to your JULIA_PKG_DEVDIR, rename the cloned directory HiQGA removing the .jl bit and do
pkg>dev HiQGA
Here's a gist on adding your own module if you want to modify the source code. Alternatively, if you only want to use the sampling methods in HiQGA.transD_GP without contributing to the source (boo! j/k) here's another gist which is more appropriate. These gists were written originally for a package called transD_GP so you will have to modify using transD_GP to using HiQGA.transD_GP. Documentation is important and we're working on improving it before a full-release.
HPC setup on NCI
If you don't already have access to a julia binary, download the appropriate version .tar.gz from here and then untar it in a location you have write access to, like so:
cd /somwehere/home/me
+tar -xvzf /somwehere/home/me/julia-x.x.x.ta.gz
Then, in your $HOME/bin directory make a symlink to the julia binary like so:
cd ~/bin
+ln -s /somwehere/home/me/julia-x.x.x/bin/julia .
Make sure your $HOME/bin is in your $PATH else which you can check with echo $PATH | grep"$HOME/bin". If you do not see your bin directory highlighted, do export PATH=~/opt/bin:$PATH The preferred development and usage environment for HiQGA is Visual Studio Code, which provides interactive execution of Julia code through the VSCode Julia extension. To install VSCode on the National Computational Infrastructure (NCI), you need to extract the VSCode rpm package using the steps in this gist, to a location where your account has write access. You will NOT be using vscode on a gadi login node, but on OOD.
Get Julia language support from VSCode after launching the VSCode binary by going to File->Extensions by searching for Julia. If after installation it doesn't find the Julia binary, go to File->Extensions->Julia->Manage(the little gear icon) and manually type in /home/yourusername/bin/julia in the "Executable Path" field.
It is also useful to use Revise.jl to ensure changes to the package are immediately reflected in a running Julia REPL (this is the reason that Revise is a dependency on some example scripts as noted above). More information on a workflow to use Revise during development can be found here.
Installing MPI.jl and MPIClusterManagers.jl on NCI
We have found that the safest bet for MPI.jl to work without UCX issues on NCI is to use intel-mpi. In order to install MPI.jl and configure it to use the intel-mpi provided by the module intel-mpi/2021.10.0, following the example below.
$ module load intel-mpi/2021.10.0
$ julia
julia > ]
@@ -62,4 +63,4 @@
rmprocs(workers())
exit()
Run the code after loading the intel-mpi module you have linked MPI.jl against with
module load intel-mpi/2021.10.0
mpirun -np 3 julia mpitest.jl
and you should see output like:
[ Info: size is 3
-[ Info: there are 2 workers
This is the basic recipe for all the cluster HiQGA jobs on NCI. After the call to manager = MPIClusterManagers.start_main_loop(MPI_TRANSPORT_ALL), standard MPI execution stops, and we return to an explicit manager-worker mode with code execution only continuing on the manager which is Julia process 1.
The next time you start julia you have HiQGA ready for use with
julia> using HiQGA
navigate to the examples folder to run some example scripts. You can end here as a regular user, however for development mode see below.
For installing development mode pre-release versions
pkg> dev HiQGA
Make a pull request if you wish to submit your change – we welcome feature additions. If you want to switch back to the official version from development mode, do
pkg> free HiQGA
References for AEM and CSEM physics
- Blatter, D., Key, K., Ray, A., Foley, N., Tulaczyk, S., & Auken, E. (2018). Trans-dimensional Bayesian inversion of airborne transient EM data from Taylor Glacier, Antarctica. Geophysical Journal International, 214(3)
- Brodie, R. C. (2010). Holistic inversion of airborne electromagnetic data. PhD thesis, Australian National University.
- Ray, A., & Key, K. (2012). Bayesian inversion of marine CSEM data with a trans-dimensional self parametrizing algorithm. Geophysical Journal International, 191(3), 1135-1151.
Settings
This document was generated with Documenter.jl version 1.2.1 on Tuesday 12 December 2023. Using Julia version 1.9.4.
This is the basic recipe for all the cluster HiQGA jobs on NCI. After the call to manager = MPIClusterManagers.start_main_loop(MPI_TRANSPORT_ALL), standard MPI execution stops, and we return to an explicit manager-worker mode with code execution only continuing on the manager which is Julia process 1.
The next time you start julia you have HiQGA ready for use with
julia> using HiQGAnavigate to the examples folder to run some example scripts. You can end here as a regular user, however for development mode see below.
For installing development mode pre-release versions
pkg> dev HiQGAMake a pull request if you wish to submit your change – we welcome feature additions. If you want to switch back to the official version from development mode, do
pkg> free HiQGAReferences for AEM and CSEM physics
- Blatter, D., Key, K., Ray, A., Foley, N., Tulaczyk, S., & Auken, E. (2018). Trans-dimensional Bayesian inversion of airborne transient EM data from Taylor Glacier, Antarctica. Geophysical Journal International, 214(3)
- Brodie, R. C. (2010). Holistic inversion of airborne electromagnetic data. PhD thesis, Australian National University.
- Ray, A., & Key, K. (2012). Bayesian inversion of marine CSEM data with a trans-dimensional self parametrizing algorithm. Geophysical Journal International, 191(3), 1135-1151.