-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[x] Passes pre-commit checks
- Loading branch information
0 parents
commit eefcddb
Showing
115 changed files
with
11,017 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,81 @@ | ||
########################################################################################## | ||
# | ||
# DESCRIPTION: Development Docker Container. Installs the following: | ||
# * miniconda | ||
# * Oh-My-Zsh | ||
# * awesome-vim | ||
# | ||
# GPU SUPPORT: In order to get GPU support the nvidia image's CUDA version must | ||
# match the version on the host computer. | ||
# | ||
# RENDERING: If rendering to the screen is required then execute the following | ||
# prior to starting the container: | ||
# | ||
# ```console | ||
# xhost +local:root | ||
# ``` | ||
# | ||
# AUTHOR: W. Li | ||
# VERSION: 1.0 | ||
# CREATED: 12/15/2023 | ||
########################################################################################## | ||
|
||
# Source Image (this should match the CUDA version on the host) | ||
# FROM nvidia/cuda:11.4.1-base-ubuntu20.04 | ||
FROM ubuntu:20.04 | ||
|
||
# Labels | ||
LABEL version="1.0" | ||
LABEL description="Miniconda Dev Image" | ||
|
||
# Expose ports here | ||
# EXPOSE 8080 | ||
|
||
# Install basic packages needed for development | ||
RUN yes Y | apt update | ||
RUN yes Y | apt-get update | ||
RUN yes Y | apt upgrade | ||
RUN yes Y | apt install vim | ||
RUN apt-get install -y --no-install-recommends \ | ||
git \ | ||
wget \ | ||
g++ \ | ||
gcc \ | ||
&& rm -rf /var/lib/apt/lists/* | ||
|
||
# Setup shell | ||
RUN sh -c "$(wget -O- https://github.com/deluan/zsh-in-docker/releases/download/v1.1.1/zsh-in-docker.sh)" -- \ | ||
-t robbyrussell \ | ||
-p git -p ssh-agent -p 'history-substring-search' \ | ||
-a 'bindkey "\$terminfo[kcuu1]" history-substring-search-up' \ | ||
-a 'bindkey "\$terminfo[kcud1]" history-substring-search-down' | ||
|
||
# Setup Shell | ||
SHELL ["/bin/zsh", "-c"] | ||
|
||
# Miniconda requires that the path point to its python binary | ||
ENV PATH="/root/miniconda3/bin:${PATH}" | ||
ARG PATH="/root/miniconda3/bin:${PATH}" | ||
|
||
# Source python versions: https://repo.anaconda.com/miniconda/ | ||
RUN wget \ | ||
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \ | ||
&& mkdir /root/.conda \ | ||
&& bash Miniconda3-latest-Linux-x86_64.sh -b \ | ||
&& rm -f Miniconda3-latest-Linux-x86_64.sh | ||
|
||
RUN conda init zsh | ||
|
||
# Setup Vim-Awesome | ||
RUN git clone --depth=1 https://github.com/amix/vimrc.git ~/.vim_runtime \ | ||
&& sh ~/.vim_runtime/install_basic_vimrc.sh | ||
|
||
# Setup working directory | ||
WORKDIR /projects | ||
|
||
# Copy folder/files into container (must be within context directory) | ||
# Process: local files -> artifact dir -> container dir | ||
# COPY <local_path> <container_path> | ||
|
||
# Environment | ||
ENV SHELL /bin/zsh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
# 📒 Description | ||
## 🐳 Docker | ||
* Primary method for building and maintaining **un-secure** containers. | ||
* Must have root access in order to work. | ||
* Deployable to any machine. | ||
* [Additional Information](#🐳-docker-notes) | ||
|
||
## 🦉 Apptainer | ||
* Primary method for building and maintaining **secure** containers. | ||
* Used primarily in settings where root access is not possible. | ||
* Deployable to any machine. | ||
* [Additional Information](#🦉-apptainer-notes) | ||
|
||
## 📦 Packages | ||
The default development containers will these packages by default. Only the essentials are included. | ||
|
||
* [Miniconda](https://docs.conda.io/projects/miniconda/en/latest/) | ||
* [Oh My Bash](https://github.com/ohmybash/oh-my-bash) | ||
* [Oh My Zsh](https://ohmyz.sh) | ||
* [Precommit](https://pre-commit.com) | ||
|
||
# 🐳 Docker Notes | ||
|
||
## 🎼 Docker Compose | ||
* Allows you to create multiple different builds using different Docker files. | ||
* The context directory can be set from the `docker_compose.yml` file. | ||
* More options than a standard Docker file. | ||
|
||
## 👟 Running Commands | ||
* In Docker each `RUN` command is treated as a new shell instance. This means that commands cannot persist across different RUN commands. | ||
|
||
```Docker | ||
RUN command1 # this command is run in its own shell! | ||
RUN command2 # this command is run in its own shell! | ||
``` | ||
|
||
## 🗃️ Copying Folders/Files | ||
* If you need to copy files or directories into the Docker container, first put them in an **artifacts** directory that can be seen from the context directory. | ||
|
||
```Docker | ||
COPY <artifact_directory> <container_path> | ||
``` | ||
## 📂 Context Directory | ||
|
||
* By default Dockerfiles are only aware of what is in their **context directory**. A context directory is the directory where a Dockerfile is located in. It cannot access any files outside of the context directory. | ||
|
||
* The context directory can be set from the command line or from a `docker_compose.yml` file. | ||
|
||
* It is recommended that all folders/files be copied be placed in an **artifacts** directory that is under the context directory. | ||
|
||
## 💾 Out of Memory | ||
* Occasionally you will need to clean your docker images and docker volumes in order to recover memory. | ||
|
||
```console | ||
docker volume prune | ||
docker image prune | ||
docker container prune | ||
``` | ||
|
||
# 🦉 Apptainer Notes |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
import subprocess | ||
|
||
image_name = "development-base" | ||
|
||
# Build the image from the development Dockerfile | ||
subprocess.run(["docker", "build", "-t", image_name, "."]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at: | ||
// https://github.com/microsoft/vscode-dev-containers/tree/v0.191.0/containers/ubuntu | ||
{ | ||
"name": "dev-python", | ||
"image": "development-base", | ||
"build": { | ||
"dockerfile": "Dockerfile", | ||
}, | ||
// Add arguments that run with the container | ||
// These are arguments needed for GPU support on a Linux machine | ||
"runArgs": [ | ||
// "--net=host", | ||
// "-m=10g", | ||
// "--shm-size=10g", | ||
// "--gpus=all", | ||
// "-v", | ||
// "/tmp/.X11-unix:/tmp/.X11-unix", | ||
// "-e", | ||
// "DISPLAY", | ||
// "--device", | ||
// "/dev/dri" | ||
], | ||
"customizations": { | ||
"vscode": { | ||
// Set *default* container specific settings.json values on container create. | ||
"settings": {}, | ||
// Add the IDs of extensions you want installed when the container is created. | ||
"extensions": [ | ||
"eamodio.gitlens", | ||
"littlefoxteam.vscode-python-test-adapter", | ||
"ms-python.python", | ||
"mhutchie.git-graph", | ||
"njpwerner.autodocstring", | ||
"pkief.material-icon-theme", | ||
] | ||
} | ||
}, | ||
// Use 'forwardPorts' to make a list of ports inside the container available locally. | ||
"forwardPorts": [], | ||
// Use 'postCreateCommand' to run commands after the container is created. | ||
"postCreateCommand": "uname -a" | ||
// Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root. | ||
// "remoteUser": "vscode" | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
# CI/CD | ||
|
||
## Workflows | ||
GitHub workflows with continuous integration is included. The GitHub workflows send their tasks to GitHub runners by default. If you want to route jobs to your **self-hosted** runners make sure to add a **label**. The steps are: | ||
|
||
|
||
> Settings › Actions › Runners › Self Hosted Runner › Create Label | ||
You can now have your workflows send jobs to `macos-latest` runner by specifying the following in the workflow.yml file: | ||
|
||
```yaml | ||
jobs: | ||
test: | ||
# Set the OS and python versions | ||
runs-on: macos-latest | ||
``` | ||
> [!IMPORTANT] | ||
> If your self-hosted runner is given a label that also matches a GitHub runner (i.e. `ubuntu-latest`) then your job can be scheduled by self-hosted or GitHub runners. This is useful for situations where you want the job to have the option of running on either kind of runners. | ||
|
||
## Badges | ||
The badge at the top of the README.md will update its status to display whether the CI process succeeds/fails. Modify it to point to your own repo when starting a new project. | ||
|
||
## Git Runner Limits | ||
If you are running into rate limiting issues with Git runner you will need to add an authorization token to your Git actions. The GitHub token must be generated from the public GitHub and *not* your self-hosted GitHub. This is because the libraries being installed are sourced from the public GitHub. | ||
|
||
|
||
* Generate a GitHub token [**here**](https://github.com/settings/tokens). | ||
* Navigate to Repo › Settings › Secrets and Variables. | ||
* Add secret **GH_GITHUB_COM_TOKEN** and paste the generated GitHub token. | ||
|
||
The `checks.yml` workflow includes code to display your rate limits. See this [**template**](https://github.com/actions/setup-python/pull/443#issuecomment-1206776401) for a full example. | ||
|
||
## Artifact Upload/Download Errors | ||
If you are running self-hosted Git runners, keep in mind that git-action artifacts will upload/download to public GitHub. If your goal is to pass artifacts within a self-hosted runner, the best way is to save files to a shared space. See a discussion [**here**](https://github.com/orgs/community/discussions/26165). | ||
|
||
If you want to upload/download artifacts to public GitHub using self-hosted runner then you must ensure that your CA certificates are used. These variables should be set in your `.env` file of your self-hosted runner. | ||
|
||
```bash | ||
NODE_EXTRA_CA_CERTS=/path/to/ca-certificates.pem # .crt and .pem are interchangeable | ||
NODE_TLS_REJECT_UNAUTHORIZED=0 # disable CA checking for debugging purposes | ||
``` | ||
|
||
## Runner Debugging | ||
To enable debugging see [**here**](https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging). The secret `ACTIONS_RUNNER_DEBUG=true` should be set, and debugging messaging will show up in the runner logs. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
########################################################################################## | ||
# DESCRIPTION: Run pre-commit on all files in repo. | ||
# AUTHOR: W. Li | ||
# VERSION: 1.0 | ||
# CREATED: 1/6/2024 | ||
# | ||
# References: | ||
# * https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-pythonCommon | ||
# | ||
########################################################################################## | ||
|
||
name: pre-commit | ||
|
||
on: | ||
pull_request: | ||
push: | ||
branches: [main] | ||
|
||
########################################################################################## | ||
# Jobs | ||
########################################################################################## | ||
jobs: | ||
######################################################################################## | ||
# Pre-commit checks. | ||
######################################################################################## | ||
pre-commit: | ||
runs-on: ubuntu-latest | ||
steps: | ||
# Checkout repo | ||
- name: Checkout Repo | ||
uses: actions/checkout@v4 | ||
# Setup Python | ||
- name: Set up Python 3.10 | ||
uses: actions/setup-python@v5 | ||
with: | ||
token: ${{ secrets.GH_GITHUB_COM_TOKEN }} | ||
python-version: "3.10" | ||
# Run pre-commit | ||
- name: Run pre-commit | ||
uses: pre-commit/[email protected] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
########################################################################################## | ||
# DESCRIPTION: Generate Sphinx and pushes them to Git Pages. | ||
# AUTHOR: W. Li | ||
# VERSION: 1.0 | ||
# CREATED: 5/23/2024 | ||
# | ||
# References: | ||
# * https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-pythonCommon | ||
# | ||
########################################################################################## | ||
|
||
name: sphinx | ||
|
||
on: | ||
push: | ||
branches: ["main"] | ||
|
||
# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages | ||
permissions: | ||
contents: read | ||
pages: write | ||
id-token: write | ||
|
||
# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued. However, do NOT cancel in-progress runs as we want to allow these production deployments to complete. | ||
concurrency: | ||
group: "pages" | ||
cancel-in-progress: false | ||
|
||
########################################################################################## | ||
# Jobs | ||
########################################################################################## | ||
jobs: | ||
######################################################################################## | ||
# Build the static HTML files we want to publish. | ||
######################################################################################## | ||
build: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
fail-fast: false | ||
steps: | ||
# Checkout the source repo | ||
- name: Checkout source | ||
uses: actions/checkout@v4 | ||
# Setup the environment with specified python version | ||
- name: Set up Python 3.10 | ||
uses: actions/setup-python@v5 | ||
with: | ||
token: ${{ secrets.GH_GITHUB_COM_TOKEN }} | ||
python-version: "3.10" | ||
# Setup Homebrew | ||
- name: Set up Homebrew | ||
id: set-up-homebrew | ||
uses: Homebrew/actions/setup-homebrew@master | ||
- name: Install Homebrew packages | ||
run: brew install allure | ||
# Install poetry and disable virtual environments | ||
- name: Install Poetry | ||
uses: snok/install-poetry@v1 | ||
with: | ||
virtualenvs-create: false | ||
virtualenvs-in-project: false | ||
installer-parallel: true | ||
# Install nox | ||
- name: Install dependencies | ||
run: | | ||
pip install nox | ||
# Use nox to generate the document | ||
- name: Generate Sphinx artifacts | ||
run: | | ||
export PYTHONPATH=${{ github.workspace }} | ||
nox -r -s build | ||
# Upload the artifact to the next job | ||
- name: Upload Sphinx artifact | ||
uses: actions/upload-artifact@v4 | ||
with: | ||
name: Sphinx artifact | ||
path: docs/build/html/ | ||
|
||
######################################################################################## | ||
# Deploy the static HTML files to Github Pages. | ||
######################################################################################## | ||
deploy: | ||
needs: [build] | ||
environment: | ||
name: github-pages | ||
url: ${{ steps.deployment.outputs.page_url }} | ||
runs-on: ubuntu-latest | ||
steps: | ||
# We first need to checkout the repo for the code. | ||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
# Now download the artifact from the previous build for Sphinx. | ||
- name: Download Sphinx artifact | ||
uses: actions/download-artifact@v4 | ||
with: | ||
name: Sphinx artifact | ||
path: docs/build/html/ | ||
# Note: Github pages expects all of your HTML static files to reside in docs/ | ||
- name: Display structure of downloaded files | ||
run: ls -al | ||
working-directory: docs/build/html/ | ||
# We are now ready to upload the artifact folder to Github pages. | ||
- name: Setup Pages | ||
uses: actions/configure-pages@v4 | ||
- name: Upload artifact | ||
uses: actions/upload-pages-artifact@v3 | ||
with: | ||
path: docs/build/html/ | ||
# Once the upload is complete we can deploy | ||
- name: Deploy to GitHub Pages | ||
id: deployment | ||
uses: actions/deploy-pages@v4 |
Oops, something went wrong.