Skip to content

Commit

Permalink
Initial commit
Browse files Browse the repository at this point in the history
[x] Passes pre-commit checks
  • Loading branch information
destin-v committed Sep 25, 2024
0 parents commit eefcddb
Show file tree
Hide file tree
Showing 115 changed files with 11,017 additions and 0 deletions.
81 changes: 81 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
##########################################################################################
#
# DESCRIPTION: Development Docker Container. Installs the following:
# * miniconda
# * Oh-My-Zsh
# * awesome-vim
#
# GPU SUPPORT: In order to get GPU support the nvidia image's CUDA version must
# match the version on the host computer.
#
# RENDERING: If rendering to the screen is required then execute the following
# prior to starting the container:
#
# ```console
# xhost +local:root
# ```
#
# AUTHOR: W. Li
# VERSION: 1.0
# CREATED: 12/15/2023
##########################################################################################

# Source Image (this should match the CUDA version on the host)
# FROM nvidia/cuda:11.4.1-base-ubuntu20.04
FROM ubuntu:20.04

# Labels
LABEL version="1.0"
LABEL description="Miniconda Dev Image"

# Expose ports here
# EXPOSE 8080

# Install basic packages needed for development
RUN yes Y | apt update
RUN yes Y | apt-get update
RUN yes Y | apt upgrade
RUN yes Y | apt install vim
RUN apt-get install -y --no-install-recommends \
git \
wget \
g++ \
gcc \
&& rm -rf /var/lib/apt/lists/*

# Setup shell
RUN sh -c "$(wget -O- https://github.com/deluan/zsh-in-docker/releases/download/v1.1.1/zsh-in-docker.sh)" -- \
-t robbyrussell \
-p git -p ssh-agent -p 'history-substring-search' \
-a 'bindkey "\$terminfo[kcuu1]" history-substring-search-up' \
-a 'bindkey "\$terminfo[kcud1]" history-substring-search-down'

# Setup Shell
SHELL ["/bin/zsh", "-c"]

# Miniconda requires that the path point to its python binary
ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"

# Source python versions: https://repo.anaconda.com/miniconda/
RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh

RUN conda init zsh

# Setup Vim-Awesome
RUN git clone --depth=1 https://github.com/amix/vimrc.git ~/.vim_runtime \
&& sh ~/.vim_runtime/install_basic_vimrc.sh

# Setup working directory
WORKDIR /projects

# Copy folder/files into container (must be within context directory)
# Process: local files -> artifact dir -> container dir
# COPY <local_path> <container_path>

# Environment
ENV SHELL /bin/zsh
60 changes: 60 additions & 0 deletions .devcontainer/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# 📒 Description
## 🐳 Docker
* Primary method for building and maintaining **un-secure** containers.
* Must have root access in order to work.
* Deployable to any machine.
* [Additional Information](#🐳-docker-notes)

## 🦉 Apptainer
* Primary method for building and maintaining **secure** containers.
* Used primarily in settings where root access is not possible.
* Deployable to any machine.
* [Additional Information](#🦉-apptainer-notes)

## 📦 Packages
The default development containers will these packages by default. Only the essentials are included.

* [Miniconda](https://docs.conda.io/projects/miniconda/en/latest/)
* [Oh My Bash](https://github.com/ohmybash/oh-my-bash)
* [Oh My Zsh](https://ohmyz.sh)
* [Precommit](https://pre-commit.com)

# 🐳 Docker Notes

## 🎼 Docker Compose
* Allows you to create multiple different builds using different Docker files.
* The context directory can be set from the `docker_compose.yml` file.
* More options than a standard Docker file.

## 👟 Running Commands
* In Docker each `RUN` command is treated as a new shell instance. This means that commands cannot persist across different RUN commands.

```Docker
RUN command1 # this command is run in its own shell!
RUN command2 # this command is run in its own shell!
```

## 🗃️ Copying Folders/Files
* If you need to copy files or directories into the Docker container, first put them in an **artifacts** directory that can be seen from the context directory.

```Docker
COPY <artifact_directory> <container_path>
```
## 📂 Context Directory

* By default Dockerfiles are only aware of what is in their **context directory**. A context directory is the directory where a Dockerfile is located in. It cannot access any files outside of the context directory.

* The context directory can be set from the command line or from a `docker_compose.yml` file.

* It is recommended that all folders/files be copied be placed in an **artifacts** directory that is under the context directory.

## 💾 Out of Memory
* Occasionally you will need to clean your docker images and docker volumes in order to recover memory.

```console
docker volume prune
docker image prune
docker container prune
```

# 🦉 Apptainer Notes
6 changes: 6 additions & 0 deletions .devcontainer/build.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
import subprocess

image_name = "development-base"

# Build the image from the development Dockerfile
subprocess.run(["docker", "build", "-t", image_name, "."])
44 changes: 44 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.191.0/containers/ubuntu
{
"name": "dev-python",
"image": "development-base",
"build": {
"dockerfile": "Dockerfile",
},
// Add arguments that run with the container
// These are arguments needed for GPU support on a Linux machine
"runArgs": [
// "--net=host",
// "-m=10g",
// "--shm-size=10g",
// "--gpus=all",
// "-v",
// "/tmp/.X11-unix:/tmp/.X11-unix",
// "-e",
// "DISPLAY",
// "--device",
// "/dev/dri"
],
"customizations": {
"vscode": {
// Set *default* container specific settings.json values on container create.
"settings": {},
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"eamodio.gitlens",
"littlefoxteam.vscode-python-test-adapter",
"ms-python.python",
"mhutchie.git-graph",
"njpwerner.autodocstring",
"pkief.material-icon-theme",
]
}
},
// Use 'forwardPorts' to make a list of ports inside the container available locally.
"forwardPorts": [],
// Use 'postCreateCommand' to run commands after the container is created.
"postCreateCommand": "uname -a"
// Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
// "remoteUser": "vscode"
}
45 changes: 45 additions & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# CI/CD

## Workflows
GitHub workflows with continuous integration is included. The GitHub workflows send their tasks to GitHub runners by default. If you want to route jobs to your **self-hosted** runners make sure to add a **label**. The steps are:


> Settings › Actions › Runners › Self Hosted Runner › Create Label
You can now have your workflows send jobs to `macos-latest` runner by specifying the following in the workflow.yml file:

```yaml
jobs:
test:
# Set the OS and python versions
runs-on: macos-latest
```
> [!IMPORTANT]
> If your self-hosted runner is given a label that also matches a GitHub runner (i.e. `ubuntu-latest`) then your job can be scheduled by self-hosted or GitHub runners. This is useful for situations where you want the job to have the option of running on either kind of runners.

## Badges
The badge at the top of the README.md will update its status to display whether the CI process succeeds/fails. Modify it to point to your own repo when starting a new project.

## Git Runner Limits
If you are running into rate limiting issues with Git runner you will need to add an authorization token to your Git actions. The GitHub token must be generated from the public GitHub and *not* your self-hosted GitHub. This is because the libraries being installed are sourced from the public GitHub.


* Generate a GitHub token [**here**](https://github.com/settings/tokens).
* Navigate to Repo › Settings › Secrets and Variables.
* Add secret **GH_GITHUB_COM_TOKEN** and paste the generated GitHub token.

The `checks.yml` workflow includes code to display your rate limits. See this [**template**](https://github.com/actions/setup-python/pull/443#issuecomment-1206776401) for a full example.

## Artifact Upload/Download Errors
If you are running self-hosted Git runners, keep in mind that git-action artifacts will upload/download to public GitHub. If your goal is to pass artifacts within a self-hosted runner, the best way is to save files to a shared space. See a discussion [**here**](https://github.com/orgs/community/discussions/26165).

If you want to upload/download artifacts to public GitHub using self-hosted runner then you must ensure that your CA certificates are used. These variables should be set in your `.env` file of your self-hosted runner.

```bash
NODE_EXTRA_CA_CERTS=/path/to/ca-certificates.pem # .crt and .pem are interchangeable
NODE_TLS_REJECT_UNAUTHORIZED=0 # disable CA checking for debugging purposes
```

## Runner Debugging
To enable debugging see [**here**](https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging). The secret `ACTIONS_RUNNER_DEBUG=true` should be set, and debugging messaging will show up in the runner logs.
40 changes: 40 additions & 0 deletions .github/workflows/pre-commit.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
##########################################################################################
# DESCRIPTION: Run pre-commit on all files in repo.
# AUTHOR: W. Li
# VERSION: 1.0
# CREATED: 1/6/2024
#
# References:
# * https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-pythonCommon
#
##########################################################################################

name: pre-commit

on:
pull_request:
push:
branches: [main]

##########################################################################################
# Jobs
##########################################################################################
jobs:
########################################################################################
# Pre-commit checks.
########################################################################################
pre-commit:
runs-on: ubuntu-latest
steps:
# Checkout repo
- name: Checkout Repo
uses: actions/checkout@v4
# Setup Python
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
token: ${{ secrets.GH_GITHUB_COM_TOKEN }}
python-version: "3.10"
# Run pre-commit
- name: Run pre-commit
uses: pre-commit/[email protected]
112 changes: 112 additions & 0 deletions .github/workflows/sphinx.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
##########################################################################################
# DESCRIPTION: Generate Sphinx and pushes them to Git Pages.
# AUTHOR: W. Li
# VERSION: 1.0
# CREATED: 5/23/2024
#
# References:
# * https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-pythonCommon
#
##########################################################################################

name: sphinx

on:
push:
branches: ["main"]

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
contents: read
pages: write
id-token: write

# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued. However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
concurrency:
group: "pages"
cancel-in-progress: false

##########################################################################################
# Jobs
##########################################################################################
jobs:
########################################################################################
# Build the static HTML files we want to publish.
########################################################################################
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
steps:
# Checkout the source repo
- name: Checkout source
uses: actions/checkout@v4
# Setup the environment with specified python version
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
token: ${{ secrets.GH_GITHUB_COM_TOKEN }}
python-version: "3.10"
# Setup Homebrew
- name: Set up Homebrew
id: set-up-homebrew
uses: Homebrew/actions/setup-homebrew@master
- name: Install Homebrew packages
run: brew install allure
# Install poetry and disable virtual environments
- name: Install Poetry
uses: snok/install-poetry@v1
with:
virtualenvs-create: false
virtualenvs-in-project: false
installer-parallel: true
# Install nox
- name: Install dependencies
run: |
pip install nox
# Use nox to generate the document
- name: Generate Sphinx artifacts
run: |
export PYTHONPATH=${{ github.workspace }}
nox -r -s build
# Upload the artifact to the next job
- name: Upload Sphinx artifact
uses: actions/upload-artifact@v4
with:
name: Sphinx artifact
path: docs/build/html/

########################################################################################
# Deploy the static HTML files to Github Pages.
########################################################################################
deploy:
needs: [build]
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
runs-on: ubuntu-latest
steps:
# We first need to checkout the repo for the code.
- name: Checkout
uses: actions/checkout@v4
# Now download the artifact from the previous build for Sphinx.
- name: Download Sphinx artifact
uses: actions/download-artifact@v4
with:
name: Sphinx artifact
path: docs/build/html/
# Note: Github pages expects all of your HTML static files to reside in docs/
- name: Display structure of downloaded files
run: ls -al
working-directory: docs/build/html/
# We are now ready to upload the artifact folder to Github pages.
- name: Setup Pages
uses: actions/configure-pages@v4
- name: Upload artifact
uses: actions/upload-pages-artifact@v3
with:
path: docs/build/html/
# Once the upload is complete we can deploy
- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
Loading

0 comments on commit eefcddb

Please sign in to comment.