Skip to content

Latest commit

 

History

History
88 lines (68 loc) · 3.78 KB

README.md

File metadata and controls

88 lines (68 loc) · 3.78 KB
Raw

Job Manager API Server: dsub

Thin shim around dsub.

Development

Currently the Job Manager does not support starting jobs directly. This must be done through the dsub CLI (see Getting Started to get setup).

google

  1. See Getting Started on Google Cloud for setup instructions and examples on how to run jobs on Google Cloud Platform.
  2. Ensure you have correctly set Application Default Credentials and logged into gcloud.

local

  1. Install Docker
  2. Create a local tmp directory for the local provider to store job data:
  3. See Getting Started With The Local Provider for examples on how to start local jobs.
mkdir /tmp/dsub-local

Running Tests

To run unit and integration tests on the python-flask app, install tox. Integration tests with the google provider also require access to the bvdp-jmui-testing Google Cloud project. File a Github issue assigned to @bfcrampton to gain access.

cd servers/dsub
# Run all the tests
tox -- -s
# Run all local provider tests
tox jobs/test/test_jobs_controller_local.py
# Run specific local provider test
tox jobs/test/test_jobs_controller_local.py:TestJobsControllerLocal.test_abort_job
# Run only google provider tests
tox jobs/test/test_jobs_controller_local.py
# Run specific google provider test
tox jobs/test/test_jobs_controller_local.py:TestJobsControllerLocal.test_abort_job

Generating requirements.txt

requirements.txt is autogenerated from requirements-to-freeze.txt. The latter lists only direct dependencies. To regenerate run:

virtualenv --python=/usr/bin/python2 /tmp/dsub-server-requirements
source /tmp/dsub-server-requirements/bin/activate

Then, from the dsub directory of this repo:

pip install -r requirements-to-freeze.txt
pip freeze | sort -f | sed 's/^jm-utils.*/\.\.\/jm_utils/g' > requirements.txt
deactivate

The sed command above replaces jm-utils=x.y.z with ../jm_utils, which is required to allow pip to install from the local jm_utils directory.

Fine-tune Gunicorn parameters

  • By default, the shim layer uses 5 Gunicorn workers. You can override the default number of workers by:

    export GUNICORN_CMD_ARGS="--workers=$NUMBER_OF_WORKERS"

    You can even use dynamic number of workers(based on the number of CPU cores), which is recommended, by:

    export GUNICORN_CMD_ARGS="--workers=$((2 * $(getconf _NPROCESSORS_ONLN 2>/dev/null || getconf NPROCESSORS_ONLN 2>/dev/null || echo 2) + 1))"

  • By default, the shim layer uses sync Gunicorn worker type. Since Job Manager also comes with gevent workers, you can override the default worker type by:

    export GUNICORN_CMD_ARGS="--worker-class gevent"

  • For convenience, you can consolidate the parameters in one command:

    export GUNICORN_CMD_ARGS="--workers=$((2 * $(getconf _NPROCESSORS_ONLN 2>/dev/null || getconf NPROCESSORS_ONLN 2>/dev/null || echo 2) + 1)) --worker-class gevent"

before you run the shim container.