Merge pull request #141 from lucyparsons/develop

Release v0.1.2

.gitignore

@@ -6,6 +6,12 @@
 # Credentials
 dbcred*
 
+# Local environment files
+**/.env
+
+# Swapfiles from our text editors
+*.swp
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -23,7 +29,6 @@ dist/
 downloads/
 eggs/
 .eggs/
-lib/
 lib64/
 parts/
 sdist/
@@ -73,7 +78,9 @@ target/
 Gemfile.lock
 
 #Puppet module stuff
-vagrant/puppet/modules/
 vagrant/puppet/.tmp
 .librarian
 .vagrant
+
+# Editor files
+.idea/

.travis.yml

@@ -0,0 +1,13 @@
+language: python
+python:
+  - "2.7"
+install:
+before_script:
+  - pip install -e .
+  - pip install -r requirements.txt
+  - pip install coveralls
+  - cd OpenOversight
+script:
+  - py.test -v --cov=app
+after_success:
+  - coveralls

CONTRIB.md

@@ -8,52 +8,56 @@ When you come to implement your new feature, you should branch off `develop` and
 
 ## Development Environment
 
-If you don't have bundler installed:
+Our standard development environment is an Ubuntu 14 VM. We manage it with Vagrant, which means you'll need Vagrant and Virtualbox installed to start out.
 
-`gem install bundler`
+Make sure that vagrant and Virtualbox are installed, and then run:
 
-Then provision the VM:
+`vagrant up`
 
-`rake vagrant:provision`
-
-This brings the vagrant box up and you can now SSH into it:
+This creates a new, pristine virtual machine and provisions it to be an almost-copy of production with a local test database. (Behind the scenes, this is all happening via the files in vagrant/puppet.) If everything works, you should get a webserver listening at `http://localhost:3000` you can browse to on your host machine. In addition, you can now SSH into it:
 
 `vagrant ssh`
 
-You can access the PostgreSQL development database via psql using:
-
-`psql  -h localhost -d openoversight-dev -U openoversight --password`
+The app, as provisoined, is running under gunicorn, which means that it does not dynamically reload your changes.
 
-with the password `terriblepassword`. 
+If you run the app in debug mode, you can see these changes take effect on every update, but certain changes will kill the server in a way some of us find really irritating. To do this:
 
-For the webapp, the credentials for the testing/development environment are expected to be in a file `$PGPASS`, so set that up: 
+`vagrant ssh` (if you're not already there)
+```
+$ sudo service gunicorn stop
+ * Stopping Gunicorn workers
+ [oo] *
+vagrant@vagrant-ubuntu-trusty-64:~$ cd /vagrant/OpenOversight/
+vagrant@vagrant-ubuntu-trusty-64:/vagrant$ ~/oovirtenv/bin/python ./run.py
+ * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
+ * Restarting with stat
+ * Debugger is active!
+```
 
-`echo "localhost:5432:openoversight-dev:openoversight:terriblepassword" >> ~/.pgpass`
+You can access your PostgreSQL development database via psql using:
 
-`echo "export PGPASS=~/.pgpass" >> ~/.bashrc`
+`psql  -h localhost -d openoversight-dev -U openoversight --password`
 
-`source ~/.bashrc`
+with the password `terriblepassword`.
 
-In the `/vagrant/OpenOversight` directory, there is a script to create the database:
+The provisioning step already does this, but in case you need it, in the `/vagrant/OpenOversight` directory, there is a script to create the database:
 
 `python create_db.py`
 
 If the database doesn't already exist, `create_db.py` will set it up and store the version in a new folder `db_repository`. 
 
-
-After you create the database, run `test_data.populate()` to put test officers, assignments, and images into the database. 
-
-
-
-
+In the event that you need to create or delete the test data, you can do that with
+`python test_data.py --populate`
+or
+`python test_data.py --cleanup`
 
 ## Running Unit Tests
 
  Run tests with `nose`:
 
-```nosetests -v```
+```~/oovirtenv/bin/python ~/oovirtenv/bin/nosetests```
 
-Note: One could put `test_data.populate()` into `setUp` and `test_data.cleanup()` into `tearDown` but in this case we want the data to stay in the database so that we can play with the web application so we should just have vagrant run that during the provisioning of the development VM. 
+Note: One could put `test_data.populate()` into `setUp` and `test_data.cleanup()` into `tearDown` but in this case we want the data to stay in the database so that we can play with the web application so we should just have vagrant run that during the provisioning of the development VM.
 
 ## Migrating the Database
 
@@ -63,3 +67,23 @@ If you e.g. add a new column or table, you'll need to migrate the database. You
 
 to do this.
 `python upgrade_db.py` and `python downgrade_db.py` can also be used as necessary. Note that I followed [this tutorial](http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-iv-database) to set this up.
+
+## Changing the Development Environment
+
+If you're making massive changes to the development environment provisioning, you should know that Vagrant and the Puppet modules that provision the box use Ruby, so you'll want some reasonably-modern Ruby. Anything in the 2.0-2.2 area should work. Puppet has some annoying interactions where puppet 3 doesn't work with ruby 2.2, though, so you might have to get creative on modern OSes.
+
+If you don't have bundler installed:
+
+`gem install bundler`
+
+If you don't have rake installed:
+
+`bundle install`
+
+Then provision the VM:
+
+`rake vagrant:provision`
+
+Puppet modules are dropped into place by librarian-puppet, and there's a rake task that'll do it without the headache of remembering all the paths and such:
+
+`rake vagrant:build_puppet`

DEPLOY.md

@@ -4,13 +4,7 @@ When this application is deployed you will need to do some setup. These instruct
 
 # Dependencies
 
-At minimum you will need to install the pip packages using pip:
- * flask
- * werkzeug
- * Flask-WTF
- * psycopg2
- * sqlalchemy
- * gunicorn
+We distribute a `requirements.txt` file listing the things the application depends upon. `pip install -r requirements.txt` will install prerequisites.
 
 You may also need the `libpq-dev` package if psycopg2 fails to install.
 
@@ -54,30 +48,21 @@ server {
 ```
 ##  Gunicorn
 
-You will want to run gunicorn in the background with multiple worker threads. You can instantiate gunicorn by running `gunicorn -w 4 -b 127.0.0.1:4000 app:app &` and then run ps -ef | grep gunicorn to see the running PIDs. Execute gunicorn out of the OpenOversight directory that has the `app/` directory in it.
+You will want to run gunicorn in the background with multiple worker threads. You can instantiate gunicorn by running `gunicorn -w 4 -b 127.0.0.1:4000 app:app &` and then `run ps -ef | grep gunicorn` to see the running PIDs. Execute gunicorn out of the OpenOversight directory that has the `app/` directory in it to test that the app comes up.
 
-##  Database Configuration
+##  Application Configuration
 
-Your database credentials should be set in the `app/` folder in a file called dbcred.py. An example configuration is below (assuming an Amazon RDS).
+We configure the database by setting a .env file in the OpenOversight directory of the repository. A sample .env is:
 ```
-user =""
-password = ""
-host = "infomarmationfromamazon.rds.amazonaws.com"
-port = 6666
-```
-
-## CSRF attacks
-
-Change the secret key used for generating tokens to prevent cross-site request forgery (CSRF) attacks in `config.py`:
-
-```
-WTF_CSRF_ENABLED = True
-SECRET_KEY = 'changemeplzorelsehax'
+SQLALCHEMY_DATABASE_URI="postgresql://openoversight:terriblepassword@localhost/openoversight-dev"
+SECRET_KEY=terriblecsrftoken
 ```
+The parts of the database URI are the user, password, server, and database respectively used to connect to the application.
+The CSRF token should be a random string of reasonable length. 'terriblecsrftoken' is, of course, a terrible CSRF token.
 
 # Systemd
 
-You can write a simple systemd unit file to launch OpenOversight on boot. We defined ours in `/etc/systemd/system/openoversight.service`. You should create the proper usernames and groups that are defined in the unit file since this allows you to drop privileges on boot. This unit file was adopted from this [DigitalOcean guide](https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-centos-7). We define a static environment variable called `PGPASS` which is loaded by systemd before it drops priviledges for ngingx. More details can be found in `CONTRIB.md`.
+You can write a simple systemd unit file to launch OpenOversight on boot. We defined ours in `/etc/systemd/system/openoversight.service`. You should create the proper usernames and groups that are defined in the unit file since this allows you to drop privileges on boot. This unit file was adopted from this [DigitalOcean guide](https://www.digitalocean.com/community/tutorials/how-to-serve-flask-applications-with-gunicorn-and-nginx-on-centos-7). More details can be found in `CONTRIB.md`.
 
 ```
 [Unit]
@@ -89,13 +74,15 @@ User=nginx
 Group=nginx
 WorkingDirectory=/home/nginx/oovirtenv/OpenOversight/OpenOversight
 Environment="PATH=/home/nginx/oovirtenv/bin"
-Environment="PGPASS=~/.pgpass"
-ExecStart=/usr/local/bin/gunicorn -w 16 -b 127.0.0.1:4000 --timeout 90 app:app 
+ExecStart=/usr/local/bin/gunicorn -w 4 -b 127.0.0.1:4000 --timeout 90 app:app 
 
 [Install]
 WantedBy=multi-user.target
 ```
+# Python Fabric
+
+We use [Python Fabric](http://www.fabfile.org/) to manage our deployments and database backups. A sample fabric file is found in `fabric.py`. The usage is `fab host command`, so for example `fab staging deploy` would deploy our latest commits to the staging server.
 
 # Contact
 
-If you're running into installation problems, please open an issue or email us `info AT lucyparsonslabs DOT com`
+If you're running into installation problems, please open an issue or email us `info AT lucyparsonslabs DOT com`.

OpenOversight/app/__init__.py

@@ -1,8 +1,29 @@
-from flask import Flask
+from flask import Flask, render_template
 from flask_sqlalchemy import SQLAlchemy
+from config import config
 
-app = Flask(__name__)
-app.config.from_object('config')
-db = SQLAlchemy(app)
+def create_app(config_name='default'):
+    app = Flask(__name__)
+    app.config.from_object(config[config_name])
+    config[config_name].init_app(app)
+    from .models import db
+    db.init_app(app)
 
-from app import views, models, utils
+    from .main import main as main_blueprint
+    app.register_blueprint(main_blueprint)
+
+    @app.errorhandler(404)
+    def page_not_found(e):
+        return render_template('404.html'), 404
+
+    @app.errorhandler(403)
+    def forbidden(e):
+        return render_template('403.html'), 403
+
+    @app.errorhandler(500)
+    def internal_error(e):
+        return render_template('500.html'), 500
+
+    return app
+
+app = create_app()

OpenOversight/app/config.py

@@ -0,0 +1,58 @@
+import os
+from os.path import expanduser
+from dotenv import load_dotenv, find_dotenv
+
+load_dotenv(find_dotenv())
+
+basedir = os.path.abspath(os.path.dirname(__file__))
+
+
+class BaseConfig(object):
+    # DB SETUP
+    SQLALCHEMY_MIGRATE_REPO = os.path.join(basedir, 'db_repository')
+    SQLALCHEMY_TRACK_MODIFICATIONS = False
+
+    # File Upload Settings
+    UNLABELLED_UPLOADS = 'uploads/'
+    ALLOWED_EXTENSIONS = set(['png', 'jpg', 'jpeg', 'mpeg', 'mp4'])
+
+    # pagination
+    OFFICERS_PER_PAGE = os.environ.get('OFFICERS_PER_PAGE', 20)
+
+    # Form Settings
+    WTF_CSRF_ENABLED = True
+    SECRET_KEY = 'changemeplzorelsehax'
+
+    NUM_OFFICERS = 15000
+    SEED = 666
+
+    @staticmethod
+    def init_app(app):
+        pass
+
+
+class DevelopmentConfig(BaseConfig):
+    DEBUG = True
+    SQLALCHEMY_DATABASE_URI = os.environ.get('SQLALCHEMY_DATABASE_URI')
+
+
+class TestingConfig(BaseConfig):
+    TESTING = True
+    SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'
+    WTF_CSRF_ENABLED = False
+
+
+class ProductionConfig(BaseConfig):
+    SQLALCHEMY_DATABASE_URI = os.environ.get('SQLALCHEMY_DATABASE_URI')
+
+    @classmethod
+    def init_app(cls, app): # pragma: no cover
+        Config.init_app(app)
+
+
+config = {
+    'development': DevelopmentConfig,
+    'testing': TestingConfig,
+    'production': ProductionConfig,
+    'default': DevelopmentConfig
+}

OpenOversight/app/db_repository/README

@@ -0,0 +1,4 @@
+This is a database migration repository.
+
+More information at
+http://code.google.com/p/sqlalchemy-migrate/

OpenOversight/app/db_repository/manage.py

@@ -0,0 +1,5 @@
+#!/usr/bin/env python
+from migrate.versioning.shell import main
+
+if __name__ == '__main__':
+    main()

OpenOversight/app/db_repository/migrate.cfg

@@ -0,0 +1,25 @@
+[db_settings]
+# Used to identify which repository this database is versioned under.
+# You can use the name of your project.
+repository_id=database repository
+
+# The name of the database table used to track the schema version.
+# This name shouldn't already be used by your project.
+# If this is changed once a database is under version control, you'll need to
+# change the table name in each database too.
+version_table=migrate_version
+
+# When committing a change script, Migrate will attempt to generate the
+# sql for all supported databases; normally, if one of them fails - probably
+# because you don't have that database installed - it is ignored and the
+# commit continues, perhaps ending successfully.
+# Databases in this list MUST compile successfully during a commit, or the
+# entire commit will fail. List the databases your application will actually
+# be using to ensure your updates to that database work properly.
+# This must be a list; example: ['postgres','sqlite']
+required_dbs=[]
+
+# When creating new change scripts, Migrate will stamp the new script with
+# a version number. By default this is latest_version + 1. You can set this
+# to 'true' to tell Migrate to use the UTC timestamp instead.
+use_timestamp_numbering=False