STYLE: Update CONTRIBUTING.md to test slicer/apidocs webhook

[jcfr]

This PR is indented to test that the slicer API docs builder is effectively triggered.

[jcfr]

Et voila !

It works, the following set of documentation are now automatically generated and published:

• If updated, master branch is always built and published on https://slicer.github.io/apidocs.slicer.org/master/
• Branches associated with a PR (e.g test-apidocs-webhook associated with STYLE: Update CONTRIBUTING.md to test slicer/apidocs webhook #690) is published as https://slicer.github.io/apidocs.slicer.org/test-apidocs-webhook/
• Documentation for tagged releases is also published:
  • https://slicer.github.io/apidocs.slicer.org/v4.5/
  • https://slicer.github.io/apidocs.slicer.org/v4.6/

Note that each time a patch release is done (e.g v4.6.3), documentation is pushed to the v4.6 and overwrite the previous one. (If needed, this could easily be changed)

infrastructure

Now, few words about the infrastructure enabling this:

• Slicer/slicer-apidocs-builder: A pip installable package able to checkout a specific Slicer version, build corresponding doxygen documentation and then publish it as a subfolder on a particular branch of a project (e.g see directories associated with Slicer/apidocs.slicer.org@gh-pages). Last, it also allows to update the GitHub status state and description associated with of a given commit. This is the slicer/apidocs check you can see above !

• Slicer/apidocs.slicer.org: This is the project allowing to publish the documentation to the gh-pages branch. The associated CircleCI configuration uses slicer-apidocs-builder to build&publish a given version of the documentation (based on environment variables and also update the GitHub status for the corresponding commit.

• Slicer/github-circleci-trigger: This one is the "glue" making the magic happen. It is basically a flask application deployable on https://heroku.com. It implements the webhook https://github-circleci-trigger.herokuapp.com/postreceive that is associated with Slicer/Slicer project and trigger the CircleCI build associated with Slicer/apidocs.slicer.org project. The new build is parametrized with SLICER_REPO_NAME, SLICER_REPO_BRANCH, SLICER_REPO_TAG and SLICER_REPO_REVISION.

What is nice about heroku is that they provide a free "dyno" instance (512 MB RAM │ 1 web/1 worker) along with a pool of 1000 hours per account. The instance falls asleep after 30mins and is wake up fairly quickly. More than enough for implementing a simple webhook !

why a dedicated webhook ?

We could have added/updated a travis/circleci/appveyor config directly within the Slicer project to build and publish the documentation but ...

... since we want the documentation to also be published for pull request and we do not want outside random developers to discover our secrets (aka GitHub token) that we would be using to publish the documentation, having a dedicated webhook allow to both nicely decouple things and leverage additional CircleCI workers.

Notes:

• the Github token used in the CircleCI build that is associated with Slicer/apidocs.slicer.org project belong to @slicerbot who has has been granted access to only few relevant projects.

• the webhook will activate only if client IP is from GitHub knows IPs. See https://developer.github.com/v3/meta/

• a secret set when configuring the hook and specific to the dyno instance running the webook is also used to validate provenance of the request. It set X-Hub-Signature is the request and check hash matches as explained in https://developer.github.com/webhooks/securing/

next

Here is a list of todos:

• webhook
  • rename webhook server from https://github-circleci-trigger.herokuapp.com/postreceive to https://slicer-webhooks.herokuapp.com/postreceive (or https://slicer-apidocs-webhooks.herokuapp.com/postreceive). Note that having a more general project github-circleci-trigger on GitHub still makes sense but the specific instance of the server is specific to Slicer ....and should be named accordingly
  • add an endpoint accessible from outside world that is showing the properties of the server ...
  • add an endpoint allowing to get the status of the apidocs documentation for given branch, tag, ... it could be used for a badge
• website
  • Check with @pieper and @mhalle so that they setup redirect from http://apidocs.slicer.org to https://slicer.github.io/apidocs.slicer.org. The custom domain has already been added to the GitHub project. Configuring the DNS record is only what is left. See https://help.github.com/articles/quick-start-setting-up-a-custom-domain/
  • Setup a direct from https://www.slicer.org/doc/html/ to apidocs.slicer.org/master/
  • Review layout, doxygen documentation should probably be pushed to apidocs.slicer.org/master/doxygen
  • add top level page to apidocs.slicer.org
  • publish documentation from pull request into a different website/project (e.g Slicer/apidocs-preview.slicer.org). This will avoid polluting the main domain. That repository would be cleared up once a week.
  • include Python API documentation
• manual trigger
• In order to trigger build and publication of the previous release, the github-circleci-trigger project should be usable as CLI to trigger bulds.

Note also that since the whole system has been put together in only few days, I definitively expect some hiccup.

screenshots

webhook in action

webhook configuration

[ihnorton]

The Doxygen publication Python script is great. I think the PR integration is "nice-to-have" but I wonder if it is worth the extra complication of hooks/Heroku/CircleCI/etc.

Please consider whether this will be maintainable by community members. I'm not sure how Heroku and CircleCI permissions work, but I think we need to aim for several people having enough permissions to implement fixes independently. We can all figure this stuff out, but that is less likely to happen if we must also chase access to several different systems in order to even start debugging. (which is the case for the current API doc builder, as well as the nightly factory and other infrastructure)

[lassoan]

I agree, simpler is better and it would be important for all core developers to be able to access the infrastructure and troubleshoot problems. If external access to Kitware computers is difficult then we can set up documentation builds on perklab-factory and give remote access to all core developers.

[jcfr]

Thanks for the feedback.

Just to refocus the discussion, in this case, building the doxygen per se was never the issue.

The problem was:

• on the deployment side where the script (in charge of downloading the built documentation and installing it on the Slicer webserver) wasn't updated when the infrastructure transitioned to digital ocean for hosting wiki et al. More details here.
• we didn't have monitoring in place to report there was a problem

After talking with @pieper and @mhalle, updating digital ocean to serve the page is not feasible. For that reason, we looked into github pages.

The solution discussed in this PR is intended to:

• give clear feedback when the documentation is being generated, failed to build or is deployed. Thanks to the slicer/apidocs context
• streamline the release process removing manual steps (before we didn't even bother having API documentation for a previous version of Slicer ...)
• allow the community to maintain, debug and improve the infrastructure
• leverage existing free resources available for open source project. No "who's credit card" dilemma.

Access to account by the community:

• management of the heroku can be shared

• management of the slicerbot GitHub user will be shared by associated the usermail with a private mailing list. That way, any one could reset the password if needs to be.