Allow usage of different graders

[nwalters512]

All of Zephyr is currently hardcoded for the Catch grader. We'll need to allow Zephyr to use different graders depending on the course. It's likely that each piece of Zephyr will need a different solution for this.

Student CLI

For student usage, the workflow is simply running zephyr in an assignment directory. So, Zephyr will need to understand how to grade an assignment based only on the directory. For that, I'm thinking a zephyr.yml file in the root of an assignment that names the required grader. We can read that file to determine what to do. We could even be clever like npm and recursively search upward in the filesystem looking for a zephyr.yml file to identify an assignment root.

Staff CLI

This should likely be configurable via the config.yml that Zephyr already expects to set various repo info, rosters, etc. This could maybe also be overridable on a per-assignment basis, with assignment config taking precedence over course config.

PrairieLearn

Given that each course + grader could have a different set of dependencies, my current thought is actually that we'll just build one Docker image per grader. We could provide a plumbing package so that the only thing needed to do to wire up a grader to PL is something very roughly like

import grader from './grader.js'
import plRunner from '@illinois/zephyr-pl-runner'

// grader-specific code here...

plRunner.run(grader);

[nmagerko]

Would it be better to handle PrairieLearn-related stuff with a different issue? It seems like that would be a specific grader implementation (like a docker-runner implementation), but maybe I'm misunderstanding.

For now I was just planning to change Zephyr so that the student and staff CLI don't reference zephyr-catch-grader in particular when handling a run. Instead they'd just use some configurable input as you described.

[nwalters512]

Yep, that makes sense. As long as we keep the interface in such a way that it's still usable from PL, we're good.