fix(backend): Synced ScheduledWorkflow CRs on apiserver startup

[hbelmiro]

Resolves: #11296

This PR changes apiserver to patch existing ScheduledWorkflow CRs for each Job on startup to reflect the current KFP version deployed.

Testing

1. Create recurring runs

2. Make sure the recurring runs are running

3. Patch their swf CRs to force failures
   3.1 Get the swf CRs

   kubectl get swf
   NAME                             AGE
   heya8gqgg                        27m
   howdyxt5fj                       26m
   runofpipelinewithconditio5zwcj   60m

   3.2 For each swf patch them with an invalid workflow spec to force failures. At this point, the recurring runs will start to fail due to the invalid spec.

   kubectl patch scheduledworkflow heya8gqgg --type='merge' -p '{"spec":{"workflow":{"spec":{"dynamicKey1":"dynamicValue1","dynamicKey2":"dynamicValue2"}}}}'

4. Build a new apiserver image

5. Edit the ml-pipeline deployment to use the new apiserver image

6. The new apiserver pod will fix the swf CRs and the recurring runs will run successfully again

See a video demonstrating the test:

kfp-issue-11296.mp4

Checklist:

• You have signed off your commits
• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• backend/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hbelmiro]

@HumairAK @chensun can you guys PTAL?

[mprahl]

The persistence agent seems to already have a reconcile loop for scheduled workflows. If I'm reading the code right, on start up, it'll reconcile everything and then handle creates, updates, and deletes.

Could the migration logic be added to the persistence agent instead?

[hbelmiro]

It probably could. But that doesn't sound like a persistence agent's responsibility. It doesn't sound like API server's responsibility too, but I think it fits better there since we have a jobStore.

My original plan was to do it in https://github.com/kubeflow/pipelines/tree/master/backend/src/crd/controller/scheduledworkflow, but we would have to make HTTP calls to the API server. Then we decided to leave it in the API server.

It would be nice to hear others' opinions about that.

[mprahl]

@hbelmiro I agree that persistence agent isn't the best fit. That controller you linked could be a good fit.

I think we should consider adding a KFP version column to the jobs table so that you can skip generating and updating scheduled workflows if the version matches.

If were to pursue using the scheduled workflow controller, here is an idea:

• Have the scheduled workflow controller query the API server health endpoint at start up to get the tag_name value to see what version of KFP we are on. In the background, it could keep querying the API server to see if the version changed.
• The scheduled workflow controller's reconcile loop checks an annotation of pipelines.kubeflow.org/version and if the value of that annotation doesn't match tag_name, then the workflow definition is updated and the annotation value is set to the current version.
• When the API server creates a ScheduledWorkflow object, it sets the pipelines.kubeflow.org/version annotation.

[hbelmiro]

I love this idea. My only concern is about making HTTP calls to the API server.
How about implementing it in a follow-up PR?

[mprahl]

Sure, no problem! What do you think of the other comnment of adding a KFP version column to the jobs table so that you can skip generating and updating scheduled workflows if the version matches?

[hbelmiro]

I like that too. But other than https://github.com/kubeflow/pipelines/blob/master/VERSION, I'm not aware of a field/file from where I can get the version. Are you?
Otherwise, maybe @HumairAK and @chensun can recommend something.

[mprahl]

@hbelmiro I think TAG_VERSION can be used like is surfaced from the health endpoint.

[hbelmiro]

@mprahl where do you see it?

[mprahl]

I agree with your approach of not exiting with an error code in this case, but I'm curious what others think. My concern is that it'd be hard to know that this failed as a KFP admin but it also doesn't seem warranted to keep the API server from running if this can't succeed.

[hbelmiro]

If I was implementing this from scratch, I'd definitely exit with an error. But since the original behavior (the one that this PR aims to fix) is to run the API server with outdated swfs, existing deployments may start to fail after an upgrade even when they would work with outdated swfs.
At the same time, silently ignoring it and the outdated swfs "working" silently may be even more dangerous than just exiting with an error. (Now while I'm writing this, I'm tilting more to exit with an error)

Yeah, let's see what others think.

[mprahl]

Should this ignore not found errors? I could see the case where the database still has a reference to the scheduled workflow but the object no longer exists on the Kubernetes cluster.

[hbelmiro]

I'm not sure I'm following. Are you suggesting to ignore the error when the CR is not found? I'm already doing that, but not at this line. tmpl.ScheduledWorkflow(jobs[i]) actually just converts a Job object to a ScheduleWorkflow object.

[hbelmiro]

This PR depends on:

• chore(backend): Fixed namespace in job creation #11480
• chore(backend): Fixed ServiceAccount in job creation #11481

I included those commits here while the PRs are not merged.

cc @mprahl