Proposal: implement a manifest style validation tool for krew-index specifically

[ahmetb]

Currently, our plugin manifest validation functionality which is in the YAML parsing path has some style/cosmetic checks that have no effect on plugin installation. This is a result of not separating "cosmetic validation" and "functional validation".

If we continue on this trajectory, we will be adding more "cosmetic validation" in future versions, which will break older plugin manifests. So we can't do that. Furthermore, with custom plugin indexes (#566), we cannot control the plugin manifests out in the wild –but we have to be able to continue working with them.

I propose that we strip off these "cosmetic validations" from the logic in krew machinery, and create a krew_index_manifest_validator tool, specifically for krew-index that runs as part of its CI for each PR.

This would help reduce code review toil we have been manually enforcing in krew-index, as well as separate structural validation vs cosmetic validation, such as:

• shortDescription must not contain line break characters
• shortDescription must not have a a trailing period (.)
• description must be wrapped at 80 character lines
• explicit files: {from: '*', to: '.'} is not needed, drop
• after executing files on specified archive, there should be a LICENSE file in the directory.
• [... other stuff that I can't think about right now...]

---

The problem is: We already have some validation in cmd/validate-krew-manifest tool that is run only for krew-index. This validation:

1. runs all structural validation through Validate()
2. ensures: all spec.platforms[] items are used
3. ensures: no overlapping spec.platform[].selectors
4. ensures: all spec.platforms[] install fine

• point 1: Already validated by krew, during parsing
• point 2: Not a functional validation, if there's unused platforms[], krew continues to work fine.
• point 3: I believe Krew continues to work fine when there are overlapping selectors –though, the behavior is undefined (not sure which one would be picked up, as we don't guarantee it). We could move this to "functional validation", though I'm not sure we totally should.
• point 4: Not "structural validation", just ensures plugins merged to krew-index are installable, we can't enforce this in custom indexes.

So maybe we just make cmd/validate-krew-manifest more about "styling" and make it clear it's meant for krew-index.

---

/kind proposal
/cc @corneliusweig

[corneliusweig]

This is a sound plan. The only thing I'd add is that we strive to build our krew-index manifest validation tool in a way that it can be re-used by other projects. After all, everybody should be able to easily check what we think are good practices for plugin manifests.

[ahmetb]

yeah. I think it would be reusable by public indexes out of the box. private indexes have different requirements (no LICENSE etc).

[ahmetb]

tl;dr I'm thinking we should rename the existing tool to krew_index_manifest_validator and extend its capabilities to more cosmetics validation.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[corneliusweig]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[corneliusweig]

/remove-lifycycle stale
/freeze
I think this is important long-term and we should work on this eventually.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[corneliusweig]

/reopen
/remove-lifecycle rotten
/good-first-issue

[k8s-ci-robot]

@corneliusweig: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten
/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.