Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

customDatasources: allow to run external binaries to generate result #23410

Open
secustor opened this issue Jul 17, 2023 · 3 comments · May be fixed by #33272
Open

customDatasources: allow to run external binaries to generate result #23410

secustor opened this issue Jul 17, 2023 · 3 comments · May be fixed by #33272
Labels
datasource:custom priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others type:feature Feature (new functionality)

Comments

@secustor
Copy link
Collaborator

secustor commented Jul 17, 2023

Describe the proposed change(s).

This feature is only intended for self hosted instances because of the security implications!
Also it should be not enabled by default!

There should be a new global config option for limiting the allowed commands: allowedCustomDatasourceCommands.
This should mimic allowedPostUpgradeCommands.

Further there should be a new option

{
  "customDatasources": {
    "foo": {
      "registryCommandTemplates": [
        "echo {\"foo\":\"bar\"}' > myRegistry.json",
        "cat myRegistry.json"
      ],
      "...": ""
    }
  }
}

This will run all commands of registryCommands serially and will capture the output ( stdout ) of all commands and concat it at the and.

The resulting string is then be processed depending on the chosen format

Describe why we need/want these change(s).

#23286 (comment)

Currently if using the custom datasource there is no way we can support every use case or API. A step to allow even more concepts would be to option to handover the gathering of the raw data to an external process.

An example would be fetching files from an network share and then handing over the contents to Renovate or running queries from CLI tools such as aws or az for the respective cloud providers.

@secustor secustor added priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others status:ready datasource:custom type:feature Feature (new functionality) labels Jul 17, 2023
@chasecaleb
Copy link

This would be great! I would be equally happy if renovate could read versions from a (JSON) file instead of running commands too, which would accomplish the same thing if you run the commands right before running renovate. I think this might even be preferable because it decouples renovate from the registry commands and it's easier and safer to implement for renovate.

I have a few use cases at work where I'm considering writing a lightweight service that runs commands like this and then wraps the result in a customDatasource-compatible API response, but it would be nice to skip all the indirection and extra work of writing and deploying a service. Amazon EKS addons is one example: there isn't a Renovate data source and they don't have a (documented) API for it, but you can use the aws CLI to output them.

@z0rc
Copy link

z0rc commented Jan 4, 2024

they don't have a (documented) API for it

They do: https://docs.aws.amazon.com/eks/latest/APIReference/API_DescribeAddonVersions.html

Side note, maybe EKS addons datasource can be implemented as native option? Just like https://docs.renovatebot.com/modules/datasource/aws-machine-image/. As currently this DescribeAddonVersions API require AWS auth, which can be a nuisance on its own.

@taraspos
Copy link
Contributor

taraspos commented Sep 9, 2024

In case of someone looking to renovate EKS Addons, I published (a bit over-engineered, but viable) solution that works well for me with GitHub Actions self-hosted renovate:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datasource:custom priority-3-medium Default priority, "should be done" but isn't prioritised ahead of others type:feature Feature (new functionality)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants