Monitoring strategy

[alexpdp7]

I'd like to set up monitoring for Talos clusters, mostly to check for available updates.

I see two strategies to do this:

• Enable Talos API access from Kubernetes and create a simple HTTP deployment that exposes an endpoint that compares the installed Talos version to the latest available on GH. It could also do the same for Kubernetes using the Kubernetes API.
• Ship a Talos config along with talosctl and kubectl, and a script that uses both commands to do the same thing, but outside the cluster.

I think I like best the first option best, but do people have any preference?

Also, is the os:reader role designed for this? Should I use os:admin instead? I checked the docs and I see no documentation about generating a specific role.

Another option would be to expose a monitoring endpoint. Although, I'd suggest using HTTP instead of gRPC.

[alexpdp7]

Ah, I can extract the Talos version from the nodes. Implementing something at https://github.com/alexpdp7/talos-check .

[alexpdp7]

... and that repo above now builds the crudest container image possible to do that, has instructions to deploy the image to a cluster, and how to monitor using Nagios' check_http.

[smira]

Such controller doesn't have to run in the cluster btw, and os:reader is enough to query Version API.

The way we have this implemented in Omni is the following:

• grab all tags for the ghcr.io/siderolabs/installer container
• parse them, and filter the versions using the supported upgrade contracts:

  talos/pkg/machinery/compatibility/talos_version.go

  Lines 41 to 77 in 9e8603f

  	// UpgradeableFrom checks if the current version of Talos can be used as an upgrade for the given host version.
  	func (v *TalosVersion) UpgradeableFrom(host *TalosVersion) error {
  	var (
  	minHostUpgradeVersion, maxHostDowngradeVersion *version.Version
  	deniedHostUpgradeVersions []*version.Version
  	)
  	switch v.majorMinor {
  	case talos12.MajorMinor: // upgrades to 1.2.x
  	minHostUpgradeVersion, maxHostDowngradeVersion = talos12.MinimumHostUpgradeVersion, talos12.MaximumHostDowngradeVersion
  	deniedHostUpgradeVersions = talos12.DeniedHostUpgradeVersions
  	case talos13.MajorMinor: // upgrades to 1.3.x
  	minHostUpgradeVersion, maxHostDowngradeVersion = talos13.MinimumHostUpgradeVersion, talos13.MaximumHostDowngradeVersion
  	deniedHostUpgradeVersions = talos13.DeniedHostUpgradeVersions
  	case talos14.MajorMinor: // upgrades to 1.4.x
  	minHostUpgradeVersion, maxHostDowngradeVersion = talos14.MinimumHostUpgradeVersion, talos14.MaximumHostDowngradeVersion
  	deniedHostUpgradeVersions = talos14.DeniedHostUpgradeVersions
  	default:
  	return fmt.Errorf("upgrades to version %s are not supported", v.version.String())
  	}
  	if host.version.Core().LessThan(minHostUpgradeVersion) {
  	return fmt.Errorf("host version %s is too old to upgrade to Talos %s", host.version.String(), v.version.String())
  	}
  	if host.version.Core().GreaterThanOrEqual(maxHostDowngradeVersion) {
  	return fmt.Errorf("host version %s is too new to downgrade to Talos %s", host.version.String(), v.version.String())
  	}
  	for _, denied := range deniedHostUpgradeVersions {
  	if host.version.Equal(denied) {
  	return fmt.Errorf("host version %s is denied for upgrade to Talos %s", host.version.String(), v.version.String())
  	}
  	}
  	return nil
  	}

[alexpdp7]

Ah, that is nice! But I'm not handy with Go :)

Well, I polished a bit https://github.com/alexpdp7/talos-check , so it has a CLI tool that generates manifests to deploy it (instead of using kubectl create commands. I find it very tedious to compose Kubernetes manifests file. The CLI tool uses a bizarre way to generate the manifests), so I think it can be reasonably usable.

[alexpdp7]

(My talos-check repo is now working correctly for this. It's a deployment that runs in cluster, using a cluster role with list/get nodes.)