[common] Add honeytail as an addon

[mterhar]

Description of the change

This is similar to promtail but rather than sending to grafana, it goes to Honeycomb which has a free plan that should cover homelab usage. Not trying to add a costly subscription.

It defaults to off so most users should see this change as a noop. If they want to enable tailing for any given service, they can do so using the addon yaml.

Benefits

I prefer Honeycomb to Promtail and would like this as the low-effort option to get something into honeycomb. The next iteration on this is to upstream actual open telemetry emission to the various projects so the output is more useful and meaningful.

Possible drawbacks

The way data is sent to Honeycomb requires that it be parsed into key/value pairs. Many of the applications in k8s-at-home emit a more traditional style of console.line or even multi-line outputs. It requires a bit of Regex effort.

For example:

• Emby

  • Creates multiple log files for transcodes so those logs don't get sent
  • Requires this regex to send back the latency and other data(?P<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d+) (?P<level>\w+) \w+: (?P<protocol>\S+) (?P<method>\S+) (?P<status>\S+) to (?P<client_ip>\S+). Time: (?P<latency_ms>\d+)ms. (?P<url>http.*$)

• Sonarr and Radarr

  • Log output is much simpler so this regex gets what's there: (?P<time>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d+)\W(?P<level>\w+)\W(?P<method>\w+)\W(?P<msg>.*)
  • Doesn't include latency information
  • Exceptions are blown out into dozens of lines

• Nzbgets logging is a nightmare and unlikely to be usable without being rewritten or otel instrumented

Additional information

Checklist

• Chart version bumped in Chart.yaml according to semver.

[ghost]

👋 @mterhar, thanks for taking the time to submit this PR. 🙏🏽

We have noticed that the PR title does not adhere to our desired format.
Could you please make sure that it follows the pattern [<lowercase chart name>] <Update description>?
Example: [radarr] Update image version.

Thanks!

[onedr0p]

While I appreciate the PR we were actually thinking about deprecating promtail in favor of Vector. Vector is a much better choice because it supports a lot of different sinks including honeycomb, loki, and others.

https://vector.dev/docs/reference/configuration/sinks/honeycomb/

I would rather have one addon for Vector than maintain a bunch of different addons for each log aggregator. I hope that makes sense.

[mterhar]

I would rather have one addon for Vector than maintain a bunch of different addons for each log aggregator. I hope that makes sense.

Sounds look a superior future plan. As I threw in the description, what's really best is having the ability to trace from the ingress through the app servers so the handoff latency can be tracked. I planned on using this PR's contents as a starting point to enhance apps with the ability to generate spans and then figure out the best ingress approach that can start a span and include the right baggage.

Vector as a log shipper is fine but abandons the otel aspect and so we'd be stuck with classic grep-style troubleshooting.

[onedr0p]

I am curious, since Sonarr and most applications log to stdout/stderr, why not deploy the Vector agent with helm and have that run as a daemonset on your nodes to ship container logs to Honeycomb? I am doing this but with using Loki.

E.g.
https://github.com/onedr0p/home-ops/tree/main/cluster/apps/monitoring/vector

Here I am sending container logs and journald logs (from the OS) to Loki (which would be Honeycomb in your instance). The main reason we included promtail was because some containers do not log to stdout/stderr like Plex. Promtail is used to read the log files in the Plex container and ship them to Loki, which if we replaced the promtail addon with Vector would open up to shipping file logs to any sink supported by Vector.

[mterhar]

As discussed in Discord...

• this PR is a partial step and it's an old method
• The real goal is yo start sending some stuff to a destination that can receive traces and submit PRs to the apps so they're do open telemetry
• Using vector instead is probably a more flexible option
• Using an operator like vector to ship all container logs is solving a different problem
• Honeycomb has an argent/operator too but the types of things sent to it aren't super useful

We can leave this as a draft for now. I will see about redoing the Dockerfile so it pulls vector and configures a honeycomb sink.