Metrics collection guidance

[adriangb]

Is there a high-level playbook for metrics collection?

I imagine that:

• app/language specific things (like garbage collection metrics) must be collected by the sdk at the app-level
• host metrics like CPU usage can probably be collected many different ways (app/sdk level, scraping, etc.). Is there any guidance on this?
• infrastructure metrics (like cluster CPU usage) obviously can only be collected at the OTEL collector level. Are there any examples of setting this up? I suppose leveraging Prometheus here is one way to go (applies to above as well), but I'm not sure if that's appropriate for clean slate (i.e. currently unmonitored) projects?

Thanks!

[Konig-Corey]

Hello,

I am by no means a supporter of this project so Im sure there others that will chime in with their own thoughts. The past week I have been doing a proof of concept around using open telemetry as a full replacement for our host metric collection + app metric collection + application tracing. In my findings I have found some things that would likely help point you in the right direction.

Host metric collection: the receiver: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/hostmetricsreceiver can be used to collect metrics about the host (cpu, load, disk, memory etc)

App metric collection: Many languages support statsD. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver This allows otel to listen for statsd metrics over http

Additional App Metric + Tracing: open telemetry has their own listener that can either use grpc or http and can be used to receive both metrics and traces. There is pretty common support for this with libraries for a variety of langaues like go, python, and java. https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver

Those three receivers can be paired with a variety of exporters to send that data wherever youd like. One thing I noticed as well was that you may have to use some processesors like "resource" or "resourcedetection" to append additional metadata to metrics. We leveraged the "resource" processor to append some custom labels on our metrics like auto scaling group, service etc.

I hope this helps you in the right direction. :)