Exporter Developer (=me) wants to switch from Prometheus scrape endpoints to OTel push clients, pushing into Prometheus

[thediveo]

Background/Warning: I'm the at least mildly confused guy from one of the OTel transition presentation slide decks that can't yet his head around the Go OTel SDK, so begging for help here. I simply struggle getting the picture with meter providers, propagators, meters, ...

Please note that I'm not in the situation of instrumenting HTTP handlers or similar where the OTel Go SDK has lots of examples. Instead, I'm in the situation of having collected metrics via completely OTel-independent means, such as Linux kernel APIs, et cetera. Unfortunately, for the metrics I need to collect, there are often neither Prometheus nor OTel exporters available, so that nicely puts food on my table.

Coming from the Prometheus Go client, in the past I just whipped up something that implements the prometheus.Collector interface, that is, Describe(...) and Collect(...), often just using MustNewConstMetric(...) for creating collect-able metrics from the data I gathered via different means outside the Prometheus and OTel SDKs. Then simply providing the client's scrape endpoint with a promhttp.HandlerFor(...) using the prometheus.NewRegistry().Register(mycollector). I'm not sure if I miss the obvious SDK parallels here, but I'm not seeing the OTel SDK wood for its trees.

So, I simply want to export a bunch of counter metrics, easy to do so far using the Prometheus Go client API. Lots of documentation also covering my use case where I collect the metrics outside the SDK. Now looking at the OTel metrics SDK 😭

Which elements do I need to create and wire up in case of the OTel Go SDK, to push my metrics gathered outside of the SDK to, say, Prometheus using the OTel protocol? (HTTP one in case of Prometheus?)

Please note that I don't want to throw the OTel collector into the mix, making an otherwise minimalist setup overly complex.

[dmathieu]

You may be looking for this prometheus example, which shows how to create metrics and expose them to the prometheus client.
https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/examples/prometheus/main.go

[thediveo]

This unfortunately doesn't look like I was asking for. First, I already have a working Prometheus exporter that is easy to maintain and have no reason to replace it with a less well understood OTel package. In fact, Prometheus' Go client SDK has the "const" Elements exactly for my use case, whereas OTel seems to lack them.

Second, I want to push export using the OTel protocol, not Prometheus protocol, even if I then push into Prometheus as one configuration using the OTel protocol.

Third, I already get metrics and counters elsewhere; your code suggestion would need to calculate the differences between the real metrices and the current OTel metrics state and then add the difference. You'll surely agree that this would be not only awkward, but not passing any code review.

How can I push metrics that come from sources outside the OTel meter objects with their own hidden state, via the OTel protocol into an OTel receiver? Surely the OTel SDK writers support this use case?

If not, then this would basically mean that we need to scratch OTel from our system architecture blueprints, something that would be sad, but if there is no sufficiently semi-mature SDK support coming along, when why waste DEV resources on an SDK that cannot replace the existing ones? Hopefully, OTel isn't another instance of the famous XKCD comic about standards...?

[dmathieu]

If you're using Prometheus and don't wish to change, that's fine. But the OTel SDK won't be able to help you there.

However, if you add an OpenTelemetry Collector to your stack, you can use the prometheus receiver to fetch data from your metrics endpoint.
The collector also has a prometheus exporter that would allow you to export OTLP data back to a prometheus instance.

The Prometheus library doesn't allow exporting data in OTLP. If that's the feature you need, you'd have to reach out to them. We don't control the their roadmap.
They do include an OTLP receiver though: https://prometheus.io/docs/guides/opentelemetry/

[thediveo]

I have no idea what you are answering to, something gone missing?

Let me repeat for the third time,

• some counter metrics outside your SDK,
• how do I get these metrics to the OTel protocol speaking grpc-over-http metrics exporter?
• I'm then feeding the OTel protocol into Prometheus, the database.

I explicitly mentioned keeping the initial system configuration minimal, so no OTel collector. I also differentiated above explicitly between Prometheus the SDK, Prometheus the protocol, and Prometheus the database. I never asked to have OTel from the Prometheus SDK, no idea how you came up with that based on what I explained step-wise.

[dmathieu]

I got what you mean, though maybe my previous answer wasn't clear enough.
You have metrics emitted by the Prometheus SDK, and you want them turned into OTLP within your app.

What I'm saying is that this is not something the SDK can do, and AFAIK, it's also not something prometheus can do.

So you can either switch out of prometheus to use the OTel SDK, setup a collector to retrieve the metrics you emit and turn them to OTLP. Or not use OTel.

[thediveo]

Hm, maybe we're not yet on the same page, but at least nearer now. I'm absolutely willing to use the OTel SDK (and I don't expect bridging from a Prometheus SDK Collector into the OTel SDK). Looks like what pellared mentions is the missing link, the Producer interface that a Reader (PeriodicReader) can pull metrics from.

[pellared]

I think you might be looking for https://pkg.go.dev/go.opentelemetry.io/otel/sdk/metric#WithProducer and use it with a periodic reader and OTLP exporter.

[thediveo]

Yes, that looks like the missing part to feed metrics from outside the SDK meters/instruments into the Readers.

Now, Produce produces ScopeMetrics, with a scope and the metrics. For the Scope.Name what do I use, that is, are there preferred semantic rules? A Go package import path if that does not later make accessing timeseries awkward?

Since the produced metrics need an instrument name, that would be the name to use to later "address" the set of timeseries (with distinguishing label/attribute sets?)?

[pellared]

Correct.

Correct.