Python instrumentation library for metrics

[overmeulen]

Hello,

We're currently doing a POC in our company, using the metrics part of the python instrumentation library.
I know this part is still in alpha but we faced a problem that I wanted to share.
After running our python code for a few hours, the collector-agent to which we're sending our metrics enters an infinite loop of export errors because it's trying to send more than 4Mb of data in one shot which is the default limit with gRPC.
The root cause of the problem seems to come from the python library that keeps the last value of each metric (even if we don't emit those metrics anymore) and keep sending them to the collector with the rest at each collection interval.
https://github.com/open-telemetry/opentelemetry-python/blob/v1.10a0/opentelemetry-sdk/src/opentelemetry/sdk/metrics/__init__.py#L265
This explains why after some time I reach this 4Mb limit.
Note that in my case I only have cumulative metrics and I even configured the MeterProvider as stateless so I don't really need for the library to keep the last value...
Is the implementation of the missing feature (the TODO in the link) already planned ?
Do you have any visibility on the next releases of the metrics SDK ?

Thanks,
Olivier

[lzchen]

@ocelotl

[ocelotl]

looking into this...

[ocelotl]

Hello @overmeulen

Sorry for the late reply. The development of the new metrics is still in prototype. Progress is being made in this PR. Right now this is changing very quickly as the metrics spec progresses.

Do you need a workaround for your issue?

[overmeulen]

Hello @ocelotl,

I don't have much expertise in GO but is there anyway I can help ?

Thanks,
Olivier

[ocelotl]

Sorry, I missed this discussion, I am looking into it right now. Why do you mention Go?

[ocelotl]

Hello @overmeulen

I submitted a possible workaround here.

Please contact me at Slack, I have a few questions.

[overmeulen]

Sorry I meant Python, I was thinking about the collector when I wrote this :)
Thanks for the workaround !
I'm trying to join the Slack channel but the link is inactive... https://slack.cncf.io/

[ocelotl]

Sorry I meant Python, I was thinking about the collector when I wrote this :)
Thanks for the workaround !
I'm trying to join the Slack channel but the link is inactive... https://slack.cncf.io/

Hm, pretty sure Slack is active 🙂 (I am using it right now). Maybe this can help?

Our channel is #otel-python ✌️