Operationalization of Promitor - Some key questions.

[sujith-r]

Hi @tomkerkhove ,

We were exploring options for making Azure PaaS metrics available for Prometheus to scrape and send to Grafana cloud and great to see Promitor in action (we were able to set up our test instance quickly and is working well). Thank you for this wonderful project !!!. We eventually will head to production monitoring, need your expert opinion on the following.

Can you please guide us on the following questions,

1. Each scraper background job equal to an API call? In other words each metric definition for a given resource corresponds to an API call?

2. If we have lot of resources within a subscription to be monitored (for eg:- 50+ Postgres, 50+ CosmosDB, 100+ storage accounts, 100+ container registries, 100+ key vaults, 50+ Azure SQL, EventGrid, DataShare etc......), in your experience what should be the ideal scraping interval to reduce api throttling (1 min, 2 mins, 5 mins ...)? Do you suggest single Promitor instance handling the metrics scrapping for this subscription(btw the numbers may increase going forward) ?

3. When we say throttling are we reffering to Azure Resource Manager throttling - https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling, in this case 12000 reads per subscription per hour? Is this beacuse of the Azure Resource Graph calls which we make as part of resource discovery?

4. We saw a lot of TaskCancelledException in the promitor-scraper-agent logs periodically. Does this mean that the scraper-agent is cancelling requests to contain throttling?

System.Threading.Tasks.TaskCanceledException: The operation was canceled.
at Microsoft.Rest.RetryDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ProviderRegistrationDelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Promitor.Integrations.AzureMonitor.RequestHandlers.AzureResourceManagerThrottlingRequestHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) in /src/Promitor.Integrations.AzureMonitor/AzureResourceManagerThrottlingRequestHandler.cs:line 77

5. Somewhere in one of the blogs read about an issue wrt open file descriptors, hope that got fixed?

6. For generic scraper (eg:- for Azure DataShare), resource discovery is not an option, right?

7. Any operational best practices to be followed in production?

Thanks,
Sujith

[tomkerkhove]

It is my pleasure and I'm happy to help you out - Thank you for using Promitor! Just out of curiosity, can you share what company you work for?

To answer your questions:

Each scraper background job equal to an API call? In other words each metric definition for a given resource corresponds to an API call?

Yes, however, they do multiple APIs calls.

If we have lot of resources within a subscription to be monitored (for eg:- 50+ Postgres, 50+ CosmosDB, 100+ storage accounts, 100+ container registries, 100+ key vaults, 50+ Azure SQL, EventGrid, DataShare etc......), in your experience what should be the ideal scraping interval to reduce api throttling (1 min, 2 mins, 5 mins ...)?

This is a very hard question to answer as it depends on your landscape and how "eager" you are. For example, you can scrape all these resources for a limited set of metrics or you can scrape all metrics but for fewer services.

I typically recommend to scrape the metrics that you need to reduce the load, which allows you to scrape more resources at a higher rate.

Promitor exposes a metric with respect to the available calls which should help you get an indication of how you are doing.

It's best to start scraping with bigger windows and scrape more frequently for the ones you need/can handle.

Do you suggest single Promitor instance handling the metrics scrapping for this subscription(btw the numbers may increase going forward) ?

It depends on how you want to approach it, but it will not impact the amount of services and scraping velocity since the rate limiting is on Azure.

We do have some work coming up to be more reliable/highly-available but that's still in the backlog.

We saw a lot of TaskCancelledException in the promitor-scraper-agent logs periodically. Does this mean that the scraper-agent is cancelling requests to contain throttling?

Odd, if it happens again, can you share the full log and check our ARM throttling metric please?

When we say throttling are we reffering to Azure Resource Manager throttling - docs.microsoft.com/en-us/azure/azure-resource-manager/management/request-limits-and-throttling, in this case 12000 reads per subscription per hour? Is this beacuse of the Azure Resource Graph calls which we make as part of resource discovery?

As far as I know, Azure Resource Graph is not included, but it does very few calls anyway so I would not be concerned here.

Somewhere in one of the blogs read about an issue wrt open file descriptors, hope that got fixed?

This is still open on #798 because it is due to a Microsoft SDK that we depend on, unfortunately.

For generic scraper (eg:- for Azure DataShare), resource discovery is not an option, right?

No unfortunately not, what scrapers are you missing?

[sujith-r]

Thanks for the response, btw, my company is Finastra.

[sujith-r]

We saw a lot of TaskCancelledException in the promitor-scraper-agent logs periodically. Does this mean that the scraper-agent is cancelling requests to contain throttling?

Odd, if it happens again, can you share the full log and check our ARM throttling metric please?

Yes, will share if we observe that again. The promitor_ratelimit_arm metric was showing good number of calls still available (like 8k, 9k+), may be throttling at the resource provider end (probably storage account and few more providers those having lesser limits). I believe we need to consider both ARM throttling and the resource provider throttling. Is this correct?
In that case as you suggested scrape metrics from such providers less frequently.

For generic scraper (eg:- for Azure DataShare), resource discovery is not an option, right?

No unfortunately not, what scrapers are you missing?

As of now Azure Data Share, Azure Data Factory.

[tomkerkhove]

Tracking new scrapers in #1678 & #359

[tomkerkhove]

Is it ok if we mark this as answered or do you have more questions?

PS: Let me know if you want to be a listed end-user.