[metrics]: Prometheus cloudwatch exporter - Not picking more than one or some dimension for MQ

[tavincent]

Hello Team,

We are trying to use cloudwatch as a datasource for prometheus as we have a managed RabbitMQ in AWS which we need to monitor using alertmanager/prometheus.

We have defined the following in our K8s config to scrape metrics and it is getting metrics in Broker level.

      - aws_namespace: AWS/AmazonMQ
        aws_metric_name: MessageCount 
        aws_dimensions: [Broker]
        aws_statistics: [Maximum]

However as per aws documentation we can use multiple dimensions for this metrics "MessageCount", which will help me list the queue level message count as well rather than a Broker level aggregate.

AWS: " aws cloudwatch list-metrics --namespace AWS/RabbitMQ" Region : "us-east-1"

        {
            "Namespace": "AWS/AmazonMQ",
            "MetricName": "MessageCount",
            "Dimensions": [
                {
                    "Name": "Broker",
                    "Value": "mybroker"
                },
                {
                    "Name": "VirtualHost",
                    "Value": "/"
                },
                {
                    "Name": "Queue",
                    "Value": "myqueue"
                }
            ]
        },

• AWS service: AWS MQ - RabbitMQ
• CloudWatch namespace:
• Link to metrics documentation for this service:
• AWS region of the exporter: us-east-1
• AWS region of the service:us-east-1

Whenever i used more than one dimension in my config i get the following error, even if the dimension is allowed by AWS as per ( aws cloudwatch list-metrics --namespace AWS/RabbitMQ).

from my yaml:

      - aws_namespace: AWS/AmazonMQ
        aws_metric_name: MessageCount 
        aws_dimensions: [Broker, Queue]
        aws_statistics: [Maximum]

---

WARNING: (listDimensions) ignoring metric AWS/AmazonMQ:MessageCount due to dimensions mismatch
Feb 23, 2023 7:58:30 AM io.prometheus.cloudwatch.DefaultDimensionSource listDimensions

---

I need to understand why i am not able to use multiple dimensions in my yaml. FYI: I am able to list all these queues and everything is working as expected the aws console - cloudwatch logs.

Appreciate your help.

--
Taniya

[matthiasr]

Interesting, thank you for the complete report!

The AWS docs are listing two MessageCount metrics – once as a RabbitMQ broker metric with only the Broker dimension, once as a RabbitMQ queue metric with all three dimensions. I think this is confusing the exporter.

@or-shachar you may be interested in this one. Is there a workaround that would allow @tavincent to bypass the check for now?

[matthiasr]

I think the exporter needs to handle this situation better, since we can't expect Amazon to stop having this collision 😅

[tavincent]

Thanks Matthias, I'd also like to mention that this dimension's mismatch is not limited to a single RabbitMQ metric, so when you review please note that we have this collision for all RabbitMQ metrics (currently, only Broker dimension is accepted for all metrics).

[tavincent]

Hi Matthias,

Any ETA on when this bug would be fixed on exporter end?

[matthiasr]

This exporter is a volunteer-driven project, there are no commitments or timelines. The quickest way to get issues fixed is to make a PR!

[thameezb]

Hitting this issue with HTTPCode_Target_4XX_Count metric. It returns a different set of dimensions for different LBs. That is any of the following: Loadbalancer, TargetGroups, AvailabilityZone

Therefore the collector cannot be used for to track this metric

[ppcololo]

I don't know - maybe I'm late but I'm using this code to fetch MessageCount

  - aws_namespace: AWS/AmazonMQ
    aws_metric_name: MessageCount
    aws_dimensions: [Broker, VirtualHost, Queue]
    aws_statistics: [Maximum]