ZBX_NOTSUPPORTED: Cannot open cpuacct.stat file on AWS ECS

[ThomasVdBerge]

Having an issue whit monitoring of docker containers that were started by Amazon ECS.
I am using the dockbix xxl agent

When I check the location of the file it does indeed not exist (the folder). However, the docker container is running when checked via docker ps.

Docker containers that were started on the same server, but manually (so not with ECS), are being monitored fine

This is the error that is being thrown by the agent

    15:20180102:151403.523 Docker's socket response: {"Id":"53f8b5a76026c0484948dbb720991f771106d32adca9eb75a88ce5d3afa78f0e","Created":"2 (and a lot more info....)
    15:20180102:151403.523 Item [Id] found in the received JSON object: 53f8b5a76026c0484948dbb720991f771106d32adca9eb75a88ce5d3afa78f0e
    15:20180102:151403.523 zbx_module_docker_inspect_exec OK: 53f8b5a76026c0484948dbb720991f771106d32adca9eb75a88ce5d3afa78f0e
    15:20180102:151403.523 Metric source file: /rootfs/cgroup/cpuacct/docker/53f8b5a76026c0484948dbb720991f771106d32adca9eb75a88ce5d3afa78f0e/cpuacct.stat
    15:20180102:151403.523 Cannot open metric file: '/rootfs/cgroup/cpuacct/docker/53f8b5a76026c0484948dbb720991f771106d32adca9eb75a88ce5d3afa78f0e/cpuacct.stat', container doesn't run
    15:20180102:151403.523 Sending back [0]```

[ThomasVdBerge]

Seems the problem is the fact that Amazon created their own 'folder structure'for storing the cgroup stuff.

/rootfs/cgroup/cpuacct/ecs/48cb34f5-cd39-4d59-9bb9-89f3c1611afd/53f8b5a76026c0484948dbb720991f771106d32adca9eb75a88ce5d3afa78f0e

Instead of

/rootfs/cgroup/cpuacct/docker/53f8b5a76026c0484948dbb720991f771106d32adca9eb75a88ce5d3afa78f0e/cpuacct.stat

So the folder is different (ecs <> docker) and the task definition is used in the folder path.

[jangaraj]

Valid point, Thomas. Module tries to detect proper cgroup dir location only once, during module initialization. It's not prepared for this ECS case.

[ThomasVdBerge]

I think this can be resolved by a small fix.

For ECS, docker inspect has the following value set:

"CgroupParent": "/ecs/48cb34f5-cd39-4d59-9bb9-89f3c1611afd",

for Non ECS, this value is set:

"CgroupParent": "",

So technically, taking the CgroupParent folder into account should be sufficient

[jangaraj]

Yes, 👍 It looks like ECS uses:

--cgroup-parent - Optional parent cgroup for the container

It will need additional docker inspect call.

[Introozzo]

Hi everyone!
Fixed
The patch is for cgroup-parent branch. It just adds empty parent folder to the cgroup acct files path search list. It seems its quite common to put parent cgroups under root path in case if custom cgroup is used manually (not with k8s etc).

cgroup_parent.patch.zip

[dodikvt]

Hi,
bit confused but is there a working solution for monitoring AWS ECS docker containers started with CgroupParent option ?

[joaofsoares]

Hi lads,

Did you try use ECS_ENABLE_TASK_CPU_MEM_LIMIT=false on your ecs.config file?
It changes the way your agent creates the path.