Skip to content

Commit

Permalink
[CCI-1818] Add documentation for CCM EBS shared cost allocation (#20994)
Browse files Browse the repository at this point in the history
  • Loading branch information
kaankoseler authored and MaelNamNam committed Jan 17, 2024
1 parent 69d3cf2 commit 3d15f1d
Showing 1 changed file with 41 additions and 6 deletions.
47 changes: 41 additions & 6 deletions content/en/cloud_cost_management/container_cost_allocation.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ Datadog Cloud Cost Management (CCM) automatically allocates EC2 compute cost in

{{< img src="cloud_cost/container_cost_allocation/cost_allocation_table.png" alt="Cloud cost allocation table showing requests and idle costs over the past week" style="width:100%;" >}}

Additionally, CCM allocates the cost of non-local EBS Volumes used as Persistent Volumes in Kubernetes to individual pods that claim those volumes.

## Prerequisites

1. Set up and configure [AWS Cloud Cost integration][1].
Expand All @@ -30,16 +32,27 @@ Datadog Cloud Cost Management (CCM) automatically allocates EC2 compute cost in

## Cost allocation

Cost allocation divides EC2 compute costs in the [Cost and Usage Report][4] (CUR) into individual costs for each pod or task running on the instance. These divided costs are then enriched with tags from the nodes, pods, and tasks, which allows you to break down costs by any associated dimensions.
Cost allocation divides EC2 and EBS volume costs in the [Cost and Usage Report][4] (CUR) into associated individual tasks or pods. These divided costs are then enriched with tags from nodes, pods, tasks, and volumes. This lets you break down costs by any associated dimensions.


### Kubernetes

For Kubernetes allocation, a Kubernetes node is joined with its associated EC2 instance costs. The node's cluster name and all node tags are added to the entire EC2 compute cost for the node. This allows you to associate cluster-level dimensions with the cost of the instance, without considering the pods scheduled to the node.
#### Compute

For Kubernetes compute allocation, a Kubernetes node is joined with its associated EC2 instance costs. The node's cluster name and all node tags are added to the entire EC2 compute cost for the node. This allows you to associate cluster-level dimensions with the cost of the instance, without considering the pods scheduled to the node.

Next, Datadog looks at all of the pods running on that node for the day. The cost of the node is allocated to the pod based on the resources it has used and the length of time it ran. This calculated cost is enriched with all of the pod's tags.

**Note**: Only _tags_ from pods and nodes are added to cost metrics. To include labels, enable labels as tags for [nodes][7] and [pods][8].

#### Persistent volume storage

For Kubernetes Persistent Volume storage allocation, Persistent Volumes (PV), Persistent Volume Claims (PVC), nodes, and pods are joined with their associated EBS volume costs. All associated PV, PVC, node, and pod tags are added to the EBS volume cost line items.

Next, Datadog looks at all of the pods that claimed the volume on that day. The cost of the volume is allocated to a pod based on the resources it used and the length of time it ran. These resources
include the provisioned capacity for storage, IOPS, and throughput. This allocated cost is enriched with all of the pod's tags.


### ECS on EC2

For ECS allocation, Datadog determines which tasks ran on each EC2 instance used for ECS. If you enable AWS Split Cost Allocation, the metrics allocate ECS costs by usage instead of reservation, providing more granular detail.
Expand All @@ -52,24 +65,33 @@ ECS tasks that run on Fargate are already fully allocated in the CUR. CCM enrich

### Everything else

Any cost other than EC2, computed for instances hosting Kubernetes pods or ECS tasks, is given the same value and tags as the source metric, `aws.cost.amortized`.
The information on this page pertains to EC2 compute and EBS costs for EC2 instances hosting Kubernetes pods or ECS tasks. All other costs are given the same value and tags as the source metric, `aws.cost.amortized`.


## Understanding spend

Using the `allocated_spend_type` tag, you can visualize the spend category associated with your cost at the Kubernetes node, ECS host, or cluster level. Costs are allocated into three spend types:
Using the `allocated_spend_type` tag, you can visualize the spend category associated with your cost at the Kubernetes node, ECS host, EBS volume, or cluster level. Costs are allocated into three spend types:

### Compute

- Usage: Cost of memory and cpu being used or reserved by workloads.
- Workload idle: Cost of memory and cpu that is being reserved and allocated but not used by workloads.
- Cluster idle: Cost of memory and cpu not reserved by workloads in a cluster.

### Persistent volumes

- Usage: Cost of provisioned IOPS, throughput, or storage being used by workloads. Storage cost is based on the maximum amount of volume storage used that day. IOPS and throughput costs are based on the average amount used that day.
- Workload idle: Cost of provisioned IOPS, throughput, or storage not being used by workloads. Storage cost is based on the maximum amount of storage used that day. IOPS and throughput costs are based on the average amount used that day. *Note: This tag is only available if you have enabled `Resource Collection` in your [**AWS Integration**][9]. To prevent being charged for `Cloud Security Posture Management`, ensure that during the `Resource Collection` setup, the `Cloud Security Posture Management` box is unchecked.*
- Cluster idle: Cost of provisioned IOPS, throughput, or storage for volumes not claimed by any pods that day.

## Cost metrics

When the prerequisites are met, new AWS cost metrics automatically appear.

| AWS Cost Metric | Description |
| --- | ----------- |
| `aws.cost.amortized.shared.resources.allocated` | EC2 costs allocated by the CPU & memory used by a pod or ECS task, using a 60:40 split for CPU & memory respectively. <br> *Based on `aws.cost.amortized`* |
| `aws.cost.net.amortized.shared.resources.allocated` | Net EC2 costs allocated by CPU & memory used by a pod or ECS task, using a 60:40 split for CPU & memory respectively. <br> *Based on `aws.cost.net.amortized`, if available* |
| `aws.cost.amortized.shared.resources.allocated` | EC2 costs allocated by the CPU & memory used by a pod or ECS task, using a 60:40 split for CPU & memory respectively. Also includes allocated EBS costs. <br> *Based on `aws.cost.amortized`* |
| `aws.cost.net.amortized.shared.resources.allocated` | Net EC2 costs allocated by CPU & memory used by a pod or ECS task, using a 60:40 split for CPU & memory respectively. Also includes allocated EBS costs. <br> *Based on `aws.cost.net.amortized`, if available* |
| `aws.cost.amortized.mem.allocated` | EC2 costs allocated by memory used by a pod or ECS task. <br> *Based on `aws.cost.amortized`* |
| `aws.cost.net.amortized.mem.allocated` | Net EC2 costs allocated by memory used by a pod or ECS task <br> *Based on `aws.cost.net.amortized`, if available* |
| `aws.cost.amortized.cpu.allocated` | EC2 costs allocated by CPU used by a pod or ECS task <br> *Based on `aws.cost.amortized`* |
Expand All @@ -96,6 +118,7 @@ Datadog consolidates and applies additional tags from various sources to cost me
| `allocated_resource:cpu` | Cost of CPU resources. *Only available for `.shared.resources.allocated` metrics.* |
| `allocated_resource:memory` | Cost of Memory resources. *Only available for `.shared.resources.allocated` metrics.* |
| `allocated_resource:managed_service_fee` | Cost of cloud provider managed service fees. *Only available for `.shared.resources.allocated` metrics.* |
| `allocated_resource:persistent_volume` | Cost of non-local EBS volumes used as Persistent Volumes in Kubernetes. *Only available for `.shared.resources.allocated` metrics.* |

### Kubernetes

Expand All @@ -119,6 +142,17 @@ In addition to ECS task tags, the following out-of-the-box tags are applied to c
| `is_aws_ecs_on_fargate` | All costs associated with running ECS on Fargate. |
| `is_cluster_idle` | The cost of unreserved CPU or memory on EC2 instances running ECS tasks. *Only available for `.cpu.allocated` or `.mem.allocated` metrics.*|

### Persistent volumes

In addition to Kubernetes pod and Kubernetes node tags, the following out-of-the-box tags are applied to cost metrics.

| Out-of-the-box tag | Description |
| --- |----------------------------------------------------------------------------------------------------------------------------------------------|
| `persistent_volume_reclaim_policy` | The Kubernetes Reclaim Policy on the Persistent Volume. |
| `storage_class_name` | The Kubernetes Storage Class used to instantiate the Persistent Volume. |
| `volume_mode` | The Volume Mode of the Persistent Volume. |
| `ebs_volume_type` | The type of the AWS EBS volume. Can be `gp3`, `gp2`, or others. |

## Further reading

{{< partial name="whats-next/whats-next.html" >}}
Expand All @@ -131,3 +165,4 @@ In addition to ECS task tags, the following out-of-the-box tags are applied to c
[6]: /infrastructure/containers/orchestrator_explorer?tab=datadogoperator
[7]: /containers/kubernetes/tag/?tab=containerizedagent#node-labels-as-tags
[8]: /containers/kubernetes/tag/?tab=containerizedagent#pod-labels-as-tags
[9]: https://app.datadoghq.com/integrations/amazon-web-services

0 comments on commit 3d15f1d

Please sign in to comment.