Skip to content

Commit

Permalink
[DOCS-6598] Add Manage Logs and Metrics with Terraform (#21081)
Browse files Browse the repository at this point in the history
* add doc

* Apply suggestions from code review

* Update content/en/logs/guide/manage_logs_and_metrics_with_terraform.md
  • Loading branch information
maycmlee authored Dec 18, 2023
1 parent c5064ce commit 26118eb
Show file tree
Hide file tree
Showing 3 changed files with 63 additions and 0 deletions.
1 change: 1 addition & 0 deletions content/en/logs/guide/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ cascade:
{{< nextlink href="logs/guide/getting-started-lwl" >}}Logging Without Limits™ Guide{{< /nextlink >}}
{{< nextlink href="logs/guide/correlate-logs-with-metrics" >}}Correlate Logs with Metrics{{< /nextlink >}}
{{< nextlink href="logs/guide/best-practices-for-log-management" >}}Best Practices for Log Management{{< /nextlink >}}
{{< nextlink href="logs/guide/manage_logs_and_metrics_with_terraform" >}}Manage Logs and Metrics with Terraform{{< /nextlink >}}
{{< /whatsnext >}}

<br>
Expand Down
61 changes: 61 additions & 0 deletions content/en/logs/guide/manage_logs_and_metrics_with_terraform.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: Manage Logs and Metrics with Terraform
kind: guide
disable_toc: false
---

## Overview
You can use [Terraform][1] to interact with the Datadog API and manage your logs and metrics. This guide provides example use cases and includes links to commonly used Datadog resources and data sources in the Terraform registry.

You can also [import][2] your existing resources into your Terraform configuration, and reference existing resources as Terraform [data sources][3].

## Set up the Datadog Terraform Provider

If you haven't already, configure the [Datadog Terraform provider][4] to interact with Datadog APIs through a Terraform configuration.

## Log configuration

### Set up multiple indexes

Set up [multiple indexes][5] if you want to segment your logs for different retention periods or daily quotas, usage monitoring, and billing. For example, if you have logs that only need to be retained for 7 days, while other logs need to be retained for 30 days, use multiple indexes to separate out the logs by the two retention periods. See [Inclusion filters][6] and [Exclusion filters][7] documentation for information on defining queries for them. Since ingested logs go into the first index whose filter they match, [order your indexes][8] according to your use case.

### Set up a custom pipeline

Log pipelines are a chain of sequential processors that extract meaningful information or attributes from the content to reuse as facets. Each log that goes through the pipelines is matched against each pipeline filter. If it matches the filter, then all the processors are applied to the log before moving to the next pipeline. Set up a [custom pipeline][9] to parse and enrich your logs. See the [Processors documentation][10] for details on the available processors. You can also [reorder your pipelines][11] to make sure logs are getting processed in the correct order.

Integration pipelines are automatically installed when you send logs from certain sources (for example, the NGINX integration). You can reorder these pipelines with the [logs integration pipelines resource][12].

### Set up multiple archives for long-term storage

Set up [Log Archives][13] if you want to store your logs for longer periods of time. Log Archives sends your logs to a storage-optimized system, such as Amazon S3, Azure Storage, or Google Cloud Storage. You can also [reorder your archives][14] as needed.

### Generate metrics from ingested logs

[Generate log-based metrics][15] to summarize log data from your ingested logs. For example, you can generate a count metric of logs that match a query or match a distribution metric of a numeric value contained in the logs, such as request duration. See [Generate Metrics from Ingested Logs][16] for more information.

## Metric configuration

A metric's metadata includes the metric name, description, and unit. Use the [metric metadata resource][17] to modify the information.

Tags add dimensions to your metrics so that they can be filtered, aggregated, and compared in visualizations. Use the [metric tag configuration resource][18] to modify your metrics tags in Terraform. See [Getting Started with Tags][19] for more information on using tags.


[1]: https://www.terraform.io/
[2]: https://developer.hashicorp.com/terraform/cli/import
[3]: https://developer.hashicorp.com/terraform/language/data-sources
[4]: /integrations/terraform/
[5]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_index
[6]: /logs/log_configuration/indexes/#indexes-filters
[7]: /logs/log_configuration/indexes/#exclusion-filters
[8]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_index_order
[9]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_custom_pipeline
[10]: /logs/log_configuration/processors/?tab=ui
[11]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_pipeline_order
[12]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_integration_pipeline
[13]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_archive
[14]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_archive_order
[15]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/logs_metric
[16]: https://docs.datadoghq.com/logs/log_configuration/logs_to_metrics/
[17]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/metric_metadata
[18]: https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/metric_tag_configuration
[19]: /getting_started/tagging/
1 change: 1 addition & 0 deletions content/en/metrics/guide/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ cascade:
{{< nextlink href="metrics/guide/interpolation-the-fill-modifier-explained/" >}}Interpolation and the fill modifier{{< /nextlink >}}
{{< nextlink href="metrics/guide/why-does-zooming-out-a-timeframe-also-smooth-out-my-graphs" >}}Why does zooming out a timeframe also smooth out my graphs?{{< /nextlink >}}
{{< nextlink href="metrics/guide/what-is-the-granularity-of-my-graphs-am-i-seeing-raw-data-or-aggregates-on-my-graph" >}}What is the granularity of my graphs? Am I seeing raw data or aggregates on my graph?{{< /nextlink >}}
{{< nextlink href="logs/guide/manage_logs_and_metrics_with_terraform" >}}Manage Logs and Metrics with Terraform{{< /nextlink >}}
{{< /whatsnext >}}

{{< whatsnext desc="Custom metrics guides:" >}}
Expand Down

0 comments on commit 26118eb

Please sign in to comment.