Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(DOCS-6845) Add enrichment options and screenshots #20997

Merged
merged 3 commits into from
Dec 20, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 28 additions & 8 deletions content/en/network_monitoring/devices/netflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,34 +53,54 @@

## Aggregation

The Datadog Agent automatically aggregates the received NetFlow data in order to limit the number of records sent to the platform while maintaining most of the information. By default there is a 5 minute aggregation interval, during which flow recordings which share the same identifying information (source and destination address and port, protocol, and so forth) will be aggregated together. Additionally, the Datadog Agent can detect ephemeral ports and remove them. As a result, you may see Flows with `port:*`.
The Datadog Agent automatically aggregates the received NetFlow data in order to limit the number of records sent to the platform while maintaining most of the information. By default there is a 5 minute aggregation interval, during which flow recordings which share the same identifying information (source and destination address and port, protocol, and so forth) are aggregated together. Additionally, the Datadog Agent can detect ephemeral ports and remove them. As a result, you may see Flows with `port:*`.

Check notice on line 56 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L56

[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.
Raw output
{"message": "[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 56, "column": 1}}}, "severity": "INFO"}

Check warning on line 56 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L56

[Datadog.words] Use 'to' instead of 'in order to'.
Raw output
{"message": "[Datadog.words] Use 'to' instead of 'in order to'.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 56, "column": 70}}}, "severity": "WARNING"}

Check notice on line 56 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L56

[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.
Raw output
{"message": "[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 56, "column": 174}}}, "severity": "INFO"}
buraizu marked this conversation as resolved.
Show resolved Hide resolved

## Enrichment

Your NetFlow data is processed by the Datadog backend and are enriched with the available metadata from your devices and interfaces. Enrichment is based on the NetFlow exporter IP and the interface indexes. To disambiguate possible collisions between reused private IPs, you can configure a different `namespace` for each Agent configuration file (with the setting `network_devices.namespace`)
Your NetFlow data is processed by the Datadog backend and enriched with the available metadata from your devices and interfaces. Enrichment is based on the NetFlow exporter IP and the interface indexes. To disambiguate possible collisions between reused private IPs, you can configure a different `namespace` for each Agent configuration file (with the setting `network_devices.namespace`).

If the NetFlow exporter IP is one of the device IPs, but not the one configured on the SNMP integration, Datadog will attempt to locate the device that the exporter IP belongs to, and will enrich your NetFlow data with it is as long as the match is unique.
If the NetFlow exporter IP is one of the device IPs, but not the one configured on the SNMP integration, Datadog attempts to locate the device that the exporter IP belongs to, and enriches your NetFlow data with it is as long as the match is unique.

Check notice on line 62 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L62

[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.
Raw output
{"message": "[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 62, "column": 1}}}, "severity": "INFO"}

### Cloud provider IP enrichment

Check warning on line 64 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L64

[Datadog.headings] 'Cloud provider IP enrichment' should use sentence-style capitalization.
Raw output
{"message": "[Datadog.headings] 'Cloud provider IP enrichment' should use sentence-style capitalization.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 64, "column": 5}}}, "severity": "WARNING"}

Datadog enriches IPs with public cloud provider service and region for IPv4 addresses, so you can filter for flow records from a specific service and region.

Check notice on line 66 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L66

[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.
Raw output
{"message": "[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 66, "column": 1}}}, "severity": "INFO"}

{{< img src="network_device_monitoring/netflow/netflow_cloud_provider_ip_enrichment.png" alt="Netflow IPs enriched with cloud provider name, region, and service" width="80%" >}}

### Port enrichment

Datadog enriches ports in NetFlow with IANA (Internet Assigned Numbers Authority) data to resolve well known port mappings (such as Postgres on 5432 and HTTPS on 443). This can be seen when searching for source or destination application names on NetFlow.

Check notice on line 72 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L72

[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.
Raw output
{"message": "[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 72, "column": 1}}}, "severity": "INFO"}

{{< img src="network_device_monitoring/netflow/netflow_iana_port_mappings.png" alt="The NetFlow page filtered by @destination.application_name and displaying names for ports such as HTTPS" width="80%" >}}

#### Custom port enrichment

You can also add your own custom enrichments to map specific ports to an application name (for example, if a custom service runs on a specific port). This makes it easier for network engineers and their teams to interpret and query NetFlow data with human-readable names.

Check notice on line 78 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L78

[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.
Raw output
{"message": "[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 78, "column": 1}}}, "severity": "INFO"}

From the Configuration tab in NetFlow, click **Add Enrichment** to upload the CSV file containing your custom enrichments.
buraizu marked this conversation as resolved.
Show resolved Hide resolved

{{< img src="network_device_monitoring/netflow/new_enrichment.png" alt="The New Enrichment Mapping modal in the Netflow configuration tab" width="80%" >}}

This means customers can see the service and region for IPv4 addresses that map to the public IP ranges from cloud providers and filter for flow records from a specific service and/or region.

Check notice on line 84 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L84

[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.
Raw output
{"message": "[Datadog.sentencelength] Try to keep your sentence length to 25 words or fewer.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 84, "column": 1}}}, "severity": "INFO"}

Check warning on line 84 in content/en/network_monitoring/devices/netflow.md

View workflow job for this annotation

GitHub Actions / vale

[vale] content/en/network_monitoring/devices/netflow.md#L84

[Datadog.words] Use 'and', 'or', or 'either or' instead of 'and/or'.
Raw output
{"message": "[Datadog.words] Use 'and', 'or', or 'either or' instead of 'and/or'.", "location": {"path": "content/en/network_monitoring/devices/netflow.md", "range": {"start": {"line": 84, "column": 178}}}, "severity": "WARNING"}

## Visualization

You can find the NetFlow page on the [Network Devices page][5].
{{< img src="network_device_monitoring/netflow/netflow_page.png" alt="NetFlow Page" >}}
{{< img src="network_device_monitoring/netflow/netflow_page.png" alt="NetFlow Page" width="80%" >}}

This data is also available in Dashboards, Notebooks, and more for more precise queries and for correlating with other sources of data.
{{< img src="network_device_monitoring/netflow/notebook.png" alt="Notebook" >}}
{{< img src="network_device_monitoring/netflow/notebook.png" alt="Notebook" width="80%" >}}

## Retention

NetFlow data is retained for 30 days by default.


## Further Reading

{{< partial name="whats-next/whats-next.html" >}}


[1]: https://app.datadoghq.com/account/settings/agent/latest
[2]: /network_monitoring/devices/snmp_metrics/
[3]: /agent/configuration/agent-configuration-files/?tab=agentv6v7#agent-main-configuration-file
[4]: /agent/configuration/agent-commands/?tab=agentv6v7#start-stop-and-restart-the-agent
[5]: https://app.datadoghq.com/infrastructure/devices?facets=&viewTab=netflow
[5]: https://app.datadoghq.com/infrastructure/devices?facets=&viewTab=netflow
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading