Replies: 8 comments 4 replies
-
@jacobmarble is there any way that you can recommend to debug this on the influxdb side? It appears that the receiver does not receive anything (otherwise Collector it would be visible in the output of the @perrinj3 Meta: use code blocks to make the config that you posted readable. |
Beta Was this translation helpful? Give feedback.
-
@perrinj3 what version of otel-collector-contrib are you using? I have not considered user/pass/org/bucket configs in this receiver. If that's a requirement for you, then it can probably be added without too much trouble. Please try this docker compose. It uses otel-collector-contrib version 0.35.0 configured to receive InfluxDB Line Protocol, and exports to STDOUT. If this works for you, then you should be able to troubleshoot in the Splunk exporter next. docker-compose.yml: services:
generate-metrics:
# https://github.com/open-fresh/avalanche
# Prometheus metrics at http://localhost:9090/metrics
image: quay.io/freshtracks.io/avalanche:latest
command: --port 9090 --value-interval=9 --series-interval=600 --metric-interval=3600
stop_grace_period: 1s
links:
- telegraf
telegraf:
# Configuration in telegraf.toml
image: telegraf:1.20-alpine
volumes:
- ./telegraf.toml:/etc/telegraf/telegraf.conf:ro
links:
- opentelemetry-collector
opentelemetry-collector:
# Configuration in otelcol.yml
image: otel/opentelemetry-collector-contrib:0.35.0
command: /otelcolcontrib --config /config.yaml --log-level INFO
stop_grace_period: 1s
volumes:
- ./otelcol.yml:/config.yaml:ro otelcol.yml: receivers:
influxdb:
endpoint: 0.0.0.0:8086
processors:
exporters:
file:
path: /dev/stdout
extensions:
service:
extensions: []
pipelines:
metrics:
receivers: [influxdb]
processors: []
exporters: [file] telegraf.toml: [[inputs.prometheus]]
urls = ["http://generate-metrics:9090/metrics"]
metric_version = 1
[[outputs.influxdb_v2]]
urls = ["http://opentelemetry-collector:8086"]
organization = "my-org"
bucket = "my-bucket" |
Beta Was this translation helpful? Give feedback.
-
Hi Jacob
I was running 0.34.0 but upgraded to 0.35.0 after you asked about this.
At this stage we don’t require user/pass/org/bucket configs in this receiver but it may be needed in the future. I believe that Influxdb 2.0 makes authentication mandatory so I think it would be needed then. We are running currently influxdb 1.8 to avoid the authentication issues
It would make sense to be able to filter what data is received from Influxdb but at this stage we are just evaluating OpenTelemetry’s capabilities and I just wanted to know if there were any other config options.
The evaluation setup is running K6, grafana, InfluxDb and the otel-collector-contrib on a single Windows 2012 server. I installed otel using the msi so the collector is running as a Windows Service.
K6 writes to Influxdb and the otel collector is configured to receive from Influxdb and export to Splunk which is running on a separate Windows server.
What I have found is that if the otel collector service is running and connected to InfluxDb then k6 can’t write to InfluxDb. Grafana can’t connect either.
However I’ve found that if I follow the below sequence data eventually finds its way into Splunk.
1. Stop the otel collector service
2. Run k6 test to feed data into Influxdb
3. Stop k6 test and start the otel collector service
It’s not very consistent though and it can take some time for the data to appear.
Any thoughts on what it might be/
Regards
John
From: Jacob Marble ***@***.***>
Sent: Tuesday, 21 September 2021 9:55 AM
To: open-telemetry/opentelemetry-collector-contrib ***@***.***>
Cc: John Perrin ***@***.***>; Mention ***@***.***>
Subject: Re: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
@perrinj3<https://github.com/perrinj3> what version of otel-collector-contrib are you using?
I have not considered user/pass/org/bucket configs in this receiver. If that's a requirement for you, then it can probably be added without too much trouble.
Please try this docker compose. It uses otel-collector-contrib version 0.35.0 configured to receive InfluxDB Line Protocol, and exports to STDOUT.
If this works for you, then you should be able to troubleshoot in the Splunk exporter next.
docker-compose.yml:
services:
generate-metrics:
# https://github.com/open-fresh/avalanche
# Prometheus metrics at http://localhost:9090/metrics
image: quay.io/freshtracks.io/avalanche:latest
command: --port 9090 --value-interval=9 --series-interval=600 --metric-interval=3600
stop_grace_period: 1s
links:
- telegraf
telegraf:
# Configuration in telegraf.toml
image: telegraf:1.20-alpine
volumes:
- ./telegraf.toml:/etc/telegraf/telegraf.conf:ro
links:
- opentelemetry-collector
opentelemetry-collector:
# Configuration in otelcol.yml
image: otel/opentelemetry-collector-contrib:0.35.0
command: /otelcolcontrib --config /config.yaml --log-level INFO
stop_grace_period: 1s
volumes:
- ./otelcol.yml:/config.yaml:ro
otelcol.yml:
receivers:
influxdb:
endpoint: 0.0.0.0:8086
processors:
exporters:
file:
path: /dev/stdout
extensions:
service:
extensions: []
pipelines:
metrics:
receivers: [influxdb]
processors: []
exporters: [file]
telegraf.toml:
[[inputs.prometheus]]
urls = ["http://generate-metrics:9090/metrics"]
metric_version = 1
[[outputs.influxdb_v2]]
urls = ["http://opentelemetry-collector:8086"]
organization = "my-org"
bucket = "my-bucket"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#5269 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASE3UA3F5ATYTHSEKTFJ7ZLUC7CXDANCNFSM5EMSYFYA>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
"This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."
|
Beta Was this translation helpful? Give feedback.
-
I’ve done some more investigation and found the following.
· I start Influxdb and it listens on the default port 8086
· I then try to start an opentelemetry collector with following receiver defined
receivers:
influxdb:
endPoint: "0.0.0.0:8086"
The opencollector fails with message
cannot start receivers: failed to bind to address 0.0.0.0:8086: listen tcp 0.0.0.0:8086: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
What is wrong with this please? I’ve looked at the readme and don’t understand why the OpenTel collector is trying to listen to port 8086 which Influxdb is already listening on.
How is Opebtel meant to receive from Influxdb?
Thanks
From: Jacob Marble ***@***.***>
Sent: Wednesday, 22 September 2021 1:24 AM
To: open-telemetry/opentelemetry-collector-contrib ***@***.***>
Cc: John Perrin ***@***.***>; Mention ***@***.***>
Subject: Re: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
I am not familiar with K6, and my only thought on the workaround is that perhaps one of the processes opens a port, and another tries to open the same port? I like to test these things in a Docker Compose because it makes it easier to reproduce the problem, for oneself, and for others.
Regarding InfluxDB 2.0 compatibility, I had not implemented those user/pass/org/bucket details because most people asking about the plugin were planning to write to Telegraf. Clearly, that needs to be rectified.
#5321<#5321>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#5269 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASE3UA7GDASV3ABAWQYIRFDUDCPPFANCNFSM5EMSYFYA>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
"This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."
|
Beta Was this translation helpful? Give feedback.
-
Hi Jacob
Sorry if I gave the impression I was ignoring your feedback. It’s more a lack of understanding on my part of how this all interconnects.
What we typically do is run K6 which feeds metrics directly into Influxdb. We can then display these metrics via Chronograf or Grafana
What I incorrectly assumed is that the OpenTelemetry receiver would be able to import the K6 metrics stored in Influxdb and export them to Splunk.
I now get what you’re saying. ie we configure K6 to point to port 8086 (OpenTelemetry Collector) and then export to Splunk.(as I originally posted0
So apologies for completely misunderstanding this. I’m very new to Opentelemetry and still coming up to speed on the conceptual stuff.
I get it now and really appreciate the help.
Thanks
John
From: Jacob Marble ***@***.***>
Sent: Thursday, 23 September 2021 3:04 PM
To: open-telemetry/opentelemetry-collector-contrib ***@***.***>
Cc: John Perrin ***@***.***>; Mention ***@***.***>
Subject: Re: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
John-
Yes, if you run two processes (OpenTelemetry Collector and InfluxDB) listening on the same port (8086), then only one process can open the port.
In your original post, you said:
Ultimately I want to receive metrics from InfluxDB and export them to Splunk.
And the config in your original post contains:
receivers:
influxdb:
endpoint: "localhost:8086"
When you say you want to "receive metrics from InfluxDB", what exactly do you mean? InfluxDB emits a few internal metrics via a Prometheus metrics HTTP endpoint, but it is a used as a service which receives and stores metrics.
If you mean "receive metrics via InfluxDB Line Protocol" then don't run InfluxDB. You'll be able to write InfluxDB Line Protocol to port 8086 (OpenTelemetry Collector), and watch the metrics appear in whatever exporter you configure.
Again, simplify your test environment. The docker compose config I posted is intended to do that. If you have some question about my earlier response, then please provide feedback, but it feels like it was ignored.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#5269 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASE3UA6LXROQ52VMRYB23CLUDKYNXANCNFSM5EMSYFYA>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
"This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."
|
Beta Was this translation helpful? Give feedback.
-
Hi Jacob,
Yes that’s the plan and I have it partially working. Ie I’m doing
K6 -> OpenTelemetry Collector -> Splunk
And I get valid metrics appearing in Splunk
However K6 reports an error every time it tries to write to the OpenTelemetry InfluxDB receiver
Error writing metrics to influxDB Url: http://iisau401mel5048.globaltest.anz.com:8087/write?db=k6, responseCode: 400, responseBody: failed to append to the batch
I get the same error if I point JMeter to the OpenTelemetry InfluxDB receiver so it’s not confined to K6 and I suspect the error message is coming from OpenTelemetry
Any idea what’s happening here?
Update to this issue. I upgraded Jmeter and aren't getting the error now.
With K6 the following issue has been created
#5582
Perhaps this is what is happening?
Thanks
John
From: Jacob Marble ***@***.***>
Sent: Saturday, 25 September 2021 4:52 AM
To: open-telemetry/opentelemetry-collector-contrib ***@***.***>
Cc: John Perrin ***@***.***>; Mention ***@***.***>
Subject: Re: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
John, I think I was tired when I wrote my last message. I'm happy to help, just happier in the morning, perhaps.
configure K6 to point to port 8086 (OpenTelemetry Collector) and then export to Splunk
Sounds like you propose moving from:
application -> InfluxDB -> Grafana/Chronograf
to:
application -> OpenTelemetry Collector -> Splunk
Since K6 already emits InfluxDB Line Protocol, your plan looks sound to me.
Cheers!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#5269 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASE3UA7QKIJUEHGPAWJ3HIDUDTCE7ANCNFSM5EMSYFYA>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
"This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication."
|
Beta Was this translation helpful? Give feedback.
-
From: John Perrin ***@***.***>
Sent: Thursday, 2 December 2021 12:04 PM
To: John Perrin ***@***.***>
Subject: RE: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
Hi Jacob
Just following up on issue
#5582
It looks like no one has been assigned to this issue so I’m asking your advice.
Summary is that the OpenTel Influxdb metrics receiver is responding to writes from K6 with a 202: Write Accepted.
K6 expects a 204 or a 200 and k6 support say they are adhering to the InfluxDB line protocol standard.
This causes a problem as K6 reports the writes as an error even though the metrics are flowing through correctly and it tends to break the pipeline.
Is this a configurable response within the OpenTelemetry InfluxDB receiver?
Regards,
John Perrin
From: John Perrin
Sent: Tuesday, 5 October 2021 11:26 AM
To: open-telemetry/opentelemetry-collector-contrib ***@***.***>
Subject: RE: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
Hi Jacob,
Yes that’s the plan and I have it partially working. Ie I’m doing
K6 -> OpenTelemetry Collector -> Splunk
And I get valid metrics appearing in Splunk
However K6 reports an error every time it tries to write to the OpenTelemetry InfluxDB receiver
Error writing metrics to influxDB Url: http://iisau401mel5048.globaltest.anz.com:8087/write?db=k6, responseCode: 400, responseBody: failed to append to the batch
I get the same error if I point JMeter to the OpenTelemetry InfluxDB receiver so it’s not confined to K6 and I suspect the error message is coming from OpenTelemetry
Any idea what’s happening here? Note that the /write?db=k6 parameter is used to specify which database to write to in InfluxDB. I’m not sure what the collector makes of this but it fails to respond at all, if it’s left out.
Thanks
John
From: Jacob Marble ***@***.******@***.***>>
Sent: Saturday, 25 September 2021 4:52 AM
To: open-telemetry/opentelemetry-collector-contrib ***@***.******@***.***>>
Cc: John Perrin ***@***.******@***.***>>; Mention ***@***.******@***.***>>
Subject: Re: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
John, I think I was tired when I wrote my last message. I'm happy to help, just happier in the morning, perhaps.
configure K6 to point to port 8086 (OpenTelemetry Collector) and then export to Splunk
Sounds like you propose moving from:
application -> InfluxDB -> Grafana/Chronograf
to:
application -> OpenTelemetry Collector -> Splunk
Since K6 already emits InfluxDB Line Protocol, your plan looks sound to me.
Cheers!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#5269 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASE3UA7QKIJUEHGPAWJ3HIDUDTCE7ANCNFSM5EMSYFYA>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication.
|
Beta Was this translation helpful? Give feedback.
-
Hi Jacob
Just following up on issue
#5582
It looks like no one has been assigned to this issue so I’m asking your advice.
Summary is that the OpenTel Influxdb metrics receiver is responding to writes from K6 with a 202: Write Accepted.
K6 expects a 204 or a 200 and k6 support say they are adhering to the InfluxDB line protocol standard.
This causes a problem as K6 reports the writes as an error even though the metrics are flowing through correctly and it tends to break the pipeline.
Is this a configurable response within the OpenTelemetry InfluxDB receiver?
Regards,
John Perrin
From: Jacob Marble ***@***.***>
Sent: Saturday, 25 September 2021 4:52 AM
To: open-telemetry/opentelemetry-collector-contrib ***@***.***>
Cc: John Perrin ***@***.***>; Mention ***@***.***>
Subject: Re: [open-telemetry/opentelemetry-collector-contrib] Configuring a InfluxDB Receiver (Discussion #5269)
John, I think I was tired when I wrote my last message. I'm happy to help, just happier in the morning, perhaps.
configure K6 to point to port 8086 (OpenTelemetry Collector) and then export to Splunk
Sounds like you propose moving from:
application -> InfluxDB -> Grafana/Chronograf
to:
application -> OpenTelemetry Collector -> Splunk
Since K6 already emits InfluxDB Line Protocol, your plan looks sound to me.
Cheers!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#5269 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASE3UA7QKIJUEHGPAWJ3HIDUDTCE7ANCNFSM5EMSYFYA>.
Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
This e-mail and any attachments to it (the "Communication") is, unless otherwise stated, confidential, may contain copyright material and is for the use only of the intended recipient. If you receive the Communication in error, please notify the sender immediately by return e-mail, delete the Communication and the return e-mail, and do not read, copy, retransmit or otherwise deal with it. Any views expressed in the Communication are those of the individual sender only, unless expressly stated to be those of Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any of its related entities including ANZ Bank New Zealand Limited (together "ANZ"). ANZ does not accept liability in connection with the integrity of or errors in the Communication, computer virus, data corruption, interference or delay arising from or in respect of the Communication.
|
Beta Was this translation helpful? Give feedback.
-
I'm trying to configure an influxdb receiver for the Otel Collector but not clear what else I need to specify in the config. Ultimately I want to receive metrics from InfluxDB and export them to Splunk. Initially I am exporting to a file for testing purposes but get no output with the following config. The collector starts without any errors. I've looked at the documentation and examples but it's not clear what the next steps should be.
To be clear my question is around the configuration for the influxdb receiver. I'd expect to have to include more than just the endpoint. eg what bucket to use and what username password to use but the readme only talks about the endpoint.
Thanks
Beta Was this translation helpful? Give feedback.
All reactions