Is it possible to send collector logs to Splunk?

[stephenhong]

I'm not referring to sending application logs to Splunk using Splunk HEC exporter. When you have logging exporter configured on your collector, your terminal shows collector logs such as

2021-08-09T12:55:28.110Z info healthcheck/handler.go:128 Health Check state change {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "status": "ready"} 2021-08-09T12:55:28.110Z info service/service.go:267 Everything is ready. Begin running and processing data.

when the collector is ready or

2021-08-09T12:55:33.511Z info exporterhelper/queued_retry.go:276 Exporting failed. Will retry the request after interval. {"component_kind": "exporter", "component_type": "otlp", "component_name": "otlp", "error": "failed to push log data via OTLP exporter: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "4.121551979s"}

when the collector fails to export data.

I want to send these collector logs to Splunk. Is there a way to configure something in the collector to capture these collector logs and send it to Splunk?

[bogdandrutu]

/cc @atoulme

[atoulme]

Since logs of the otelcollector go to stdout by default, you could potentially collect them and send them to be ingested by a service, HEC or otherwise. Did you want to set up a pipeline specifically for that, in the same way collector metrics can be sent?

[nata7che]

sorry for "crashing" this thread @atoulme, do you have any good references for how otel-collector metrics are exposed? (for monitoring the collector) tnx and sorry again for crashing... 🙏🏻

[stephenhong]

I was able to accomplish my use case by using the following command.
./otelcontribcol_linux_amd64 --config config.yaml >> ./collectorlog.txt 2>&1 &
This creates a text file with the collector logs and I use filelogreceiver to grab the logs from the file