Outcold Solutions LLC

Monitoring Docker - Version 5

Configurations for Splunk HTTP Event Collector

Configure HTTP Event Collector secure connection

Splunk by default uses self-signed certificates. Collector provides various configuration options for you to set up how it should connect to HTTP Event Collector.

Configure trusted SSL connection to the self-signed certificate

If you are using Splunk self-signed certificate, you can copy server CA certificate from $SPLUNK_HOME/etc/auth/cacert.pem and embed it in the image.

Let's create a basic configuration for the collector, that accepts the License and specifies the connection to Splunk Enterprise HTTP Event Collector with the self-signed certificate. In this configuration, we define the path to the CA server certificate that collector should trust and identify the name of the server, specified in the certificate, which is SplunkServerDefaultCert in case of default self-signed certificate.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
caPath = /config/cacert.pem
caName = SplunkServerDefaultCert

With this configuration and cacert.pem, we can build a custom image, which includes the CA certificate and configuration.

FROM outcoldsolutions/collectorfordocker:5.23.431

COPY 002.conf /config/002.conf
COPY cacert.pem /config/cacert.pem

Using this image we can set up trusted SSL connection between collector and HTTP Event Collector. Use our installation guidance to run this image.

HTTP Event Collector incorrect index behavior

HTTP Event Collector rejects payloads with the indexes that specified Token does not allow to write. When you override indexes with the annotations, it is a very common mistake to make a misprint in the index name or forget to enable writing capabilities for the token in Splunk.

Collector provides configuration how these errors should be handled with configuration incorrectIndexBehavior.

  • RedirectToDefault - this is the default behavior, which forwards events with an incorrect index to default index of the HTTP Event Collector.
  • Drop - this configuration drops events with incorrect index.
  • Retry - this configuration keeps retrying. Some pipelines, like process stats, can be blocked for the whole host with this configuration.

You can specify behavior with the configuration.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
incorrectIndexBehavior = Drop

Build the image with the embedded configuration

FROM outcoldsolutions/collectorfordocker:5.23.431

COPY 002.conf /config/002.conf

Run it with provided installation guidance.

Using proxy for HTTP Event Collector

If you need to use a Proxy for HTTP Event Collector, you can define that with the configuration. If you are using SSL connection, you need to include the certificate used by the Proxy

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
proxyUrl = http://proxy.example:4321
caPath = /config/proxie-ca.pem

Build the image with the embedded configuration

FROM outcoldsolutions/collectorfordocker:5.23.431

COPY 002.conf /config/002.conf
COPY proxie-ca.pem /config/proxie-ca.pem

Run it with provided installation guidance.

Using multiple HTTP Event Collector endpoints for Load Balancing and Fail-over

The Collector can accept multiple HTTP Event Collector URLs for Load Balancing (in case if you are using multiple hosts with the same configuration) and for fail-over.

The collector provides you with 3 different algorithms for URL selection:

  • random - choose random URL on first selection and after each failure (connection or HTTP status code >= 500)
  • round-robin - choose URL starting from the first one and bump on each failure (connection or HTTP status code >= 500)
  • random-with-round-robin - choose random url on first selection and after that in round-robin on each failure (connection or HTTP status code >= 500)`

The default value is random-with-round-robin

[general]
acceptLicense = true

[output.splunk]
urls.0 = https://hec1.example.com:8088/services/collector/event/1.0
urls.1 = https://hec2.example.com:8088/services/collector/event/1.0
urls.2 = https://hec3.example.com:8088/services/collector/event/1.0

urlSelection = random-with-round-robin

token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

Build the image with the embedded configuration

FROM outcoldsolutions/collectorfordocker:5.23.431

COPY 002.conf /config/002.conf

Run it with provided installation guidance.

Enable indexer acknowledgement

HTTP Event Collector provides an Indexer acknowledgment, which allows knowing when payload not only accepted by HTTP Event Collector but also written to the Indexer. Enabling this feature can significantly reduce the performance of the clients, including the collector. But if you need guarantees for data delivery, you can enable it for HTTP Event Collector token and in the collector configuration.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
ackUrl = https://hec.example.com:8088/services/collector/ack
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
ackEnabled = true
ackTimeout = 3m

Build the image with the embedded configuration

FROM outcoldsolutions/collectorfordocker:5.23.431

COPY 002.conf /config/002.conf

Run it with provided installation guidance.

Client certificates for collector

If you secure your HTTP Event Collector endpoint with the requirement of client certificates, you can embed them in the image and provide configuration to use them

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
clientCertPath = /config/client-cert.pem
clientKeyPath = /config/client-cert.key

Support for multiple Splunk clusters

If you need to forward logs from the same Docker instance to multiple Splunk Clusters you can configure additional Splunk output in the configuration 

[output.splunk::prod1]
url = https://prod1.hec.example.com:8088/services/collector/event/1.0
token = AF420832-F61B-480F-86B3-CCB5D37F7D0D

All other configurations will be used from the default output output.splunk.

And override the outputs for the containers like collectord.io/output=splunk::prod1.

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.