Configurations for Splunk HTTP Event Collector

Configure HTTP Event Collector secure connection

Splunk by default uses self-signed certificates. Collector provides various configuration options for you to set up how it should connect to HTTP Event Collector.

Configure trusted SSL connection to the self-signed certificate

If you are using Splunk self-signed certificate, you can copy server CA certificate from $SPLUNK_HOME/etc/auth/cacert.pem and embed it in the image.

Let's create a basic configuration for the collector, that accepts the License and specifies the connection to Splunk Enterprise HTTP Event Collector with the self-signed certificate. In this configuration, we define the path to the CA server certificate that collector should trust and identify the name of the server, specified in the certificate, which is SplunkServerDefaultCert in case of default self-signed certificate.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
caPath = /opt/collectorforlinux/etc/cacert.pem
caName = SplunkServerDefaultCert

With this configuration and cacert.pem, we can place the cacert.pem under /opt/collectorforlinux/etc/cacert.pem and restart collectorforlinux service.

HTTP Event Collector incorrect index behavior

HTTP Event Collector rejects payloads with the indexes that specified Token does not allow to write.

Collector provides configuration how these errors should be handled with configuration incorrectIndexBehavior.

• RedirectToDefault - this is the default behavior, which forwards events with an incorrect index to default index of the HTTP Event Collector.
• Drop - this configuration drops events with incorrect index.
• Retry - this configuration keeps retrying. Some pipelines, like process stats, can be blocked for the whole host with this configuration.

You can specify behavior with the configuration.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
incorrectIndexBehavior = Drop

Using proxy for HTTP Event Collector

If you need to use a Proxy for HTTP Event Collector, you can define that with the configuration. If you are using SSL connection, you need to include the certificate used by the Proxy

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
proxyUrl = http://proxy.example:4321
caPath = /opt/collectorforlinux/etc/proxie-ca.pem

Using multiple HTTP Event Collector endpoints for Load Balancing and Fail-over

The Collector can accept multiple HTTP Event Collector URLs for Load Balancing (in case if you are using multiple hosts with the same configuration) and for fail-over.

The collector provides you with 3 different algorithms for URL selection:

• random - choose random URL on first selection and after each failure (connection or HTTP status code >= 500)
• round-robin - choose URL starting from the first one and bump on each failure (connection or HTTP status code >= 500)
• random-with-round-robin - choose random url on first selection and after that in round-robin on each failure (connection or HTTP status code >= 500)`

The default value is random-with-round-robin

[general]
acceptLicense = true

[output.splunk]
urls.0 = https://hec1.example.com:8088/services/collector/event/1.0
urls.1 = https://hec2.example.com:8088/services/collector/event/1.0
urls.2 = https://hec3.example.com:8088/services/collector/event/1.0

urlSelection = random-with-round-robin

token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0

Enable indexer acknowledgement

HTTP Event Collector provides an Indexer acknowledgment, which allows knowing when payload not only accepted by HTTP Event Collector but also written to the Indexer. Enabling this feature can significantly reduce the performance of the clients, including the collector. But if you need guarantees for data delivery, you can enable it for HTTP Event Collector token and in the collector configuration.

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
ackUrl = https://hec.example.com:8088/services/collector/ack
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
ackEnabled = true
ackTimeout = 3m

Client certificates for collector

If you secure your HTTP Event Collector endpoint with the requirement of client certificates, you can embed them in the image and provide configuration to use them

[general]
acceptLicense = true

[output.splunk]
url = https://hec.example.com:8088/services/collector/event/1.0
token = B5A79AAD-D822-46CC-80D1-819F80D7BFB0
clientCertPath = /conopt/collectorforlinux/etcfig/client-cert.pem
clientKeyPath = /opt/collectorforlinux/etc/client-cert.key

Support for multiple Splunk clusters

If you need to forward logs from the same Docker instance to multiple Splunk Clusters you can configure additional Splunk output in the configuration

[output.splunk::prod1]
url = https://prod1.hec.example.com:8088/services/collector/event/1.0
token = AF420832-F61B-480F-86B3-CCB5D37F7D0D

All other configurations will be used from the default output output.splunk.

Links

• Installation
  • Start monitoring your linux environments in under 10 minutes.
  • Automatically forward host logs and forwards system stats.
  • Test our solution with the embedded 30 days evaluation license.
• Collector Configuration
  • Collector configuration reference.
• Logs forwarding
  • Configuration for logs forwarding for custom locations.
• Configurations for Splunk HTTP Event Collector
  • Configure multiple HTTP Event Collector endpoints for Load Balancing and Fail-overs.
  • Secure HTTP Event Collector endpoint.
  • Configure the Proxy for HTTP Event Collector endpoint.
• Alerts
• Troubleshooting
• Release History
• FAQ and the common questions
• License agreement
• Pricing
• Contact