Outcold Solutions LLC

Monitoring Docker - Version 5

Streaming Docker Objects from the API

Starting with version 5.9 you can poll objects from the Docker API server to Splunk. This is useful if you want to build your own alerts for configurations, that you expect from images and containers.

Configuration

In the configuration for the collectorfordocker you can find pre-configured polling for the containers and images, which are disabled by default

[input.docker_api::containers]

# disable docker events
disabled = true

path = /containers/json
inspectPath = /containers/{{.Id}}/json
interval = 5m
query = all=1
apiVersion =

# override type
type = docker_objects

# specify Splunk index
index =

# set output (splunk or devnull, default is [general]defaultOutput)
output =


[input.docker_api::images]

# disable docker events
disabled = true

path = /images/json
inspectPath = /images/{{.Id}}/json
interval = 5m
query = all=0
apiVersion =

# override type
type = docker_objects

# specify Splunk index
index =

# set output (splunk or devnull, default is [general]defaultOutput)
output =

To enable streaming of these objects, you can simple add environment variables to your collectorfordocker container

    ...
    --env "COLLECTOR__DOCKER_CONTAINERS=input.docker_api::containers__disabled=false" \
    --env "COLLECTOR__DOCKER_IMAGES=input.docker_api::images__disabled=false" \
    ...

Searching the data

With the configuration in the example above, the collectord will resend all the objects every 5 minutes. If you are planning to run the join command or populate the lookups, make sure that your search command covers more than interval, you can use for example 6 minutes.

The source name

By default the source will be in the format /docker/{key}, where key is the path of the stanza [input.docker_api::{key}].

Event format

Containers are forwarded in the format provided by the api call /containers/{id}/json and images in the format provided by the api call /images/{id}/json, see Engine API for details.

Searching the data

Considering that in the same time frame you can have the same object more than once (as an example if the object has been modified several times in 10 minutes), you need to group the objects by the unique identifier.

sourcetype="docker_objects" source="/docker/containers" |
stats latest(_raw) as _raw by Id |
spath output=Created path=Created |
spath output=Name path=Name |
spath output=Status path=State.Status |
table Created, Name, Status

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.