Object polling

When you want to alert on Docker configuration drift, you need the full container and image specs in Splunk, not just logs and metrics. Collectord can poll the Docker API and forward those objects on a schedule.

For example, drift can be a container running with an unexpected image, an environment variable that shouldn’t be there, or an image built without the labels your team requires.

Configuration

Collectord ships with three polls, all on by default:

• the containers;
• the images;
• the volumes Docker lists in /system/df, which the Volumes table of the Filesystem Usage dashboard reads.

Switch off the ones you don’t need.

 1[input.docker_api::containers]
 2
 3# disable docker events
 4disabled = false
 5
 6path = /containers/json
 7inspectPath = /containers/{{.Id}}/json
 8interval = 5m
 9query = all=1
10apiVersion =
11
12# override type
13type = docker_objects
14
15# specify Splunk index
16index =
17
18# set output (splunk or devnull, default is [general]defaultOutput)
19output =
20
21
22[input.docker_api::images]
23
24# disable docker events
25disabled = false
26
27path = /images/json
28inspectPath = /images/{{.Id}}/json
29interval = 5m
30query = all=0
31apiVersion =
32
33# override type
34type = docker_objects
35
36# specify Splunk index
37index =
38
39# set output (splunk or devnull, default is [general]defaultOutput)
40output =
41
42[input.docker_api::system]
43
44# disable docker events
45disabled = false
46
47path = /system/df
48listPath.Volumes = Volumes
49interval = 5m
50apiVersion =
51
52# override type
53type = docker_objects
54
55# specify Splunk index
56index =
57
58# set output (splunk or devnull, default is [general]defaultOutput)
59output =

The simplest way to switch one off is an environment variable on the collectorfordocker container:

1    ...
2    --env "COLLECTOR__DOCKER_CONTAINERS=input.docker_api::containers__disabled=true" \
3    --env "COLLECTOR__DOCKER_IMAGES=input.docker_api::images__disabled=true" \
4    --env "COLLECTOR__DOCKER_SYSTEM=input.docker_api::system__disabled=true" \
5    ...

Searching the data

With the configuration above, Collectord re-sends every object every 5 minutes. If you use join or populate lookups, make your search window longer than interval: 6 minutes is a safe default for a 5-minute poll.

The source name

Each event’s source follows the format /docker/{key}, where key is the suffix of the stanza name: for example, [input.docker_api::containers] produces /docker/containers.

A section with listPath.<name> sends one event per element of that list in the answer, and appends the name to the source. So [input.docker_api::system] sends each volume with the source /docker/system+volumes.

Event format

Collectord forwards containers as /containers/{id}/json returns them, images as /images/{id}/json returns them, and volumes as they appear in the Volumes list of /system/df. See the Docker Engine API for the full schema.

Searching the data

The same object can show up several times in one window: once per modification, plus the periodic poll. Group by the object’s unique identifier and take the latest version per group:

1sourcetype="docker_objects" source="/docker/containers" |
2stats latest(_raw) as _raw by Id |
3spath output=Created path=Created |
4spath output=Name path=Name |
5spath output=Status path=State.Status |
6table Created, Name, Status

Docker Swarm

When a Swarm service runs fewer tasks than it should, a task keeps failing, or a node drops out, the containers alone don’t tell you why. The answer is in the Swarm’s own services, tasks and nodes.

Collectord reads them from the Swarm API with three more polls. They are on by default, run every minute and send docker_objects events. The Swarm Services dashboard and the Alert Catalog’s Swarm templates read them.

Which host reads

Only a Swarm manager answers these APIs. Before each read, each section asks Docker’s /info where the host stands and reads only where its swarmRole says:

• leader (the shipped value) reads on the Swarm leader alone, so one host sends the Swarm’s records. Which host that is moves with the leader on an election.
• manager reads on every manager. They all send the same records, and the Monitoring Docker searches count each object once.

On a worker, on a host in no Swarm, and on Podman, the sections stay quiet. Under leader, when the Swarm loses its quorum and has no leader, Collectord logs it once per section rather than on every read.

So Collectord has to run on the managers: see Deploying on Docker Swarm.

Draining a node stops its tasks, a Collectord deployed as a stack’s global service included, but a drained leader stays the leader. So under leader, while the leader is drained, another manager that is ready, reachable and not drained reads in its place. An active manager goes before a paused one, then the first by node ID.

Tasks

Swarm keeps each task slot’s last tasks (five by default). That history is where a crash loop or an image that doesn’t pull shows up, since Swarm replaces each failed task with a new one. Sending the whole history on every read would resend every failure each minute. So with swarmTasks = notRunning (the shipped value), the tasks section sends:

• a task Swarm wants running but isn’t (pending on “no suitable node”, preparing, starting) on every read;
• a task that stopped (completed, shut down, failed, rejected, orphaned) once;
• a running task never, since the service records carry the running counts.

What it sends grows with the Swarm’s trouble rather than its size.

After a restart, it sends the stops since it last knew them handled, not the whole history again. The same goes for a host that starts reading, because it becomes the leader or reads in a drained leader’s place. So a Collectord that was down for a while sends the stops from that time that Swarm still keeps.

In both cases, it also reaches a few intervals further back for what the previous reading host may have missed, so the same stopped task can arrive twice: count tasks by their ID.

Leave swarmTasks empty to send every task on every read.

Swarm configuration

 1[input.docker_api::swarm_services]
 2
 3# disable swarm services
 4disabled = false
 5
 6path = /services
 7query = status=true
 8interval = 1m
 9apiVersion =
10swarmRole = leader
11
12# override type
13type = docker_objects
14
15# specify Splunk index
16index =
17
18# set output (splunk or devnull, default is [general]defaultOutput)
19output =
20
21[input.docker_api::swarm_tasks]
22
23# disable swarm tasks
24disabled = false
25
26path = /tasks
27interval = 1m
28apiVersion =
29swarmRole = leader
30swarmTasks = notRunning
31
32# override type
33type = docker_objects
34
35# specify Splunk index
36index =
37
38# set output (splunk or devnull, default is [general]defaultOutput)
39output =
40
41[input.docker_api::swarm_nodes]
42
43# disable swarm nodes
44disabled = false
45
46path = /nodes
47interval = 1m
48apiVersion =
49swarmRole = leader
50
51# override type
52type = docker_objects
53
54# specify Splunk index
55index =
56
57# set output (splunk or devnull, default is [general]defaultOutput)
58output =

To have every manager report, set swarmRole = manager on the three sections, for example with environment variables on the collectorfordocker service:

1    ...
2    --env "COLLECTOR__SWARM_SERVICES_ROLE=input.docker_api::swarm_services__swarmRole=manager" \
3    --env "COLLECTOR__SWARM_TASKS_ROLE=input.docker_api::swarm_tasks__swarmRole=manager" \
4    --env "COLLECTOR__SWARM_NODES_ROLE=input.docker_api::swarm_nodes__swarmRole=manager" \
5    ...

To switch one off, set disabled = true on it, as with the polls above.

The service and task records carry their specs as Docker returns them, environment variables included, as the container records always have. To keep a value such as a password out of Splunk, mask it with a [pipe.replace::<name>] section in your configuration. It rewrites every event Collectord sends, these records included:

1[pipe.replace::payments-db-password]
2patternRegex = (DB_PASSWORD=)([^"]+)
3replace = $1********

Searching Swarm records

Services and nodes arrive every minute, and a task arrives when it stops or while it isn’t running. Take each object’s newest record whole. Docker leaves empty fields out (a task with no error has no Err), so stats latest(<field>) can pair one record’s state with an older record’s error. The app’s macros find the records, and spath reads their fields:

1`macro_docker_swarm_services` |
2spath output=ID path=ID |
3sort 0 - _time | dedup docker_cluster_eval, ID |
4spath output=Name path=Spec.Name |
5spath output=Running path=ServiceStatus.RunningTasks |
6spath output=Desired path=ServiceStatus.DesiredTasks |
7table docker_cluster_eval, Name, Running, Desired

macro_docker_swarm_tasks and macro_docker_swarm_nodes find the task and node records the same way, each object keyed by its ID. A task record carries its service’s ID (ServiceID), not its name: read the name from the service records.

Alerts

The Alert Catalog in Monitoring Docker has eight Swarm templates that read these records:

• Swarm service below its desired tasks: a service ran fewer tasks than it wants in every minute of the last 15.
• Swarm tasks failing: a service’s tasks failed or were rejected repeatedly in 15 minutes.
• Swarm service update stuck: a rolling update, or its rollback, paused on failed tasks.
• Swarm node down: a node’s status isn’t ready.
• Swarm node drained or paused: a node that was active was set to drain or pause.
• Swarm manager unreachable: a manager isn’t reachable from the leader.
• Swarm quorum at risk: so few managers are reachable that losing one more would lose the quorum.
• Swarm stopped reporting: a Swarm that reported its nodes in the last day has sent none for 10 minutes.