Outcold Solutions LLC

Monitoring Docker, OpenShift and Kubernetes - Version 5.18

April 19, 2022

The main focus of this release was support for clusters using cgroupv2 and support for a different memory usage calculation.

Memory usage

There is a lot of ways to calculate memory usage. Since the release of the first version of Collectord we reported the true RSS memory, which means the actual memory that Container or the Host use.

From the cgroupv1 definition:

Only anonymous and swap cache memory is listed as part of 'rss' stat. This should not be confused with the true 'resident set size' or the amount of physical memory used by the cgroup.

'rss + mapped_file" will give you resident set size of cgroup.

But that might not be the same comparing to what you might see in other tools, including OpenShift monitoring. They show you the memory that cannot be freed. The definition from the Metrics Server:

In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure. However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate. It includes all anonymous (non-file-backed) memory since Kubernetes does not support swap. The metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim such pages.

With the release 5.18 we have changed that too, to show the memory usage in the same way as Metrics Server collects it.

Cgroupv2

You will not find a lot of clusters deployed with cgroupv2 controller. But Docker Desktop recently have changed default cgroup controller from v1 to v2. To support enthusiasts we added support for cgroupv2 controller as well. Collectord automatically detects if cgroupv2 is used instead of cgroupv1 and collects the metrics in the same manner as with cgroupv1.

Allow to specify the message group name for the fields extraction

Collectord allows you to extract fields from Container logs. And to specify what should be a message field you just need to keep first unnamed group, which Collectord will automatically use as message for the event.

We have found that to be confusing for a lot of users, and it made some the fields extraction being too complicated. So we have added a simple annotation collectord.io/logs-extractionMessageField that specifies which field should be used as message.

As an example

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  annotations:
    collectord.io/logs-extraction: '^(?P<ip_address>[^\s]+) .* \[(?P<timestamp>[^\]]+)\] (?P<message>.+)$'
    collectord.io/logs-timestampfield: timestamp
    collectord.io/logs-timestampformat: '02/Jan/2006:15:04:05 -0700'
    collectord.io/logs-extractionMessageField: message
spec:
  containers:
  - name: nginx
    image: nginx

arm64 support for OpenShift

OpenShift added support for arm64 nodes, and we have implemented images with arm64 architecture as well.

You can find more information about other minor updates by following links below.

Release notes

Upgrade instructions

Installation instructions

docker, kubernetes, openshift, splunk

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.