Outcold Solutions LLC

Monitoring Docker, OpenShift and Kubernetes - Version 5.20

April 10, 2023

(OpenShift) Cluster Resource Quotas Dashboard

The Cluster Resource Quotas dashboard provides a high-level overview of the resource quotas for all namespaces in your cluster. The dashboard shows the current usage of CPU, memory, and storage for each namespace, as well as the total quota for each resource. The dashboard also shows the percentage of the quota that is currently in use.

When selecting a quota, the dashboard shows drill down by namespace.

Cluster Resource Quotas in Monitoring OpenShift

(Kubernetes and OpenShift) Pod conditions

In both applications Monitoring Kubernetes and OpenShift, a new table is added to the Pod dashboard to show the pod conditions.

Pod Conditions

(Kubernetes and OpenShift) Event duplication for multiple Splunk HTTP Event Collector endpoints

Collectord has supported multiple Splunk HTTP Event Collector endpoints since version 5.9. But with the possibility to only send event to a single endpoint. This was a limitation for users who wanted to send events to multiple Splunk HEC endpoints. In this version, we have added support for sending events to multiple Splunk HEC endpoints at the same time. This version supports only forwarding container logs and volume logs to multiple Splunk HEC endpoints at the same time.

With annotation collectord.io/logs-output you can configure multiple Splunk HEC endpoints, for example splunk::apps and splunk::security using the comma-separated list: collectord.io/logs-output=splunk::apps,splunk::security. Assuming you have them defined in the ConfigMap like [output.splunk::apps] and [output.splunk::security].

Additionally, you can configure indexes for the endpoints in square brackets, for example collectord.io/logs-output=splunk::apps[kubernetes_logs],splunk::security[kubernetes_security].

In that case, each event will be sent to both Splunk HEC endpoints.

(Kubernetes and OpenShift) Improvements for forwarding logs from Persistent Volumes

Collectord supports forwarding logs from Persistent Volumes since version 5.11. The main purpose of supporting forwarding logs from persistent volumes is to support the use case when you want to forward logs from the applications running as Stateful Sets, like databases.

The weak point of the current implementation is that the Collectord stores acknowledgment information for each file on the host where the Persistent Volume is mounted. This means that if the Persistent Volume is remounted on a different host, the Collectord will start sending logs from the beginning, as it will not be able to find the acknowledgment information.

In this version, we have added support for storing acknowledgment information on the volume itself. You can add an annotation collectord.io/volume.{N}-logs-onvolumedatabase=true to the Pod to enable this feature. In that case Collectord creates a database file in volume root .collectord.db that stores information about all the files already sent to Splunk. So when the volume is getting attached to a different host, another Collectord instance will be able to find the database file and start forwarding logs from the last acknowledged position.

Important details about this feature, you need to mount the /rootfs directory in the Collectord container with write access. By default, the /rootfs directory is mounted as read-only.

(Kubernetes and OpenShift) Support for placeholders in the glob configuration for the Persistent Volumes

In the case, if you are mounting the same volume to multiple Pods, and you want to differentiate the logs, you can now specify the placeholders in the glob configuration. For example, if you have a volume mounted to the Pod with the name my-pod and to the Pod with the name my-pod-2 you can specify the glob configuration like this {{kubernetes_pod_name}}.log, so the Collectord will be able to identify that files my-pod.log and my-pod-2.log are coming from different Pods.

(Kubernetes, OpenShift and Docker) multi-architecture images

Collectord was available with multi-architecture images for amd64 and arm64 architectures since a version 5.16.361, but the arm64 image was always available as a separate image with prefix -arm64. Since version 5.20.400, we are providing multi-platform images for amd64 and arm64 architectures.

This simplifies the installation process for users with multi-architecture clusters.

(Kubernetes and OpenShift) Support for sending logs to ElasticSearch and OpenSearch

Large teams might have different requirements for the log management system. Some teams might prefer to use ElasticSearch or OpenSearch for log management. In this version, we have added support for sending logs to ElasticSearch and OpenSearch.

You can install the Collectord with ElasticSearch or OpenSearch support and run it in the same cluster as the Collectord for Splunk. In that case, you can configure the Collectord to send logs to both Splunk and ElasticSearch or OpenSearch.

Please read the blog post for more details.

You can find more information about other minor updates by following the links below.

Release notes

Upgrade instructions

Installation instructions

docker, kubernetes, openshift, splunk, elasticsearch

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.