Monitoring Docker, OpenShift and Kubernetes - Version 5.12 - less storage, less license, more features
October 22, 2019Reduced size for the metrics
In previous versions we were sending raw metrics from the proc file system and were doing calculation on the Splunk side to show you correct metric values, like CPU usage and Memory usage. Now, instead we are pre-calculating these metrics on the Collectord side and forwarding already user-friendly metric values. That provides a few improvements: less data to transfer over the network, less storage for the metrics, faster searches, as Splunk does not need to perform these evaluations anymore.
Global configurations with Custom Resources
Collectord watches for collectord.io configuration Custom Resources, and applies these annotations based on the selectors.
As an example you can apply the configuration to all Pods created with the image that contains nginx in the name as
apiVersion: "collectord.io/v1" kind: Configuration metadata: name: apply-to-all-nginx annotations: collectord.io/nginx--logs-replace.1-search: '^.+\"GET [^\s]+ HTTP/[^"]+" 200 .+$' collectord.io/nginx--logs-replace.1-val: '' collectord.io/nginx--logs-hashing.1-match: '(\d{1,3}\.){3}\d{1,3}' collectord.io/nginx--logs-hashing.1-function: 'fnv-1a-64' spec: kubernetes_container_image: "^nginx(:.*)?$"
Watching for namespaces (projects) and workloads
Collectord watches for the changes in the namespaces (OpenShift projects) and workloads, when it sees new or updated annotations it automatically recreates pipelines for the Pods.
[general.kubernetes] watch.namespaces = v1/namespace watch.deployments = apps/v1/deployment watch.configurations = apis/v1/collectord.io/configuration
Base macro for alerts
To be able to enable alerts only for the Production clusters you can change the macro_openshift_alerts_base (
macro_kubernetes_alerts_base or macro_docker_alerts_base) as macro_openshift_alerts_base = (openshift_cluster_eval=prod1
OR openshift_cluster_eval=prod2), that will generate alerts only for these two clusters.
Troubleshooting command
A new troubleshooting command has been introduces that can show you the list of all the annotations applied to a specific
container or pod. For example, you can see all the annotations applied to the specific pod postgres-pod running in namespace
default
kubectl exec -n collectorforkubernetes collectorforkubernetes-master-4gjmc -- /collectord describe --namespace default --pod postgres-pod --container postgres
Beta: dynamic index names based on meta fields
You can apply dynamic index names in the configurations to forward logs or stats to a specific index, based on the meta fields. For example, you can define an index as
[input.files] index = oc_{{openshift_namespace}}
In that case all the logs from the namespace foo will be forwarded to the index oc_foo and all the logs from the
namespace bar will be forwarded to index oc_bar.
This feature is in the beta, we will publish additional documentation how to set it up.
Beta: diagnostic checks
With this release we are bringing a new capability in Collectord that will allow you to diagnose and fire alerts directly
from Collectord, for now we implemented only one check for the Node Entropy
that can verify that the entropy_avail is above the threshold 800.
Next we are planning to add more alerts directly to Collectord, so this data is not going to be forwarded to Splunk, but Collectord will generate alerts directly from the nodes. These alerts will be written as part of the standard output of the Collectord containers. For example
ALARM-OFF "node-entropy" ALARM-ON "node-entropy" entropy value 720 is below the threshold 800
This feature is in the beta, by default it is disabled, and could be enabled in the YAML configuration for collectorforopenshift.yaml or collectorforkubernetes.yaml (under
diagnostics::node-entropy)
Links
You can find more information about other minor updates by following links below.
Release notes
- Monitoring OpenShift - Release notes
- Monitoring Kubernetes - Release notes
- Monitoring Docker - Release notes
