Outcold Solutions LLC

Monitoring OpenShift - Version 5

Configuration

To find server version of your OpenShift cluster use

$ oc version

If you are using a version, which is not on the list, please email us contact@outcoldsolutions.com

Using images on docker.io (hub.docker.com)

These images are built on top of Scratch images.

Current most recent version of OpenShift

Configuration files for previous releases of Collectord deployment files you can find at github.com/outcoldsolutions/collectord-configurations

Using certified images on registry.connect.redhat.com

These images are built on top of RHEL images, see outcoldsolutions/collectorforopenshift. To pull images from this registry, you need to authenticate, see instructions below.

Current most recent version of OpenShift

Configuration files for previous releases of Collectord deployment files you can find at github.com/outcoldsolutions/collectord-configurations

registry.connect.redhat.com authentication

registry.connect.redhat.com is not the same as registry.access.redhat.com, second is used for Red Hat images, first is used for certified images from partners. Second works with OpenShift cluster out of the box, first requires authentication.

You need to specify secret to authenticate with registry.connect.redhat.com. Please follow the link to learn how to use other secured registries. Allowing Pods to Reference Images from Other Secured Registries

This is an example how you can authenticate with registry.connect.redhat.com.

After applying the configuration to your OpenShift cluster, you need to create a secret for pulling images from Red Hat registry. Make sure you are in the same project/namespace as where collector is created (collectorforopenshift is a default project/namespace).

$ oc project collectorforopenshift

If you are on Linux (for macOS see below), you can log in to the registry using docker and use authentication file to create a secret in OpenShift cluster.

$ docker login registry.connect.redhat.com
Username: [redhat-username]
Password: [redhat-user-password]
Login Succeeded

Make sure to use username and not email, when you login to this registry. They both allows you to login. But if you logged in with email, you will not be able to download the image.

After that you can create a secret for pulling images using just created authentication under $HOME/.docker/config.json

$ oc --namespace collectorforopenshift secrets new rhcc .dockerconfigjson=$HOME/.docker/config.json

On macOS Docker does not store authentication data in config.json (stores in keychain instead). You cannot use it to create a secret. Instead you can create a secret from the command line with oc secrets --namespace collectorforopenshift new-dockercfg rhcc --docker-server=registry.connect.redhat.com --docker-username=<user_name> --docker-password=<password> --docker-email=<email>. Just make sure this command is not going to be saved in the bash history, as it is going to have a password in the command line. See Execute command without keeping it in history. You can execute export HISTFILE=/dev/null in this terminal session, which will stop recording any commands in history.

Link created secret rhcc to the service account we use for collector collectorforopenshift

$ oc --namespace collectorforopenshift secrets link collectorforopenshift rhcc --for=pull

If some pods have been created before you linked the secret, you will need to recreate them. You can delete all the pods under collectorforopenshift namespace, and scheduler will recreate pods with right secret for pulling images.

oc delete --namespace collectorforopenshift pods --all

Created OpenShift Objects

Configuration file collectorforopenshift.yaml creates several OpenShift Objects.

  • Project collectorforopenshift.
  • ClusterRole collectorforopenshift with limited capabilities to get, list and watch most of the various deployment objects. Collector uses this information to enrich logs and stats with openshift specific metadata.
  • ServiceAccount collectorforopenshift is used to connect to OpenShift API.
  • ClusterRoleBinding collectorforopenshift to bind a service account to a cluster role.
  • ConfigMap collectorforopenshift delivers configuration file for collector.
  • DaemonSet collectorforopenshift allows to deploy collector on none-master nodes.
  • DaemonSet collectorforopenshift-master allows to deploy collector on master nodes.
  • Deployment collectorforopenshift-addon is a single collector that needs to forward data from the whole cluster once.

Please read commentaries in collectorforopenshift.yaml file to get more deep details on all configurations and source of the logs and metrics.

Collector configuration

ConfigMap collectorforopenshift delivers configuration files for collector. These are ini files, where you can find all the default values.

Values can be overridden using environment values with the format as specified below

COLLECTOR__{ANY_NAME}={section}__{key}={value}

Configurations with environment variables are the simplest way to explore and debug quickly, but we recommend to write your configuration file based on the default provided with collectorforopenshift.yaml.

Using secrets to manage configurations

You can use OpenShift secrets and map them as an environment variable to override configurations for the collector.

As an example, we will show how you can configure HTTP Event Collector and License with secrets.

At first, make sure that collectorforopenshift namespace already exists. If it does not exist, you need to create it.

oc create namespace collectorforopenshift

oc create secret generic collectorforopenshift \
    --namespace collectorforopenshift \
    --from-literal=splunk-token="output.splunk__token=B5A79AAD-D822-46CC-80D1-819F80D7BFB0" \
    --from-literal=license="general__license="

In our YAML manifest find a configuration of environment variables for each Deployment type (2 DaemonSets and 1 Deployment) and add the following information to the environment variables

env:
- name: COLLECTOR__SPLUNK_TOKEN
  valueFrom:
    secretKeyRef:
      name: collectorforopenshift
      key: splunk-token
- name: COLLECTOR__LICENSE
  valueFrom:
    secretKeyRef:
      name: collectorforopenshift
      key: license

Apply the manifest by following installation instructions.

Attaching EC2 Metadata

You can include EC2 metadata with the forwarded data (logs and metrics) by specifying desired field name and path from Instance Metadata and User Data.

# Include EC2 Metadata (see list of possible fields https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html)
# Should be in format ec2Metadata.{desired_field_name} = {url path to read the value}
# ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
# ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type

As an example, you can modify YAML file and include two fields ec2_instance_id and ec2_instance_type

[general]
...
ec2Metadata.ec2_instance_id = /latest/meta-data/instance-id
ec2Metadata.ec2_instance_type = /latest/meta-data/instance-type
...

Placeholders in indexes and sources

You can apply dynamic index names in the configurations to forward logs or stats to a specific index, based on the meta fields. For example, you can define an index as

[input.files]

index = oc_{{openshift_namespace}}

Similarly you can change the source of all the forwarded logs like

[input.files]

source = /{{openshift_namespace}}/{{::coalesce(openshift_daemonset_name, openshift_deployment_name, openshift_statefulset_name, openshift_cronjob_name, openshift_job_name, openshift_replicaset_name, openshift_pod_name)}}/{{openshift_pod_name}}/{{openshift_container_name}}

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.