tl;dr;

To install Monitoring Kubernetes solution on Tectonic

1. Follow instructions on Monitoring Kubernetes to install collector and application.
2. Install DaemonSet with rsyslog, which forwards journald logs to /var/log/syslog.

kubectl apply -f https://raw.githubusercontent.com/outcoldsolutions/docker-journald-to-syslog/master/journald-to-syslog.yaml

Building Journald-To-Syslog image

Tectonic is the secure, automated, and hybrid enterprise Kubernetes platform. Built by CoreOS.

Our solutions for Monitoring Kubernetes works for most of the distributions of Kubernetes. And it works for Tectonic as well. It just requires small help to get the host logs.

Tectonic is using CoreOS as a host Linux for containers. CoreOS is very lightweight Linux, that does not provide package manager and require you to install additional software with containers.

CoreOS by default writes all host logs to the journald, which stores them in binary format. In most distributions, like RHEL, we recommend installing rsyslog which automatically configures streaming of all logs from journald to /var/log/. For CoreOS we should do the same, the only difference we will need to install it inside the container.

For the image outcoldsolutions/journald-to-syslog I choose debian:stretch as a base image for a simple reason. It has the right version of journalctl, built with the right set of libraries. If you will ssh to one of your CoreOS boxes, you will find out that it journald was built with +LZ4 and it is probably using it as a default.

$ journalctl --version
systemd 233
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN default-hierarchy=legacy

For example ubuntu does not have +LZ4

$ docker run --rm ubuntu journalctl --version
systemd 229
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN

But debian has it

$ docker run  --rm debian bash -c 'apt-get update && apt-get install -y systemd && journalctl --version'
systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

If you will decide to build your image to forward logs from journald to syslog, make sure to check the compatibility. You can test it by running journalctl -f over the journal files from your host Linux

docker run --rm -it \
    -v /var/log/journal/:/var/log/journal/:ro \
    -v /etc/machine-id:/etc/machine-id:ro \
    debian \
    bash -c 'apt-get update && apt-get install -y systemd && journalctl -f'

Inside Journald-To-Syslog image

rsyslog

I used rsyslog with imjournal to build this image. As an alternative, you can use syslog-ng with systemd-journal source.

The configuration I use for rsyslog

1
2
3
4

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal" StateFile="/rootfs/var/log/syslog.statefile")
$outchannel log_rotation, /rootfs/var/log/syslog, 10485760, /usr/bin/logrotate.sh
*.* :omfile:$log_rotation

Where

1. Just a default format for the syslog file output.
2. Load journal logs and use the state file /rootfs/var/log/syslog.statefile in case of this container will restart.
3. Create the output channel as a /rootfs/var/log/syslog file, and call /usr/bin/logrotate.sh if file will be larger 10Mb (10485760 bytes).
4. Stream all data (in our case only journal logs) to the created channel from (3).

log rotation

A good thing about rsyslog is that it will take care of when to call log rotation. We just need to configure logrotate. For that, I have created a bash script

#!/bin/bash

/usr/sbin/logrotate /etc/journald-to-syslog/logrotate.conf

And default configuration to keep five rotated files and rotate when file size is more than 10M (should be in sync with the size we used in rsyslog.conf).

/rootfs/var/log/syslog {
    rotate 5
    create
    size 10M
}

entrypoint.sh

To make sure that rsyslog will use the right hostname, and not the one autogenerated by docker, I also created an entrypoint.sh which can set the right hostname before it will start rsyslogd in the foreground.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

#!/bin/bash

if [[ -z "${KUBERNETES_NODENAME}" ]]; then
    hostname $(cat /etc/hostname)
else
    hostname "${KUBERNETES_NODENAME}"
fi


/usr/sbin/rsyslogd -f /etc/journald-to-syslog/rsyslog.conf -n

K8S configuration: journald-to-syslog.yaml

The last step is to schedule journald-to-syslog container. We will schedule it on every Kubernetes node with the DaemonSet workload.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: journald-to-syslog
  labels:
    app: journald-to-syslog
spec:
  updateStrategy:
    type: RollingUpdate

  selector:
    matchLabels:
      daemon: journald-to-syslog

  template:
    metadata:
      name: journald-to-syslog
      labels:
        daemon: journald-to-syslog
    spec:
      tolerations:
      - operator: "Exists"
        effect: "NoSchedule"
      - operator: "Exists"
        effect: "NoExecute"
      containers:
      - name: journald-to-syslog
        image: outcoldsolutions/journald-to-syslog
        securityContext:
          privileged: true
        resources:
          limits:
            cpu: 1
            memory: 128Mi
          requests:
            cpu: 50m
            memory: 32Mi
        volumeMounts:
        - name: machine-id
          mountPath: /etc/machine-id
          readOnly: true
        - name: hostname
          mountPath: /etc/hostname
          readOnly: true
        - name: journal
          mountPath: /var/log/journal
          readOnly: true
        - name: var-log
          mountPath: /rootfs/var/log/
      volumes:
      - name: machine-id
        hostPath:
          path: /etc/machine-id
      - name: hostname
        hostPath:
          path: /etc/hostname
      - name: var-log
        hostPath:
          path: /var/log
      - name: journal
        hostPath:
          path: /var/log/journal

Few highlights:

• line 23 allows us to schedule this DaemonSet on masters as well.
• line 38 tells journald the right ID, where to look for the journal logs.
• line 41 helps us to know the hostname of the host Linux.

Splunk and Monitoring Kubernetes application

If you have not done that yet, you can install collector and our application in Splunk by following our manual on How to get started with Kubernetes.

One small modification is required for Monitoring Kubernetes v3.0 is to change in macro_kubernetes_host_logs_kubelet from syslog_component::kubelet to syslog_component::kubelet*, as in case of Tectonic kubelet process is called kubelet-wrapper.

You can do that in Settings → Advanced Search → Search marcos, modify macro_kubernetes_host_logs_kubelet to

(`macro_kubernetes_host_logs` AND (syslog_component::kubelet* OR source="*/kubelet.log*"))

After that, you should be able to see host logs, including kubelet logs