Outcold Solutions LLC

Monitoring OpenShift - Version 5

Forwarding Audit Logs

OpenShift 4.x has audit log enabled by default. No additional configurations are required.

Our solution provides detailed Audit dashboards. By default, OpenShift does not provide audit logs. You can enable them by following instructions from OpenShift documentation Master and Node Configuration/Advanced Audit .

You need to enable audit log only on Masters. For that, you need to edit master-config.yaml file. 

sudo vi /etc/origin/master/master-config.yaml

And add configuration, to keep for example audit for the last 10 days or maximum of 3 files with the size of 100 megabytes. 

auditConfig:
  auditFilePath: "/var/lib/origin/openpaas-oscp-audit/openpaas-oscp-audit.log"
  enabled: true
  maximumFileRetentionDays: 10
  maximumFileSizeMegabytes: 100
  maximumRetainedFiles: 3
  policyFile: "/etc/origin/master/audit-policy.yaml"
  logFormat: json

Collector automatically forwards logs from /var/lib/origin/openpaas-oscp-audit/, no additional configuration is required in the collector.

We updated our configuration to use /var/lib/origin/openpaas-oscp-audit/ instead of /var/log recently. OpenShift starting from version 3.10 does not map /var/log inside the container with the master API.

You need to create an audit-policy.yaml in the specified path. 

sudo vi /etc/origin/master/audit-policy.yaml

Use our example as a reference. We are hiding most of the system activity.

Another good example of the audit-policy.yaml file is an audit profile used by GCE. 
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  # Do not log from kube-system accounts
  - level: None
    userGroups:
    - system:serviceaccounts:kube-system
    - system:serviceaccounts:openshift-infra
    - system:serviceaccounts:kube-service-catalog
    - system:serviceaccounts:openshift-template-service-broker
    - system:serviceaccounts:openshift-sdn
    - system:serviceaccounts:openshift-node
    - system:serviceaccounts:openshift-ansible-service-broker
    - system:serviceaccounts:openshift-monitoring
  - level: None
    users:
    - system:apiserver
    - system:kube-scheduler
    - system:volume-scheduler
    - system:kube-controller-manager
    - system:node
    - system:openshift-master

  # Do not log from collector
  - level: None
    users:
    - system:serviceaccount:collectorforopenshift:collectorforopenshift

  # Don't log nodes communications
  - level: None
    userGroups:
    - system:nodes

  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
    - /healthz*
    - /version
    - /swagger*

  # Log configmap and secret changes in all namespaces at the metadata level.
  - level: Metadata
    resources:
    - resources: ["secrets", "configmaps"]

  # A catch-all rule to log all other requests at the request level.
  - level: Request

Restart services after making changes. 

sudo /usr/local/bin/master-restart api
sudo /usr/local/bin/master-restart controllers
If you are using openshift 3.9 or below, the command to restart the services is sudo systemctl restart atomic-openshift-master-api atomic-openshift-master-controllers

Application has a macro, that defines how to find the audit logs macro_openshift_audit_logs. 

(`macro_openshift_host_logs` "audit.k8s.io")

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.