7 Logging #

Logging is ubiquitous throughout SUSE CaaS Platform. Some tools will only print their outputs to the currently running session shell and not create a "log file".

If you need to retain the output of these files you can tee them into a separate file (refer to Section 7.2, “Logging in skuba”).

Many other service components will produce log files or other log info streams. You can collect, store and evaluate these logs via Section 7.4, “Centralized Logging” for use with the Section 8.1, “Monitoring Stack”.

Note

If you are looking for troubleshooting logs please refer to Section 15.3, “Log collection”.

One important part of deploying and maintaining a product is to have reliable logs. Tools like skuba take the approach of printing the output to the standard output directly. This is not just common practice, but it also has the advantage that then the user has more flexibility on how to manage said output.

Thus, whenever throughout this guide we write a skuba command, take into account that the output will be printed into the standard output. If you would also like to have the logs stored somewhere else for later inspection, you can use tools like tee. For example:

skuba node bootstrap --user sles --sudo --target <IP/FQDN> <NODE_NAME> | tee <NODE_NAME>-skuba-node-bootstrap.log

Otherwise, you might want to use other tools to manage the logs for later inspection. The point being that this guide will never consider how to manage these logs because skuba itself does not. It’s up to you to manage these logs in any way you find desirable.

Moreover, skuba has also various levels of log verbosity. This is managed by the -v, --verbosity flag. This flag accepts an integer argument, ranging from 0 to 5, where a higher number means a higher level of verbosity. If you don’t pass any arguments, then 0 is assumed. We recommend using the default argument, since it will already log warnings and errors, among other relevant output, whereas 5 can be a bit overwhelming. Thus, for the above example, we would recommend something like:

skuba node bootstrap -v --user sles --sudo --target <IP/FQDN> <NODE_NAME> | tee <NODE_NAME>-skuba-node-bootstrap.log

Now the <NODE_NAME>-skuba-node-bootstrap.log will have more useful information than without the -v flag. We strongly recommend using this flag in order to get as much useful information as possible from a single run.

To track actions that have been performed on the cluster, you can enable the Kubernetes audit log during cluster bootstrap or on a running cluster.

This allows the audit logs to be written on the Kubernetes master nodes at /var/log/kube-apiserver/audit.log or the given path.

For more information on the audit log and its contents, see: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/

7.3.3 Enable Auditing On Running Cluster #

Note

The following steps take effect only on the updated master nodes. You need to repeat the following steps on every master node in the cluster.

1. Create audit policy file - audit.yaml. Here uses a simple policy for demonstration.

   apiVersion: audit.k8s.io/v1beta1
   kind: Policy
   rules:
     - level: Metadata 1

   1	The audit level of the event. This sample will log all requests at the Metadata level. For detailed information, refer to: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy

2. Create audit policy file directory on master node.

   sudo mkdir -p /etc/kubernetes/policies

3. Copy audit policy file - audit.yaml to /etc/kubernetes/policies/audit.yaml on master node.

4. Edit /etc/kubernetes/manifests/kube-apiserver.yaml.

    ...
   spec:
     containers:
     - command:
       - kube-apiserver
       - --audit-log-path=/var/log/kube-apiserver/audit.log
       - --audit-policy-file=/etc/kubernetes/policies/audit.yaml 1
       - --audit-log-maxage=30 2
       - --audit-log-maxsize=100 3
       - --audit-log-maxbackup=5 4
       - --audit-log-format=json 5
    ...
       volumeMounts:
       - mountPath: /etc/kubernetes/policies/audit.yaml 6
         name: audit-policy
         readOnly: true
       - mountPath: /var/log/kube-apiserver 7
         name: audit-logs
    ...
     volumes:
     - hostPath:
         path: /etc/kubernetes/policies/audit.yaml 8
         type: File
       name: audit-policy
     - hostPath:
         path: /var/log/kube-apiserver 9
         type: DirectoryOrCreate
       name: audit-logs
    ...

   1	Path to the YAML file that defines the audit policy configuration.
   2	The maximum number of days to retain old audit log files based on the timestamp encoded in their filename. (Default: 15)
   3	The maximum size in megabytes of the audit log file before it gets rotated. (Default: 10)
   4	The maximum number of old audit log files to retain. (Default: 20)
   5	Format of saved audits. Known formats are "legacy", "json". "legacy" indicates 1-line text format for each event. "json" indicates structured json format.
   6	The audit policy configuration file path on the api-server pod.
   7	The audit log file directory on the api-server pod.
   8	The audit policy configuration file path from the host node’s filesystem.
   9	The audit log file directory from the host node’s filesystem.

5. Restart kubelet.

   sudo systemctl restart kubelet

6. If everything is set up correctly, you should be able to see audit logs being written to /var/log/kube-apiserver/audit.log.

7.3.4 Disable Auditing #

Note

The following steps take effect only on the updated master nodes. You need to repeat the following steps on every master node in the cluster.

1. Remote access to the master node.

ssh sles@<master_node>

1. Edit /etc/kubernetes/manifests/kube-apiserver.yaml and remove the following lines.

   ...
      - --audit-log-path=/var/log/kube-apiserver/audit.log
      - --audit-policy-file=/etc/kubernetes/policies/audit.yaml
      - --audit-log-maxage=30
      - --audit-log-maxsize=100
      - --audit-log-maxbackup=5
      - --audit-log-format=json
   ...
      - mountPath: /etc/kubernetes/policies/audit.yaml
        name: audit-policy
        readOnly: true
      - mountPath: /var/log/kube-apiserver
        name: audit-logs
   ...
    - hostPath:
        path: /etc/kubernetes/policies/audit.yaml
        type: File
      name: audit-policy
    - hostPath:
        path: /var/log/kube-apiserver
        type: DirectoryOrCreate
      name: audit-logs

2. Restart kubelet.

   sudo systemctl restart kubelet

Centralized Logging is a means of collecting logs from the SUSE CaaS Platform for centralized management. It forwards system and Kubernetes cluster logs to a specified external logging service, for example, Rsyslog server.

Collecting logs in a central location can be useful for audit or debug purposes or to analyze and visually present data.

7.4.3 Log Formats #

The two supported syslog message formats are RFC 5424 and RFC 3164.

Note

The Kubernetes cluster metadata is included in the RFC 5424 message.

Example RFC 3164

2019-05-30T09:11:21.968458+00:00 worker1 k8s.system/crio[12080] level=debug msg="Endpoint successfully created" containerID=caa46f14a68e766b877af01442e58731845bb45d8ce1f856553440a02c958b2f eventUUID=e2405f2a-82ba-11e9-9a06-fa163eebdfd6 subsys=cilium-cni

Example RFC 5424

<133>1 2019-05-30T08:28:38.784214+00:00 master0 k8s.pod/kube-system/kube-apiserver-master0/kube-apiserver - - [kube_meta namespace_id="1e030def-81db-11e9-a62b-fa163e1876c9" container_name="kube-apiserver" creation_timestamp="2019-05-29T06:29:31Z" host="master0" namespace_name="kube-system" master_url="https://kubernetes.default.svc.cluster.local:443" pod_id="4aaf10f9-81db-11e9-a62b-fa163e1876c9" pod_name="kube-apiserver-master0"] 2019-05-30T08:28:38.783780355+00:00 stderr F I0530 08:28:38.783710       1 log.go:172] http: TLS handshake error from 172.28.0.19:45888: tls: client offered only unsupported versions: [300]

7.4.4 Deployment #

After you have successfully installed it, use Helm CLI to install log agents on each node, and provide customized settings via specific command options.

The only three mandatory parameters for a successful deployment of Centralized Logging are:

• server.host, default value = rsyslog-server.default.svc.cluster.local

• server.port, default value = 514

• server.protocol, default value = TCP

See Section 7.4.6, “Optional settings” for the optional parameters and their default values.

• Running the following will create the minimal working setup:

helm repo add suse https://kubernetes-charts.suse.com
helm install <RELEASE_NAME> suse/log-agent-rsyslog --namespace kube-system --set server.host=${SERVER_HOST} --set server.port=${SERVER_PORT}

Note

If not specified otherwise, Helm will install log agents with TCP as the default value for server.protocol.

Warning

Currently, Rsyslog can cause a segfault (as described in https://github.com/rsyslog/rsyslog/issues/4200) as result of a conflict when both imjournal and imfile input modules are enabled. To avoid crashing Rsyslog, imfile and imjournal need to be mutually exclusive, which means they need to be configured in the Rsyslog helm chart to one of these:

--set logs.osSystem.enabled=true \ 1
--set logs.kubernetesSystem.enabled=false \ 2
--set logs.kubernetesControlPlane.enabled=false \ 3
--set logs.kubernetesUserNamespaces.enabled=false \ 4

1	Enables imfile
2 3 4	Disables imjournal

--set logs.osSystem.enabled=false \ 1
--set logs.kubernetesSystem.enabled=true \ 2
--set logs.kubernetesControlPlane.enabled=true \ 3
--set logs.kubernetesUserNamespaces.enabled=true \ 4

1	Disables imfile
2 3 4	Enables imjournal

If it is required to send both imjournal and imfile, Rsyslog needs to be installed directly on the host nodes to send the imjournal log and again deployed via the helm chart on the Kubernetes cluster to send imfile logs.

After this step, all of the log agents will initialize then start to forward logs from each node to the configured remote Rsyslog server.

• To check the installation progress, use the helm status command:

helm status <RELEASE_NAME> --namespace kube-system

• To uninstall log agents, use the helm delete command:

helm uninstall <RELEASE_NAME> --namespace kube-system

7.4.6 Optional settings #

Note

Options with empty default values are set as not specified.

Parameter	Function	Default value
image.repository	specifies image repository to pull from	registry.suse.com/caasp/v4.5/rsyslog
image.tag	specifies image tag to pull	8.39.0
kubernetesPodAnnotationsEnabled	enables kubernetes meta annotations in pod logs	false
kubernetesPodLabelsEnabled	enables kubernetes meta labels in pod logs	false
logs.kubernetesAudit.enabled	enables Kubernetes audit logs	true
logs.kubernetesAudit.logDir	Kubernetes audit log directory	/var/log/kube-apiserver
logs.kubernetesAudit.logFile	Kubernetes audit log filename	audit.log
logs.kubernetesControlPlane.enabled	enables Kubernetes control plane logs	true
logs.kubernetesSystem.enabled	enables Kubernetes system logs (kubelet, crio)	true
logs.kubernetesUserNamespaces.enabled	enables Kubernetes user namespaces logs	false
logs.kubernetesUserNamespaces.exclude	excludes Kubernetes logs for specific namespaces	- ""
logs.osSystem.enabled	enables OS logs (auditd, kernel, wicked, zypper)	true
persistStateInterval	sets interval (number-of-messages) for data state persistency	100
queue.enabled	enables Rsyslog queue	false
queue.maxDiskSpace	sets maximum Rsyslog queue disk space in bytes	2147483648
queue.size	sets Rsyslog queue size in bytes	50000
resources.limits.cpu	sets CPU limits
resources.limits.memory	sets memory limits	512 Mi
resources.requests.cpu	sets CPU for requests	100m
resources.requests.memory	sets memory for requests	512 Mi
resumeInterval	specifies time (seconds) after failure before retry is attempted	30
resumeRetryCount	sets number of retries after first failure before the log is discarded. -1 is unlimited	-1
server.tls.clientCert	sets TLS client certificate
server.tls.clientKey	sets TLS client key
server.tls.enabled	enables TLS	false
server.tls.permittedPeer	sets a list of TLS/fingerprints or TLS/names with permission to access the server
server.tls.rootCa	specifies TLS root certificate authority