How to collect audit logs from control plane nodes in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Audit logs

Issue

  • How to collect audit logs from control plane nodes in Red Hat OpenShift Container Platform (RHOCP) 4.
  • How to collect audit logs in addition to "standard" must-gather information in RHOCP 4?

Resolution

The preferred way to collect audit logs is using the gather_audit_logs script from the must-gather as explained in gathering audit logs:

$ oc adm must-gather -- /usr/bin/gather_audit_logs

It is possible to collect the audit logs in addition to the "standard" must-gather information with the following command:

$ oc adm must-gather -- '/usr/bin/gather && /usr/bin/gather_audit_logs'

When the command finish, compress the generated must-gather.local directory:

$ tar -zvcf must-gather.tar.gz must-gather.local.[random-ID]/

Note: the audit log policy profile should be Default, WriteRequestBodies or AllRequestBodies for generating audit log events that can be analyzed. Refer to configuring the audit log policy for the different configurations.

If the oc adm must-gather command is not working for collecting the audit logs, try to collect them with the following script:

Collecting audit logs from control plane nodes

Create and execute the following script to collect kube-apiserver and openshift-apiserver audit logs from all master nodes:

$ cat <<'EOF' > collect-audit-logs.sh
LOGDIR=ocp-controlplane-logs
mkdir $LOGDIR
for path in kube-apiserver openshift-apiserver ; do
  oc adm node-logs --role=master --path=$path | while read line ; do 
    echo "Downloading $line ..." 
    node=$(echo "$line" | awk '{print $1}')
    log=$(echo "$line" | awk '{print $2}')
    mkdir -p $LOGDIR/$node/$path 2>/dev/null
    oc adm node-logs $node --path=$path/$log > $LOGDIR/$node/$path/$log
  done
done
EOF
chmod +x collect-audit-logs.sh
./collect-audit-logs.sh

The log collection might take a while:

$ ./collect-audit-logs.sh
Downloading openshift-master-0.example.com audit.log ...
Downloading openshift-master-1.example.com audit.log ...
Downloading openshift-master-2.example.com audit.log ...
Downloading openshift-master-0.example.com audit.log ...
Downloading openshift-master-1.example.com audit.log ...
Downloading openshift-master-2.example.com audit.log ...

After log collection is complete, create a tar archive and share the archive with Red Hat Technical Support:

$ tar -zvcf ocp-master-logs.tar.gz ocp-master-logs

Root Cause

The preferred way to collect audit logs is using the gather_audit_logs script from the must-gather.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.