How to collect network logs from an OpenShift 4 cluster via must-gather?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Network must-gather

Issue

  • Is it possible to gather network related info like iptables config and logs on all nodes?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

The must-gather image comes with a Content from github.com is not included.gather_network_logs script which gathers various network related logs on all nodes in a cluster. To use that script for collecting network data, the following command can be used:

$ oc adm must-gather -- /usr/bin/gather_network_logs

There are some differences in the network data collected between different Red Hat OpenShift Container Platform releases:

  • Starting with Red Hat OpenShift Container Platform 4.14, the standard must-gather scripts only collects basic network logs. The /usr/bin/gather_network_logs script collects extended network information and also the standard must-gather information. Refer to Content from github.com is not included.the commit that splits the gather_network_logs script for additional information.

  • In versions 4.12 and 4.13, the /usr/bin/gather_network_logs script is executed as part of any standard must-gather, so it is not needed to manually specify the script when running oc adm must-gather command.

  • In versions prior to 4.12, by default, no network information was gathered and using the above command is required.

  • Starting with OpenShift Container Platform 4.10, the network gather scripts collect less data as this would be collected by new sos report plugins. In the meantime, it may be necessary to run the 4.9 must-gather image, for example for the 4.9.23 image, use:

    $ oc adm must-gather --image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81e85b5b7dcc68a4561e810570493b1e859e27294a4a3192afd77d2764c990fe -- /usr/bin/gather_network_logs

  • Starting with OpenShift Container Platform 4.7, the gather_network_logs can also gather data for OVNKubernetes. Otherwise, for earlier versions, see: how to gather data for Openshift OVN-Kubernetes in OpenShift 4.6 and older.

Root Cause

The must-gather image comes with a gather_network_logs script which gathers various network related logs from all nodes in an OpenShift 4 cluster.

Diagnostic Steps

The script is Content from github.com is not included.installed in the /usr/bin path of the image and can be called upon like this:

$ oc adm must-gather -- /usr/bin/gather_network_logs
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14f97b5e9195c8f9b682523dc1febbb1f75fd60d408ac2731cdfa047aee0f43d
[must-gather      ] OUT namespace/openshift-must-gather-29wfb created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-2d9sd created
[must-gather      ] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14f97b5e9195c8f9b682523dc1febbb1f75fd60d408ac2731cdfa047aee0f43d created
[must-gather-wphp5] POD WARNING: Collecting network logs on ALL nodes in your cluster. This could take a long time.
[must-gather-wphp5] POD INFO: Waiting for node network log collection to complete ...
[must-gather-wphp5] POD INFO: Node network log collection to complete.
[must-gather-wphp5] OUT waiting for gather to complete
[must-gather-wphp5] OUT downloading gather output
[must-gather-wphp5] OUT receiving incremental file list
[must-gather-wphp5] OUT ./
[must-gather-wphp5] OUT network_logs/
[must-gather-wphp5] OUT network_logs/ip-10-0-146-190.us-east-2.compute.internal_iptables
[must-gather-wphp5] OUT network_logs/ip-10-0-146-190.us-east-2.compute.internal_ovs-xxtl5_ovsdb_log
[must-gather-wphp5] OUT network_logs/ip-10-0-146-190.us-east-2.compute.internal_ovs-xxtl5_vswitchd_log
[must-gather-wphp5] OUT network_logs/ip-10-0-146-190.us-east-2.compute.internal_ovs_dump
[...]
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.