How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4

Solution Verified - Updated 4 Feb 2026

---

Environment

• Red Hat OpenShift Container Platform (RHOCP) 4

Issue

• How to collect worker metrics to troubleshoot CPU load, memory pressure, and interrupt issues on worker nodes in OCP 4
• Openshift collect performance metrics report for CPU load, memory pressure, and interrupt issues
• Openshift collect sar report, CPU load, memory pressure, and interrupt issues
• Openshift collect custom performance metrics to troubleshoot CPU load, memory pressure, and interrupt issues
• Openshift node performance custom monitoring script

Resolution

Prerequisites

Login to the OCP cluster as a cluster-admin user using the oc utility.

Create a namespace metrics-debug and grant the default service account of metrics-debug project privileged SCC:

cat <<'EOF' > create-metrics-debug-namespace.sh 
#!/bin/bash

oc new-project metrics-debug
oc adm policy add-scc-to-user privileged -z default
EOF
bash create-metrics-debug-namespace.sh
oc project metrics-debug

Metrics and sar monitoring

Then, create a configmap that contains the metrics collection script. Adjust the configmap as needed:

cat <<'EOF' > metrics-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: metrics-scripts
  namespace: metrics-debug
data:
  install-requirements.sh: |
    #!/bin/bash
    yum install procps-ng perf psmisc hostname iproute sysstat iotop -y
  collect-metrics.sh: |
    #!/bin/bash
    # Interval in seconds
    INTERVAL=1 
    # Duration in seconds (or whatever is acceptable as an argument to "sleep", default is infinity)
    #DURATION=600
    DURATION="inf"
    echo "Gathering metrics ..."
    mkdir /metrics
    rm -Rf /metrics/*
    # Hostname of the system 
    uname -n > /metrics/hostname.txt
    hostname >> /metrics/hostname.txt
    # All PIDs
    # pidstat -p ALL -T ALL -I -l -r  -t  -u -w ${INTERVAL} > /metrics/pidstat.txt &
    # Only Monitor Active processes
    pidstat -ruwh ${INTERVAL} > /metrics/pidstat.txt &
    PIDSTAT=$!
    sar -A ${INTERVAL} > /metrics/sar.txt &
    SAR=$!
    bash -c "while true ; do date ; free -m ; sleep ${INTERVAL} ; done" > /metrics/free.txt &
    FREE=$!
    bash -c "while true ; do date ; cat /proc/softirqs; sleep ${INTERVAL}; done" > /metrics/softirqs.txt &
    SOFTIRQS=$!
    bash -c "while true ; do date ; cat /proc/interrupts; sleep ${INTERVAL}; done" > /metrics/interrupts.txt &
    INTERRUPTS=$!
    #Enable if required
    #bash -c "while true; do date ; ps aux | sort -nrk 3,3 | head -n 20 ; sleep ${INTERVAL} ; done" > /metrics/ps.txt &
    #PS=$!
    #iotop -Pobt > /metrics/iotop.txt &
    #IOTOP=$!
    echo "Metrics gathering started. Please wait for completion..."
    sleep "${DURATION}"
    kill $PIDSTAT
    kill $SAR
    kill $PS
    kill $FREE
    kill $SOFTIRQS
    kill $INTERRUPTS
    kill $IOTOP
    sync
    tar -czf /metrics.tar.gz /metrics
    echo "Done with metrics collection."
EOF
oc apply -f metrics-configmap.yaml

Adjust the INTERVAL and DURATION script variables as per script comments near them in order to configure how often metrics are taken and for how long recollection will happen.

Then, create a DaemonSet that runs with root privileges on all worker nodes to gather required metrics:

cat <<'EOF' > metrics-daemonset.yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: metrics-daemonset
  namespace: metrics-debug
  labels:
    app: metrics-daemonset
spec:
  selector:
    matchLabels:
      app: metrics-daemonset
  template:
    metadata:
      labels:
        app: metrics-daemonset
    spec:
      # Remove nodeSelector to run all all OCP nodes or change the label to run a selective nodes
      nodeSelector:
        node-role.kubernetes.io/worker: ""
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: metrics-daemonset
        image: fedora:latest
        command:
          - "/bin/bash"
          - "-c"
          - "bash /entrypoint/install-requirements.sh && bash /entrypoint/collect-metrics.sh && sleep infinity"
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: entrypoint
          mountPath: /entrypoint
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          privileged: true
      volumes:
      - name: entrypoint
        configMap:
          name: metrics-scripts
EOF
oc apply -f metrics-daemonset.yaml

Default design of this script is to capture monitoring data every second for infinity times . In case different timers are needed, tune the configmap's script.

Follow the pods' logs:

$ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
metrics-daemonset-49n8p   1/1     Running   0          8m31s
metrics-daemonset-4r9hw   1/1     Running   0          8m31s
metrics-daemonset-jbdzc   1/1     Running   0          8m31s

$ oc logs --prefix --timestamps -f -l app=metrics-daemonset
Fedora 32 openh264 (From Cisco) - x86_64        4.0 kB/s | 5.1 kB     00:01    
Fedora Modular 32 - x86_64                      2.8 MB/s | 4.9 MB     00:01    
Fedora Modular 32 - x86_64 - Updates             11 MB/s | 3.5 MB     00:00    
(...)                                
  xz-5.2.5-1.fc32.x86_64                                                        

Complete!
Gathering metrics ...
Gathering pidstat ...

If a certain capture duration is set in the script, After the monitoring duration, one will see:

Done with metrics collection.

This should be shown for each of the DaemonSet pods.

Then, collect the metrics from the desired pods. Make sure to include both the pod name as well as the node name when naming the file, before uploading it to a support case:

$ oc get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE    IP             NODE                                         NOMINATED NODE   READINESS GATES
metrics-daemonset-8h7ph   1/1     Running   0          5m6s   10.0.209.241   ip-10-0-209-241.eu-west-3.compute.internal   <none>           <none>
metrics-daemonset-jgssv   1/1     Running   0          5m6s   10.0.141.185   ip-10-0-141-185.eu-west-3.compute.internal   <none>           <none>
metrics-daemonset-ntmvm   1/1     Running   0          5m6s   10.0.163.162   ip-10-0-163-162.eu-west-3.compute.internal   <none>           <none>

$ oc cp metrics-daemonset-8h7ph:/metrics.tar.gz metrics.metrics-daemonset-8h7ph.ip-10-0-209-241.eu-west-3.compute.internal.tar.gz
tar: Removing leading `/' from member names

And verify the file:

$ tar -tf metrics.metrics-daemonset-8h7ph.ip-10-0-209-241.eu-west-3.compute.internal.tar.gz
metrics/
metrics/pidstat.txt
metrics/softirqs.txt
metrics/interrupts.txt
metrics/sar.txt
metrics/ps.txt
metrics/free.txt

One can also collect the logs for all pods with the following command:

mkdir metrics/
for pod in $(oc get pods -o name | awk -F '/' '{print $NF}'); do 
    oc cp $pod:/metrics.tar.gz metrics/metrics.$pod.tar.gz
done
tar -czf metrics.tar.gz metrics

Also, attach the output of the following to the support case:

 oc get pods -o wide

For example, a valid archive and output to provide to Red Hat Technical Support would look like this:

[root@openshift-jumpserver-0 ~]# tar -tf metrics.tar.gz 
metrics/
metrics/metrics.metrics-daemonset-78rrb.tar.gz
metrics/metrics.metrics-daemonset-s8xcb.tar.gz
oc [root@openshift-jumpserver-0 ~]# oc get pods -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP                NODE                 NOMINATED NODE   READINESS GATES
metrics-daemonset-78rrb   1/1     Running   0          35m   192.168.123.221   openshift-worker-1   <none>           <none>
metrics-daemonset-s8xcb   1/1     Running   0          35m   192.168.123.220   openshift-worker-0   <none>           <none>

Network monitoring

For network monitoring, if needed, also create the following daemonset for the monitor.sh script. Use the latest version of the monitor.sh script from https://access.redhat.com/articles/1311173#monitorsh-script-3:

mkdir monitor
cat <<'EOF' > monitor/monitor.sh
#!/bin/bash
# monitor.sh begins here
# Save this script as monitor.sh
# Allocate read write execute permissions: chmod +rwx monitor.sh
# Help available with: ./monitor.sh -h

VERSION=47

USAGE=$(cat <<-EOM
Usage: monitor.sh [-d DELAY] [-i ITERATIONS] [-h]

This script collects data relevant to network debugging.
Valid parameters are all optional.

-d DELAY
Specifies a delay between collections. Default is 30 seconds.

    Examples:
    ./monitor.sh -d 10   # 10 seconds
    ./monitor.sh -d 2    # 2 seconds

-i ITERATIONS
Specifies the number of collections. Default is to run forever.

  Examples:
  ./monitor.sh -i 10   # 10 iterations
  ./monitor.sh -i 2    # 2 iterations

-p
Disables process collection in "ss", except when SS_OPTS used.
Default is process collection enabled when SS_OPTS not provided.

  Example:
  ./monitor.sh -p

-h
Displays this help message.

  Example:
  ./monitor.sh -h

Options can be combined.

  Example:
  ./monitor.sh -d 10 -i 360    # run every 10 secs, for an hour

This script recognizes an environment variable SS_OPTS which will
override the script's default command line switches when running
the 'ss' utility.

  Example:
  env SS_OPTS="-pantoemi sport = :22" bash monitor.sh
EOM
)

## defaults

DELAY=30
ITERATIONS=-1
DEF_SS_OPTS="-noemitaup"
DEF_SS_OPTS_NOP="-noemitau"

## option parsing

REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS}

while getopts ":d:i:ph" OPT; do
    case "$OPT" in
        "d")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                DELAY="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for delay. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "i")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                ITERATIONS="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for iterations. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "p")
            REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS_NOP}
            ;;
        "h")
            echo "$USAGE"
            exit 0
            ;;
        ":")
            echo "ERROR: -$OPTARG requires an argument. Run 'monitor.sh -h' for help."
            exit 1
            ;;
        "?")
            echo "ERROR: -$OPTARG is not a valid option. Run 'monitor.sh -h' for help."
            exit 1
            ;;
    esac
done

if [ -z "$SS_OPTS" ] ; then
    if ! ss -S 2>&1 | grep -q "invalid option"; then
        REAL_SS_OPTS+="S"
    fi
fi

## reporting

if [ "$ITERATIONS" -gt 0 ]; then
    echo "Running network monitoring with $DELAY second delay for $ITERATIONS iterations."
else
    echo "Running network monitoring with $DELAY second delay. Press Ctrl+c to stop..."
fi

## one-time commands

MQDEVS=( $(tc qdisc show | awk '/^qdisc mq/{print $(NF-1)}') )

## data collection loop
while [ "$ITERATIONS" != 0 ]; do

    #start timer in background
    eval sleep "$DELAY" &

    now=$(date +%Y_%m_%d_%H)
    then=$(date --date="yesterday" +%Y_%m_%d_%H)
    rm -rf "$HOSTNAME-network_stats_$then"
    mkdir -p "$HOSTNAME-network_stats_$now"

    if ! [ -e "$HOSTNAME-network_stats_$now/version.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/version.txt"
        echo "This output created with monitor.sh version $VERSION" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "See https://access.redhat.com/articles/1311173" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Delay: $DELAY" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Iterations: $ITERATIONS" >> "$HOSTNAME-network_stats_$now/version.txt"
    echo "SS_OPTS: $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/version.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/sysctl.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/sysctl.txt"
        sysctl -a 2>/dev/null >> "$HOSTNAME-network_stats_$now/sysctl.txt"
    fi  
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-address.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-address.txt"
        ip address list >> "$HOSTNAME-network_stats_$now/ip-address.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-route.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-route.txt"
        ip route show table all >> "$HOSTNAME-network_stats_$now/ip-route.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/uname.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/uname.txt"
        uname -a >> "$HOSTNAME-network_stats_$now/uname.txt"
    fi

    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ip_neigh"
    ip neigh show >> "$HOSTNAME-network_stats_$now/ip_neigh"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    tc -s qdisc >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    if [ "${#MQDEVS[@]}" -gt 0 ]; then
        for MQDEV in "${MQDEVS[@]}"; do
            echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
            tc -s class show dev "$MQDEV" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
        done
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netstat"
    netstat -s >> "$HOSTNAME-network_stats_$now/netstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/nstat"
    nstat -az >> "$HOSTNAME-network_stats_$now/nstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ss"
    eval "ss $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/ss"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/interrupts"
    cat /proc/interrupts >> "$HOSTNAME-network_stats_$now/interrupts"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softnet_stat"
    cat /proc/net/softnet_stat >> "$HOSTNAME-network_stats_$now/softnet_stat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/vmstat"
    cat /proc/vmstat >> "$HOSTNAME-network_stats_$now/vmstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ps"
    ps -alfe >> "$HOSTNAME-network_stats_$now/ps"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/mpstat"
    eval mpstat -A "$DELAY" 1 2>/dev/null >> "$HOSTNAME-network_stats_$now/mpstat" &
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/top"
    top -c -b -n1 >> "$HOSTNAME-network_stats_$now/top"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/numastat"
    numastat 2>/dev/null >> "$HOSTNAME-network_stats_$now/numastat"
    if [ -e /proc/softirqs ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softirqs"
        cat /proc/softirqs >> "$HOSTNAME-network_stats_$now/softirqs"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat"
    cat /proc/net/sockstat >> "$HOSTNAME-network_stats_$now/sockstat"
    if [ -e /proc/net/sockstat6 ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat6"
        cat /proc/net/sockstat6 >> "$HOSTNAME-network_stats_$now/sockstat6"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netdev"
    cat /proc/net/dev >> "$HOSTNAME-network_stats_$now/netdev"
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV"; ethtool -S "$DEV" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV" 2>/dev/null; done
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; find /sys/devices/ -type f | grep "/net/$DEV/statistics" | xargs grep . | awk -F "/" '{print $NF}' >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; done
    if [ -e /proc/net/sctp ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        cat /proc/net/sctp/assocs >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-snmp"
        cat /proc/net/sctp/snmp >> "$HOSTNAME-network_stats_$now/sctp-snmp"
    fi
    if [ "$ITERATIONS" -gt 0 ]; then let ITERATIONS-=1; fi
    # Wait till background jobs are finished
    wait
done
#
# monitor.sh ends here
EOF

Create the configmap:

oc project metrics-debug
oc create configmap monitor --from-file=monitor/

cat <<'EOF' > monitor-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: monitor-scripts
  namespace: metrics-debug
data:
  install-requirements.sh: |
    #!/bin/bash
    yum install conntrack-tools procps-ng ethtool numactl psmisc hostname iproute iproute-tc sysstat net-tools -y
  collect-metrics.sh: |
    #!/bin/bash
    echo "Gathering metrics ..."
    mkdir /network-metrics
    rm -Rf /network-metrics/*
    echo "Gathering monitor metrics ..."
    cd /network-metrics
    bash -c "while true ; do date ; conntrack -L -n ; sleep 5; done" >> conntrack.txt &
    CONNTRACK=$!
    bash /monitor/monitor.sh -d 5 -i 120
    kill $CONNTRACK
    tar -czf /network-metrics.tar.gz /network-metrics
    echo "Done with network metrics collection."
EOF
oc apply -f monitor-configmap.yaml

Create another daemonset to gather network metrics:

cat <<'EOF' > monitor-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: monitor-daemonset
  namespace: metrics-debug
  labels:
    app: monitor-daemonset
spec:
  selector:
    matchLabels:
      app: monitor-daemonset
  template:
    metadata:
      labels:
        app: monitor-daemonset
    spec:
      tolerations:
      # this toleration is to have the daemonset runnable on master nodes
      # remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: monitor-daemonset
        image: fedora:latest
        command:
          - "/bin/bash"
          - "-c"
          - "bash /entrypoint/install-requirements.sh && bash /entrypoint/collect-metrics.sh && sleep infinity"
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: entrypoint
          mountPath: /entrypoint
        - name: monitor
          mountPath: /monitor
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          capabilities:
            add: ["NET_ADMIN"]
      volumes:
      - name: entrypoint
        configMap:
          name: monitor-scripts
      - name: monitor
        configMap:
          name: monitor
EOF
oc apply -f monitor-daemonset.yaml

The monitor.sh script will run every 5 seconds, for 120 iterations, hence 600 seconds. Adjust timers as needed.

Follow one of the pod's logs:

$ oc apply -f monitor-daemonset.yaml
daemonset.apps/monitor-daemonset created
[akaris@linux test]$ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
monitor-daemonset-28wn5   1/1     Running   0          5s
monitor-daemonset-57rds   1/1     Running   0          5s
monitor-daemonset-9kkgf   1/1     Running   0          5s
[akaris@linux test]$ oc logs -f monitor-daemonset-28wn5
Fedora 32 openh264 (From Cisco) - x86_64        4.1 kB/s | 5.1 kB     00:01    
Fedora Modular 32 - x86_64                      6.9 MB/s | 4.9 MB     00:00    
Fedora Modular 32 - x86_64 - Updates            2.3 MB/s | 3.5 MB     00:01    
Fedora 32 - x86_64 - Updates                     20 MB/s |  21 MB     00:01    
(...)
  xz-5.2.5-1.fc32.x86_64                                                        

Complete!
Gathering metrics ...
Gathering monitor metrics ...
Running network monitoring with 5 second delay for 120 iterations.

After about 10 minutes, one will see:

Done with metrics collection.

Then, collect the metrics from the desired pods. Make sure to include both the pod name as well as the node name when naming the file, before uploading it to a support case:

$ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
monitor-daemonset-28wn5   1/1     Running   0          2m25s
monitor-daemonset-57rds   1/1     Running   0          2m25s
monitor-daemonset-9kkgf   1/1     Running   0          2m25s

$ oc cp metrics-daemonset-8h7ph:/network-metrics.tar.gz network-metrics.metrics-daemonset-8h7ph.ip-10-0-209-241.eu-west-3.compute.internal.tar.gz
tar: Removing leading `/' from member names

And verify the file:

$ $ tar -tf network-metrics.metrics-daemonset-8h7ph.ip-10-0-209-241.eu-west-3.compute.internal.tar.gz
network-metrics/
network-metrics/conntrack.txt
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/version.txt
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sysctl.txt
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ip-address.txt
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ip-route.txt
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/uname.txt
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ip_neigh
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/tc_qdisc
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/tc_class_ens5
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/netstat
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/nstat
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ss
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/interrupts
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/softnet_stat
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/vmstat
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ps
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/mpstat
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/top
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/numastat
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/softirqs
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sockstat
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sockstat6
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/netdev
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_lo
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_ens5
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_ovs-system
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_br0
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_vxlan_sys_4789
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_tun0
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_vethd5816ed3@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_veth3c80c5c7@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_veth5aa269fe@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_veth3351bae0@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_vethc2de08b5@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_veth9552df07@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_lo
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_ens5
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_ovs-system
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_br0
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_vxlan_sys_4789
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_tun0
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_vethd5816ed3@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_veth3c80c5c7@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_veth5aa269fe@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_veth3351bae0@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_vethc2de08b5@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_veth9552df07@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_veth31eb6f0f@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_veth31eb6f0f@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/ethtool_veth292a2e62@if3
network-metrics/ip-10-0-144-3-network_stats_2020_08_26_12/sys_statistics_veth292a2e62@if3

Haproxy, Network and System monitoring

• Execute the steps given in the Prerequisites section.
• Create monitor-haproxy directory.

mkdir -p monitor-haproxy

• Create a monitor.sh script by executing the below scripted block after modifying the required DELAY value.

cat <<'EOF' > monitor-haproxy/monitor.sh
#!/bin/bash
# monitor.sh begins here
# Save this script as monitor.sh
# Allocate read write execute permissions: chmod +rwx monitor.sh
# Help available with: ./monitor.sh -h

VERSION=47

USAGE=$(cat <<-EOM
Usage: monitor.sh [-d DELAY] [-i ITERATIONS] [-h]

This script collects data relevant to network debugging.
Valid parameters are all optional.

-d DELAY
Specifies a delay between collections. Default is 30 seconds.

    Examples:
    ./monitor.sh -d 10   # 10 seconds
    ./monitor.sh -d 2    # 2 seconds

-i ITERATIONS
Specifies the number of collections. Default is to run forever.

  Examples:
  ./monitor.sh -i 10   # 10 iterations
  ./monitor.sh -i 2    # 2 iterations

-p
Disables process collection in "ss", except when SS_OPTS used.
Default is process collection enabled when SS_OPTS not provided.

  Example:
  ./monitor.sh -p

-h
Displays this help message.

  Example:
  ./monitor.sh -h

Options can be combined.

  Example:
  ./monitor.sh -d 10 -i 360    # run every 10 secs, for an hour

This script recognizes an environment variable SS_OPTS which will
override the script's default command line switches when running
the 'ss' utility.

  Example:
  env SS_OPTS="-pantoemi sport = :22" bash monitor.sh
EOM
)

## defaults

DELAY=2
ITERATIONS=-1
DEF_SS_OPTS="-noemitaup"
DEF_SS_OPTS_NOP="-noemitau"

## option parsing

REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS}

while getopts ":d:i:ph" OPT; do
    case "$OPT" in
        "d")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                DELAY="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for delay. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "i")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                ITERATIONS="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for iterations. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "p")
            REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS_NOP}
            ;;
        "h")
            echo "$USAGE"
            exit 0
            ;;
        ":")
            echo "ERROR: -$OPTARG requires an argument. Run 'monitor.sh -h' for help."
            exit 1
            ;;
        "?")
            echo "ERROR: -$OPTARG is not a valid option. Run 'monitor.sh -h' for help."
            exit 1
            ;;
    esac
done

if [ -z "$SS_OPTS" ] ; then
    if ! ss -S 2>&1 | grep -q "invalid option"; then
        REAL_SS_OPTS+="S"
    fi
fi

## reporting

if [ "$ITERATIONS" -gt 0 ]; then
    echo "Running network monitoring with $DELAY second delay for $ITERATIONS iterations."
else
    echo "Running network monitoring with $DELAY second delay. Press Ctrl+c to stop..."
fi

## one-time commands

MQDEVS=( $(tc qdisc show | awk '/^qdisc mq/{print $(NF-1)}') )

## data collection loop
while [ "$ITERATIONS" != 0 ]; do

    #start timer in background
    eval sleep "$DELAY" &

    now=$(date +%Y_%m_%d_%H)
    then=$(date --date="yesterday" +%Y_%m_%d_%H)
    rm -rf "$HOSTNAME-network_stats_$then"
    mkdir -p "$HOSTNAME-network_stats_$now"

    if ! [ -e "$HOSTNAME-network_stats_$now/version.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/version.txt"
        echo "This output created with monitor.sh version $VERSION" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "See https://access.redhat.com/articles/1311173" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Delay: $DELAY" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Iterations: $ITERATIONS" >> "$HOSTNAME-network_stats_$now/version.txt"
    echo "SS_OPTS: $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/version.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/sysctl.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/sysctl.txt"
        sysctl -a 2>/dev/null >> "$HOSTNAME-network_stats_$now/sysctl.txt"
    fi  
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-address.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-address.txt"
        ip address list >> "$HOSTNAME-network_stats_$now/ip-address.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-route.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-route.txt"
        ip route show table all >> "$HOSTNAME-network_stats_$now/ip-route.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/uname.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/uname.txt"
        uname -a >> "$HOSTNAME-network_stats_$now/uname.txt"
    fi

    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ip_neigh"
    ip neigh show >> "$HOSTNAME-network_stats_$now/ip_neigh"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    tc -s qdisc >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    if [ "${#MQDEVS[@]}" -gt 0 ]; then
        for MQDEV in "${MQDEVS[@]}"; do
            echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
            tc -s class show dev "$MQDEV" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
        done
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netstat"
    netstat -s >> "$HOSTNAME-network_stats_$now/netstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/nstat"
    nstat -az >> "$HOSTNAME-network_stats_$now/nstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ss"
    eval "ss $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/ss"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/interrupts"
    cat /proc/interrupts >> "$HOSTNAME-network_stats_$now/interrupts"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softnet_stat"
    cat /proc/net/softnet_stat >> "$HOSTNAME-network_stats_$now/softnet_stat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/vmstat"
    cat /proc/vmstat >> "$HOSTNAME-network_stats_$now/vmstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ps"
    ps -alfe >> "$HOSTNAME-network_stats_$now/ps"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/mpstat"
    eval mpstat -A "$DELAY" 1 2>/dev/null >> "$HOSTNAME-network_stats_$now/mpstat" &
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/top"
    top -c -b -n1 >> "$HOSTNAME-network_stats_$now/top"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/numastat"
    numastat 2>/dev/null >> "$HOSTNAME-network_stats_$now/numastat"
    if [ -e /proc/softirqs ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softirqs"
        cat /proc/softirqs >> "$HOSTNAME-network_stats_$now/softirqs"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat"
    cat /proc/net/sockstat >> "$HOSTNAME-network_stats_$now/sockstat"
    if [ -e /proc/net/sockstat6 ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat6"
        cat /proc/net/sockstat6 >> "$HOSTNAME-network_stats_$now/sockstat6"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netdev"
    cat /proc/net/dev >> "$HOSTNAME-network_stats_$now/netdev"
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}' | awk -F '@' '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV"; ethtool -S "$DEV" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV" 2>/dev/null; done
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}' | awk -F '@' '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; find /sys/devices/ -type f | grep "/net/$DEV/statistics" | xargs grep . | awk -F "/" '{print $NF}' >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; done
    if [ -e /proc/net/sctp ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        cat /proc/net/sctp/assocs >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-snmp"
        cat /proc/net/sctp/snmp >> "$HOSTNAME-network_stats_$now/sctp-snmp"
    fi
    if [ "$ITERATIONS" -gt 0 ]; then let ITERATIONS-=1; fi
    # Wait till background jobs are finished
    wait
done
#
# monitor.sh ends here
EOF

• Execute the following command to create a configmap from the above script.

oc project metrics-debug
oc create configmap monitor-haproxy --from-file=monitor-haproxy/monitor.sh

• Create a configmap object file(monitor-haproxy-configmap.yaml) by executing the below scripted block.

cat <<'EOF' > monitor-haproxy-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: monitor-haproxy-scripts
  namespace: metrics-debug
data:
  install-requirements.sh: |
    #!/bin/bash
    dnf install perf sysstat iotop conntrack-tools procps-ng ethtool numactl psmisc hostname iproute sysstat net-tools procps-ng util-linux iproute-tc strace -y
  collect-metrics.sh: |
    #!/bin/bash
    HAPROXY_PID=$(pidof -s haproxy)
    # Resolution in seconds
    SEC=5
    # Duration in seconds (or whatever is acceptable as an argument to "sleep")
    TIMES=300
    # Infinity data collections

    echo "Gathering metrics ..."
    mkdir /network-metrics
    rm -Rf /network-metrics/*
    echo "Gathering monitor metrics ..."
    cd /network-metrics
    # PID metrics
    pidstat -p ALL -T ALL -I -l -r  -t  -u -w $SEC > pidstat.txt &
    PIDSTAT=$!
    # SAR Metrics
    sar -A $SEC > sar.txt &
    SAR=$!
    bash -c "while true; do date ; ps aux | sort -nrk 3,3 | head -n 20 ; sleep $SEC ; done" > ps.txt &
    PS=$!
    bash -c "while true ; do date ; free -m ; sleep $SEC ; done" > free.txt &
    FREE=$!
    bash -c "while true ; do date ; cat /proc/softirqs; sleep $SEC; done" > softirqs.txt &
    SOFTIRQS=$!
    bash -c "while true ; do date ; cat /proc/interrupts; sleep $SEC; done" > interrupts.txt &
    INTERRUPTS=$!
    iotop -Pobt -d $SEC > iotop.txt &
    IOTOP=$!
    nsenter -n -t $HAPROXY_PID bash -c "while true ; do date ; conntrack -L -n ; sleep $SEC; done" >> conntrack.txt &
    CONNTRACK=$!
    
    ## Haproxy URL MON & generate strace in case of failure 
    check_healthz() {
      local url1="http://localhost:80/_______internal_router_healthz"
      local url2="http://localhost:1936/healthz"
      local status_code_url1
      local status_code_url2
      status_code_url1=$(curl -s -o /dev/null -w "%{http_code}" "$url1")
      status_code_url2=$(curl -s -o /dev/null -w "%{http_code}" "$url2")
      
      local timestamp
      timestamp=$(date '+%Y-%m-%d_%H-%M-%S')
      
      if [[ "$status_code_url1" -ne 200 ]]; then
         echo "[$(date)] Health check failed! for $url1 Expected HTTP 200 but got $status_code_url1."|tee -a failure.log

         if [[ -z "$HAPROXY_PID" ]]; then
            echo "[$(date)]  'Haproxy' process not found." |tee -a failure.log
            return 1
         fi

         echo "[$(date)] running strace on 'Haproxy' (PID: $HAPROXY_PID) for 10 seconds for $url1..."|tee -a failure.log

         # Run strace for 10 seconds
         timeout 10s strace -s 10140 -tt -T -v -f -p "$HAPROXY_PID" -o "haproxy-$timestamp-ur1.trace"

         echo "[$(date)] strace output saved to haproxy-$timestamp-ur1.trace"|tee -a failure.log
         return 1

      elif [[ "$status_code_url2" -ne 200 ]]; then
           echo "[$(date)] Health check failed! for $url2 Expected HTTP 200 but got $status_code_url2."|tee -a failure.log

           if [[ -z "$HAPROXY_PID" ]]; then
            echo "[$(date)]  'Haproxy' process not found."|tee -a failure.log
            return 1
           fi

           echo "[$(date)] running strace on 'Haproxy' (PID: $HAPROXY_PID) for $url2..."|tee -a failure.log

           # Run strace for 10 seconds
           timeout 10s strace -s 10240 -tt -T -v -f -p "$HAPROXY_PID" -o "haproxy-$timestamp-ur2.trace"
           echo "[$(date)] strace output saved to haproxy-$timestamp-ur2.trace"|tee -a failure.log
           return 1

      else
         echo "[$(date)] Health check passed (HTTP STATUS - ur1 = $status_code_url1 || ur2 = $status_code_url2)."|tee -a success.log
         return 0
       fi
    }

     # Continuous monitoring loop
    while true; do check_healthz;sleep 2;done &

    nsenter -n -t $HAPROXY_PID bash /monitor/monitor.sh -d $SEC
    #sleep 10
    #sync
    #kill $CONNTRACK
    #kill $PIDSTAT
    #kill $SAR
    #kill $PS
    #kill $FREE
    #kill $SOFTIRQS
    #kill $INTERRUPTS
    #kill $IOTOP    
    #tar -czf /network-metrics.tar.gz /network-metrics
    #echo "Done with network metrics collection."
EOF

• Execute the following command to create a configmap from the above configmap object file.

oc apply -f monitor-haproxy-configmap.yaml

• Create daemonset by executing the below scripted block.

cat <<'EOF' > monitor-haproxy-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: monitor-haproxy-daemonset
  namespace: metrics-debug
  labels:
    app: monitor-haproxy-daemonset
spec:
  selector:
    matchLabels:
      app: monitor-haproxy-daemonset
  template:
    metadata:
      labels:
        app: monitor-haproxy-daemonset
    spec:
      nodeSelector:
        node-role.kubernetes.io/workload: ""
      ## As per the above nodeSelector, this daemon set will only be available on node-role.kubernetes.io/workload node
      ## change if it is required, or remove the complete nodeSelector section
      tolerations:
      ## this toleration is to have the daemonset runnable on master nodes
      ## remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: monitor-haproxy-daemonset
        image: fedora:latest
        command:
          - "/bin/bash"
          - "-c"
          - "bash /entrypoint/install-requirements.sh && bash /entrypoint/collect-metrics.sh && sleep infinity"
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: entrypoint
          mountPath: /entrypoint
        - name: monitor
          mountPath: /monitor
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          privileged: true
      volumes:
      - name: entrypoint
        configMap:
          name: monitor-haproxy-scripts
      - name: monitor
        configMap:
          name: monitor-haproxy
EOF

• Label the required nodes with node-role.kubernetes.io/workload: "" label.

oc label node <node> node-role.kubernetes.io/workload=""

• Or change the nodeSelector label in the daemonset to node-role.kubernetes.io/master: "" or node-role.kubernetes.io/worker: "", etc as per the requirement.

• Execute the following command to create the daemon set from the above config.

oc apply -f monitor-haproxy-daemonset.yaml

• The above monitoring.sh collects data in 5 second interval, adjusts the time in the SEC variable if it is required.

• Here the collection agent daemonset runs in infinity loop, we must delete the daemonset after the issue is reproduced and after collecting the required data from the daemonset pod.

Collect metrics from the daemon pod

• Collect the metrics from the desired pods. Make sure to include both the pod name as well as the node name when naming the file, before uploading it to a support case:

$ oc get pods
NAME                      READY   STATUS    RESTARTS   AGE
monitor-daemonset-28wn5   1/1     Running   0          2m25s
monitor-daemonset-57rds   1/1     Running   0          2m25s
monitor-daemonset-9kkgf   1/1     Running   0          2m25s

$ oc rsh monitor-daemonset-28wn5
# tar -czf /<pod-name>-<nodename>.tar.gz /network-metrics
$ oc cp monitor-daemonset-28wn5:/<pod-name>-<nodename>.tar.gz ./<pod-name>-<nodename>.tar.gz

• Verify the file:

$ tar -tf ./<pod-name>-<nodename>.tar.gz
network-metrics/
network-metrics/conntrack.txt
[....]

- Delete the daemonset `monitor-haproxy-daemonset` if the required data has been collected. 
- Upload the archive file to the support ticket. 

haproxy monitoring

In order to monitor the haproxy (ingress-router) namespaces, create these DaemonSets and collect data the same as before:

# Created modified monitor.sh in a configmap
mkdir -p monitor-haproxy
cat <<'EOF' > monitor-haproxy/monitor.sh
#!/bin/bash
# monitor.sh begins here
# Save this script as monitor.sh
# Allocate read write execute permissions: chmod +rwx monitor.sh
# Help available with: ./monitor.sh -h

VERSION=47

USAGE=$(cat <<-EOM
Usage: monitor.sh [-d DELAY] [-i ITERATIONS] [-h]

This script collects data relevant to network debugging.
Valid parameters are all optional.

-d DELAY
Specifies a delay between collections. Default is 30 seconds.

    Examples:
    ./monitor.sh -d 10   # 10 seconds
    ./monitor.sh -d 2    # 2 seconds

-i ITERATIONS
Specifies the number of collections. Default is to run forever.

  Examples:
  ./monitor.sh -i 10   # 10 iterations
  ./monitor.sh -i 2    # 2 iterations

-p
Disables process collection in "ss", except when SS_OPTS used.
Default is process collection enabled when SS_OPTS not provided.

  Example:
  ./monitor.sh -p

-h
Displays this help message.

  Example:
  ./monitor.sh -h

Options can be combined.

  Example:
  ./monitor.sh -d 10 -i 360    # run every 10 secs, for an hour

This script recognizes an environment variable SS_OPTS which will
override the script's default command line switches when running
the 'ss' utility.

  Example:
  env SS_OPTS="-pantoemi sport = :22" bash monitor.sh
EOM
)

## defaults

DELAY=30
ITERATIONS=-1
DEF_SS_OPTS="-noemitaup"
DEF_SS_OPTS_NOP="-noemitau"

## option parsing

REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS}

while getopts ":d:i:ph" OPT; do
    case "$OPT" in
        "d")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                DELAY="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for delay. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "i")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                ITERATIONS="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for iterations. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "p")
            REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS_NOP}
            ;;
        "h")
            echo "$USAGE"
            exit 0
            ;;
        ":")
            echo "ERROR: -$OPTARG requires an argument. Run 'monitor.sh -h' for help."
            exit 1
            ;;
        "?")
            echo "ERROR: -$OPTARG is not a valid option. Run 'monitor.sh -h' for help."
            exit 1
            ;;
    esac
done

if [ -z "$SS_OPTS" ] ; then
    if ! ss -S 2>&1 | grep -q "invalid option"; then
        REAL_SS_OPTS+="S"
    fi
fi

## reporting

if [ "$ITERATIONS" -gt 0 ]; then
    echo "Running network monitoring with $DELAY second delay for $ITERATIONS iterations."
else
    echo "Running network monitoring with $DELAY second delay. Press Ctrl+c to stop..."
fi

## one-time commands

MQDEVS=( $(tc qdisc show | awk '/^qdisc mq/{print $(NF-1)}') )

## data collection loop
while [ "$ITERATIONS" != 0 ]; do

    #start timer in background
    eval sleep "$DELAY" &

    now=$(date +%Y_%m_%d_%H)
    then=$(date --date="yesterday" +%Y_%m_%d_%H)
    rm -rf "$HOSTNAME-network_stats_$then"
    mkdir -p "$HOSTNAME-network_stats_$now"

    if ! [ -e "$HOSTNAME-network_stats_$now/version.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/version.txt"
        echo "This output created with monitor.sh version $VERSION" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "See https://access.redhat.com/articles/1311173" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Delay: $DELAY" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Iterations: $ITERATIONS" >> "$HOSTNAME-network_stats_$now/version.txt"
    echo "SS_OPTS: $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/version.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/sysctl.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/sysctl.txt"
        sysctl -a 2>/dev/null >> "$HOSTNAME-network_stats_$now/sysctl.txt"
    fi  
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-address.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-address.txt"
        ip address list >> "$HOSTNAME-network_stats_$now/ip-address.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-route.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-route.txt"
        ip route show table all >> "$HOSTNAME-network_stats_$now/ip-route.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/uname.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/uname.txt"
        uname -a >> "$HOSTNAME-network_stats_$now/uname.txt"
    fi

    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ip_neigh"
    ip neigh show >> "$HOSTNAME-network_stats_$now/ip_neigh"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    tc -s qdisc >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    if [ "${#MQDEVS[@]}" -gt 0 ]; then
        for MQDEV in "${MQDEVS[@]}"; do
            echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
            tc -s class show dev "$MQDEV" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
        done
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netstat"
    netstat -s >> "$HOSTNAME-network_stats_$now/netstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/nstat"
    nstat -az >> "$HOSTNAME-network_stats_$now/nstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ss"
    eval "ss $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/ss"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/interrupts"
    cat /proc/interrupts >> "$HOSTNAME-network_stats_$now/interrupts"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softnet_stat"
    cat /proc/net/softnet_stat >> "$HOSTNAME-network_stats_$now/softnet_stat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/vmstat"
    cat /proc/vmstat >> "$HOSTNAME-network_stats_$now/vmstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ps"
    ps -alfe >> "$HOSTNAME-network_stats_$now/ps"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/mpstat"
    eval mpstat -A "$DELAY" 1 2>/dev/null >> "$HOSTNAME-network_stats_$now/mpstat" &
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/top"
    top -c -b -n1 >> "$HOSTNAME-network_stats_$now/top"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/numastat"
    numastat 2>/dev/null >> "$HOSTNAME-network_stats_$now/numastat"
    if [ -e /proc/softirqs ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softirqs"
        cat /proc/softirqs >> "$HOSTNAME-network_stats_$now/softirqs"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat"
    cat /proc/net/sockstat >> "$HOSTNAME-network_stats_$now/sockstat"
    if [ -e /proc/net/sockstat6 ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat6"
        cat /proc/net/sockstat6 >> "$HOSTNAME-network_stats_$now/sockstat6"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netdev"
    cat /proc/net/dev >> "$HOSTNAME-network_stats_$now/netdev"
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}' | awk -F '@' '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV"; ethtool -S "$DEV" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV" 2>/dev/null; done
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}' | awk -F '@' '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; find /sys/devices/ -type f | grep "/net/$DEV/statistics" | xargs grep . | awk -F "/" '{print $NF}' >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; done
    if [ -e /proc/net/sctp ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        cat /proc/net/sctp/assocs >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-snmp"
        cat /proc/net/sctp/snmp >> "$HOSTNAME-network_stats_$now/sctp-snmp"
    fi
    if [ "$ITERATIONS" -gt 0 ]; then let ITERATIONS-=1; fi
    # Wait till background jobs are finished
    wait
done
#
# monitor.sh ends here
EOF

oc project metrics-debug
oc create configmap monitor-haproxy --from-file=monitor-haproxy/

# Create modified collect scripts configmap

cat <<'EOF' > monitor-haproxy-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: monitor-haproxy-scripts
  namespace: metrics-debug
data:
  install-requirements.sh: |
    #!/bin/bash
    yum install conntrack-tools procps-ng ethtool numactl psmisc hostname iproute sysstat net-tools procps-ng util-linux -y
  collect-metrics.sh: |
    #!/bin/bash
    HAPROXY_PID=$(pidof -s haproxy)
    echo "Gathering metrics ..."
    mkdir /network-metrics
    rm -Rf /network-metrics/*
    echo "Gathering monitor metrics ..."
    cd /network-metrics
    nsenter -n -t $HAPROXY_PID bash -c "while true ; do date ; conntrack -L -n ; sleep 5; done" >> conntrack.txt &
    CONNTRACK=$!
    nsenter -n -t $HAPROXY_PID bash /monitor/monitor.sh -d 5 -i 120
    kill $CONNTRACK
    tar -czf /network-metrics.tar.gz /network-metrics
    echo "Done with network metrics collection."
EOF
oc apply -f monitor-haproxy-configmap.yaml

# Create modified daemonset (that co-exists with the original one)

cat <<'EOF' > monitor-haproxy-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: monitor-haproxy-daemonset
  namespace: metrics-debug
  labels:
    app: monitor-haproxy-daemonset
spec:
  selector:
    matchLabels:
      app: monitor-haproxy-daemonset
  template:
    metadata:
      labels:
        app: monitor-haproxy-daemonset
    spec:
      tolerations:
      ## this toleration is to have the daemonset runnable on master nodes
      ## remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: monitor-haproxy-daemonset
        image: fedora:latest
        command:
          - "/bin/bash"
          - "-c"
          - "bash /entrypoint/install-requirements.sh && bash /entrypoint/collect-metrics.sh && sleep infinity"
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: entrypoint
          mountPath: /entrypoint
        - name: monitor
          mountPath: /monitor
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          privileged: true
      volumes:
      - name: entrypoint
        configMap:
          name: monitor-haproxy-scripts
      - name: monitor
        configMap:
          name: monitor-haproxy
EOF
oc apply -f monitor-haproxy-daemonset.yaml

coredns monitoring

In order to monitor the coredns (openshift-dns) namespaces, create these DaemonSets and collect data the same as before:

# Created modified monitor.sh in a configmap
mkdir -p monitor-coredns
cat <<'EOF' > monitor-coredns/monitor.sh
#!/bin/bash
# monitor.sh begins here
# Save this script as monitor.sh
# Allocate read write execute permissions: chmod +rwx monitor.sh
# Help available with: ./monitor.sh -h

VERSION=47

USAGE=$(cat <<-EOM
Usage: monitor.sh [-d DELAY] [-i ITERATIONS] [-h]

This script collects data relevant to network debugging.
Valid parameters are all optional.

-d DELAY
Specifies a delay between collections. Default is 30 seconds.

    Examples:
    ./monitor.sh -d 10   # 10 seconds
    ./monitor.sh -d 2    # 2 seconds

-i ITERATIONS
Specifies the number of collections. Default is to run forever.

  Examples:
  ./monitor.sh -i 10   # 10 iterations
  ./monitor.sh -i 2    # 2 iterations

-p
Disables process collection in "ss", except when SS_OPTS used.
Default is process collection enabled when SS_OPTS not provided.

  Example:
  ./monitor.sh -p

-h
Displays this help message.

  Example:
  ./monitor.sh -h

Options can be combined.

  Example:
  ./monitor.sh -d 10 -i 360    # run every 10 secs, for an hour

This script recognizes an environment variable SS_OPTS which will
override the script's default command line switches when running
the 'ss' utility.

  Example:
  env SS_OPTS="-pantoemi sport = :22" bash monitor.sh
EOM
)

## defaults

DELAY=30
ITERATIONS=-1
DEF_SS_OPTS="-noemitaup"
DEF_SS_OPTS_NOP="-noemitau"

## option parsing

REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS}

while getopts ":d:i:ph" OPT; do
    case "$OPT" in
        "d")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                DELAY="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for delay. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "i")
            # something was passed, check it's a positive integer
            if [ "$OPTARG" -eq "$OPTARG" ] 2>/dev/null && [ "$OPTARG" -gt 0 ] 2>/dev/null; then
                ITERATIONS="$OPTARG"
            else
                echo "ERROR: $OPTARG not a valid option for iterations. Run 'monitor.sh -h' for help."
                exit 1
            fi
            ;;
        "p")
            REAL_SS_OPTS=${SS_OPTS:-$DEF_SS_OPTS_NOP}
            ;;
        "h")
            echo "$USAGE"
            exit 0
            ;;
        ":")
            echo "ERROR: -$OPTARG requires an argument. Run 'monitor.sh -h' for help."
            exit 1
            ;;
        "?")
            echo "ERROR: -$OPTARG is not a valid option. Run 'monitor.sh -h' for help."
            exit 1
            ;;
    esac
done

if [ -z "$SS_OPTS" ] ; then
    if ! ss -S 2>&1 | grep -q "invalid option"; then
        REAL_SS_OPTS+="S"
    fi
fi

## reporting

if [ "$ITERATIONS" -gt 0 ]; then
    echo "Running network monitoring with $DELAY second delay for $ITERATIONS iterations."
else
    echo "Running network monitoring with $DELAY second delay. Press Ctrl+c to stop..."
fi

## one-time commands

MQDEVS=( $(tc qdisc show | awk '/^qdisc mq/{print $(NF-1)}') )

## data collection loop
while [ "$ITERATIONS" != 0 ]; do

    #start timer in background
    eval sleep "$DELAY" &

    now=$(date +%Y_%m_%d_%H)
    then=$(date --date="yesterday" +%Y_%m_%d_%H)
    rm -rf "$HOSTNAME-network_stats_$then"
    mkdir -p "$HOSTNAME-network_stats_$now"

    if ! [ -e "$HOSTNAME-network_stats_$now/version.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/version.txt"
        echo "This output created with monitor.sh version $VERSION" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "See https://access.redhat.com/articles/1311173" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Delay: $DELAY" >> "$HOSTNAME-network_stats_$now/version.txt"
        echo "Iterations: $ITERATIONS" >> "$HOSTNAME-network_stats_$now/version.txt"
    echo "SS_OPTS: $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/version.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/sysctl.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/sysctl.txt"
        sysctl -a 2>/dev/null >> "$HOSTNAME-network_stats_$now/sysctl.txt"
    fi  
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-address.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-address.txt"
        ip address list >> "$HOSTNAME-network_stats_$now/ip-address.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/ip-route.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/ip-route.txt"
        ip route show table all >> "$HOSTNAME-network_stats_$now/ip-route.txt"
    fi
    if ! [ -e "$HOSTNAME-network_stats_$now/uname.txt" ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" > "$HOSTNAME-network_stats_$now/uname.txt"
        uname -a >> "$HOSTNAME-network_stats_$now/uname.txt"
    fi

    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ip_neigh"
    ip neigh show >> "$HOSTNAME-network_stats_$now/ip_neigh"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    tc -s qdisc >> "$HOSTNAME-network_stats_$now/tc_qdisc"
    if [ "${#MQDEVS[@]}" -gt 0 ]; then
        for MQDEV in "${MQDEVS[@]}"; do
            echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
            tc -s class show dev "$MQDEV" >> "$HOSTNAME-network_stats_$now/tc_class_$MQDEV"
        done
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netstat"
    netstat -s >> "$HOSTNAME-network_stats_$now/netstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/nstat"
    nstat -az >> "$HOSTNAME-network_stats_$now/nstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ss"
    eval "ss $REAL_SS_OPTS" >> "$HOSTNAME-network_stats_$now/ss"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/interrupts"
    cat /proc/interrupts >> "$HOSTNAME-network_stats_$now/interrupts"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softnet_stat"
    cat /proc/net/softnet_stat >> "$HOSTNAME-network_stats_$now/softnet_stat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/vmstat"
    cat /proc/vmstat >> "$HOSTNAME-network_stats_$now/vmstat"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ps"
    ps -alfe >> "$HOSTNAME-network_stats_$now/ps"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/mpstat"
    eval mpstat -A "$DELAY" 1 2>/dev/null >> "$HOSTNAME-network_stats_$now/mpstat" &
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/top"
    top -c -b -n1 >> "$HOSTNAME-network_stats_$now/top"
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/numastat"
    numastat 2>/dev/null >> "$HOSTNAME-network_stats_$now/numastat"
    if [ -e /proc/softirqs ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/softirqs"
        cat /proc/softirqs >> "$HOSTNAME-network_stats_$now/softirqs"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat"
    cat /proc/net/sockstat >> "$HOSTNAME-network_stats_$now/sockstat"
    if [ -e /proc/net/sockstat6 ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sockstat6"
        cat /proc/net/sockstat6 >> "$HOSTNAME-network_stats_$now/sockstat6"
    fi
    echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/netdev"
    cat /proc/net/dev >> "$HOSTNAME-network_stats_$now/netdev"
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}' | awk -F '@' '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV"; ethtool -S "$DEV" >> "$HOSTNAME-network_stats_$now/ethtool_$DEV" 2>/dev/null; done
    for DEV in $(ip a l | grep mtu | awk '{print $2}' | awk -F ":" '{print $1}' | awk -F '@' '{print $1}'); do echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; find /sys/devices/ -type f | grep "/net/$DEV/statistics" | xargs grep . | awk -F "/" '{print $NF}' >> "$HOSTNAME-network_stats_$now/sys_statistics_$DEV"; done
    if [ -e /proc/net/sctp ]; then
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        cat /proc/net/sctp/assocs >> "$HOSTNAME-network_stats_$now/sctp-assocs"
        echo "===== $(date +"%F %T.%N%:z (%Z)") =====" >> "$HOSTNAME-network_stats_$now/sctp-snmp"
        cat /proc/net/sctp/snmp >> "$HOSTNAME-network_stats_$now/sctp-snmp"
    fi
    if [ "$ITERATIONS" -gt 0 ]; then let ITERATIONS-=1; fi
    # Wait till background jobs are finished
    wait
done
#
# monitor.sh ends here
EOF

oc project metrics-debug
oc create configmap monitor-coredns --from-file=monitor-coredns/

# Create modified collect scripts configmap

cat <<'EOF' > monitor-coredns-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: monitor-coredns-scripts
  namespace: metrics-debug
data:
  install-requirements.sh: |
    #!/bin/bash
    yum install conntrack-tools procps-ng ethtool numactl psmisc hostname iproute sysstat net-tools procps-ng util-linux -y
  collect-metrics.sh: |
    #!/bin/bash
    readarray -t COREDNS_PIDS < <(pgrep coredns)
    echo "Gathering metrics ..."
    mkdir /network-metrics
    rm -Rf /network-metrics/*
    echo "Gathering monitor metrics ..."
    cd /network-metrics
    NSENTER_PIDS=()
    CONNTRACK_PIDS=()
    for COREDNS_PID in "${COREDNS_PIDS[@]}"
    do      
       nsenter -n -t $COREDNS_PID bash -c "while true ; do date ; conntrack -L -n ; sleep 5; done" >> conntrack-${COREDNS_PID}.txt &
       CONNTRACK_PIDS+=($!)
       mkdir -p "monitor-${COREDNS_PID}"
       pushd "monitor-${COREDNS_PID}"
       nsenter -n -t $COREDNS_PID bash /monitor/monitor.sh -d 5 -i 120 &
       NSENTER_PIDS+=($!)
       popd
    done
    wait "${NSENTER_PIDS[@]}"
    kill "${CONNTRACK_PIDS[@]}"
    tar -czf /network-metrics.tar.gz /network-metrics
    echo "Done with network metrics collection."
EOF
oc apply -f monitor-coredns-configmap.yaml

# Create modified daemonset (that co-exists with the original one)

cat <<'EOF' > monitor-coredns-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: monitor-coredns-daemonset
  namespace: metrics-debug
  labels:
    app: monitor-coredns-daemonset
spec:
  selector:
    matchLabels:
      app: monitor-coredns-daemonset
  template:
    metadata:
      labels:
        app: monitor-coredns-daemonset
    spec:
      # tolerations:
      ## this toleration is to have the daemonset runnable on master nodes
      ## remove it if your masters can't run pods
      # - key: node-role.kubernetes.io/master
      #  effect: NoSchedule
      hostPID: true
      hostIPC: true
      hostNetwork: true
      containers:
      - name: monitor-coredns-daemonset
        image: fedora:latest
        command:
          - "/bin/bash"
          - "-c"
          - "bash /entrypoint/install-requirements.sh && bash /entrypoint/collect-metrics.sh && sleep infinity"
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: entrypoint
          mountPath: /entrypoint
        - name: monitor
          mountPath: /monitor
        securityContext:
          runAsUser: 0
          runAsGroup: 0
          privileged: true
      volumes:
      - name: entrypoint
        configMap:
          name: monitor-coredns-scripts
      - name: monitor
        configMap:
          name: monitor-coredns
EOF
oc apply -f monitor-coredns-daemonset.yaml

Disclaimer

The IP addresses shown in the article are from the Red Hat test environment and don't include customer data.

SBR

• Shift

Product(s)

• Red Hat OpenShift Container Platform

Components

• cri-o

Category

• Troubleshoot

Tags

• monitoring
• ocp_4
• shift_metrics

---

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.