Troubleshooting the NTP Chrony time service in Red Hat OpenShift Container Platform

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform 4.X

Issue

  • We are seeing warning events being logged indicating possible clock drift.
  • The following message snippet might be visible, indicating clock-drift in the etcd cluster.
"rafthttp/probing_status.go:82","msg":"prober found high clock drift","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE
  • We wish to verify NTP/Chrony time service configuration.
  • We wish to check the current time status of the Nodes.

Resolution

Run the following command from the bastion node, or any client device with Cluster access, to check current NTP/CHRONY configuration, and to view the current time-sync status, for all nodes in the Cluster.

$ for node in $(oc get node -o name); do outfile=ntp-config-status-$(basename $node).log; echo "Dumping capture from $node to $outfile"\
;oc debug $node -- chroot /host bash -c 'date --iso=ns;timedatectl status;chronyc sources;chronyc sourcestats' &> ${outfile}.log;done

Output messages such as the following will be displayed as each node is queried.

Dumping capture from node/master-0.my.local.ocp4.cluster.com.au to ntp-config-status-master-0.my.local.ocp4.cluster.com.au.log
Dumping capture from node/master-1.my.local.ocp4.cluster.com.au to ntp-config-status-master-1.my.local.ocp4.cluster.com.au.log
Dumping capture from node/master-2.my.local.ocp4.cluster.com.au to ntp-config-status-master-2.my.local.ocp4.cluster.com.au.log
Dumping capture from node/worker-0.my.local.ocp4.cluster.com.au to ntp-config-status-worker-0.my.local.ocp4.cluster.com.au.log
Dumping capture from node/worker-1.my.local.ocp4.cluster.com.au to ntp-config-status-worker-1.my.local.ocp4.cluster.com.au.log
Dumping capture from node/worker-2.my.local.ocp4.cluster.com.au to ntp-config-status-worker-2.my.local.ocp4.cluster.com.au.log

Output file contents will display NTP/Chrony configuration, and the current time status.

Node experiencing issues with time-sync

 2023-09-12T20:13:58,247015493+00:00
                   Local time: Tue 2023-09-12 20:13:58 UTC
               Universal time: Tue 2023-09-12 20:13:58 UTC
                     RTC time: Tue 2023-09-12 20:14:18
                    Time zone: UTC (UTC, +0000)
    System clock synchronized: no
              NTP service: active
              RTC in local TZ: no

    MS Name/IP address         Stratum Poll Reach LastRx Last sample

    ===============================================================================
    ^? my_ts_1                  0  10     0     -     +0ns[   +0ns] +/-    0ns
    ^? my_ts_2                  0  10     0     -     +0ns[   +0ns] +/-    0ns
    ^? my_ts_3                  0  10     0     -     +0ns[   +0ns] +/-    0ns
    ^? xx.xxx.xxx.xx            0  10     0     -     +0ns[   +0ns] +/-    0ns

    Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev

    ==============================================================================
    my_ts_1                     0   0     0     +0.000   2000.000     +0ns  4000ms
    my_ts_2                     0   0     0     +0.000   2000.000     +0ns  4000ms
    my_ts_1                     0   0     0     +0.000   2000.000     +0ns  4000ms
    xx.xxx.xxx.xx               0   0     0     +0.000   2000.000     +0ns  4000ms

Node that is syncing successfully 

2023-09-12T20:14:02,620489321+00:00
                   Local time: Tue 2023-09-12 20:14:02 UTC
               Universal time: Tue 2023-09-12 20:14:02 UTC
                     RTC time: Tue 2023-09-12 20:14:02
                    Time zone: UTC (UTC, +0000)
    System clock synchronized: yes
              NTP service: active
              RTC in local TZ: no

    MS Name/IP address         Stratum Poll Reach LastRx Last sample

    ===============================================================================

    ^- my_ts_1                       4   6   377    11  -6243us[-6243us] +/-  221ms
    ^* my_ts_2                       3   6   377    39    +14us[  +32us] +/-  189ms
    ^- my_ts_1                       5   7   377   101    -10ms[  -10ms] +/- 1294ms
    ^- xx.xxx.xxx.xx                 4   6   377     8  -6334us[-6334us] +/- 1271ms

    Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev

    ==============================================================================
    my_ts_1                    6   3   322     +0.717      1.394  -6068us    31us
    my_ts_1                   10   6   582     +0.004      0.391   +170ns    40us
    my_ts_1                    6   3   650     -2.240      1.191    -11ms    46us
    xx.xxx.xxx.xx              7   5   585     -0.258      0.671  -6316us    38us
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.