How to troubleshoot chrony issues?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 7
  • Red Hat Enterprise Linux (RHEL) 8
  • Red Hat Enterprise Linux (RHEL) 9
  • Red Hat Enterprise Linux (RHEL) 10
  • Chrony

Issue

  • How to troubleshoot chrony issues.
  • chrony NTP troubleshooting techniques for accurate and reliable time sync.
  • How to check if chrony clients are synchronizing correctly with NTP servers.

Resolution

  • Chrony is composed of two main components:

    • chronyd: A daemon that operates in user space to synchronize and maintain the accuracy of the system clock.

    • chronyc: A command-line utility used to monitor the performance of chronyd and modify its operating parameters while it is running.
      The chronyd daemon can be monitored and managed using the chronyc utility. This utility provides an interactive command prompt, allowing users to execute various commands to:

      • Query the current state of chronyd.
      • Adjust its configuration dynamically.
  • By default, chronyd only accepts commands from a local instance of chronyc. However, it can be configured to allow monitoring commands from remote hosts. When enabling remote access, it is crucial to restrict access to authorized hosts to ensure security.

  • Key Points

    • Ensure you configure at least 4 functional NTP servers: 3 for verification and 1 as a backup.
    • Chrony provides several commands to monitor and troubleshoot time synchronization. Below is a list of commonly used commands
  • To get the status of NTP sources (online/offline).

     # chronyc activity
    
  • To get detailed information about configured NTP servers.

      # chronyc ntpdata
    
  • To get a list of NTP sources and their synchronization states.

     # chronyc -n sources -v
    
  • To get statistics for each NTP source.

      # chronyc -n sourcestats -v
    
  • To get the system clock synchronization status.

      # chronyc -n tracking  
    
  • To test NTP server reachability.

      # chronyd -Q 'server <NTPserver-IP> iburst'
    
  • To get the status of the chronyd service.

      # systemctl status chronyd
    
  • To manage time settings and check NTP synchronization.

      # timedatectl
    
  • This command provides an overview of the activity of the chronyd daemon, including the number of NTP sources that are online, offline, or experiencing issues. It helps determine whether chronyd is actively communicating with its configured NTP servers.

        # chronyc activity
    200 OK
    4 sources online
    0 sources offline
    0 sources doing burst (return to online)
    0 sources doing burst (return to offline)
    0 sources with unknown address
    
  • This is the output of the chronyc -n sources -v command, which provides detailed information about the configured NTP sources.

    
         # chronyc -n sources -v
         210 Number of sources = 4
    
         .-- Source mode  '^' = server, '=' = peer, '#' = local clock.
        / .- Source state '*' = current synced, '+' = combined , '-' = not combined,
       | /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
       ||                                                 .- xxxx [ yyyy ] +/- zzzz
       ||      Reachability register (octal) -.           |  xxxx = adjusted offset,
       ||      Log2(Polling interval) --.      |          |  yyyy = measured offset,
       ||                                \     |          |  zzzz = estimated error.
       ||                                 |    |           \
       MS Name/IP address         Stratum Poll Reach LastRx Last sample
       ===============================================================================
       ^+ 200.160.0.8                   2   6   377    60  -7418us[-6893us] +/-   21ms
       ^+ 201.73.152.122                2   6   377    58  -9712us[-9187us] +/-  169ms
       ^+ 200.189.40.8                  2   6   377    60    +36ms[  +37ms] +/-   83ms
       ^* 200.160.7.186                 1   6   377    56  -6725us[-6200us] +/-  176ms
    
    
  • The columns are as follows:

         M
             This indicates the mode of the source. ^ means a server, = means a peer and # indicates a locally connected reference clock.
         S
             This column indicates the selection state of the source.
    
             •   * indicates the best source which is currently selected for synchronisation.
             •   + indicates other sources selected for synchronisation, which are combined with the best source.
             •   - indicates a source which is considered to be selectable for synchronisation, but not currently selected.
             •   x indicates a source which chronyd thinks is a falseticker (i.e. its time is inconsistent with a majority of other sources, or sources specified with the trust option).
             •   ~ indicates a source whose time appears to have too much variability.
             •   ? indicates a source which is not considered to be selectable for synchronisation for other reasons (e.g. unreachable, not synchronised, or does not have enough measurements).
    
    • The second character on the server lines should always be a '-', '+', or '*' after running for a while, otherwise the server is having trouble.
    • If Reach is not 377 and chrony has been running for a while then access to that server is having connection issues.
    • If LastRx is higher than Poll time (2^X seconds) then the server is having connection issues and it should be reflected in Reach as well.
  • The chronyc -n sourcestats -v command provides detailed statistics about the performance of the time sources being used by Chrony.

      # chronyc -n sourcestats -v
      210 Number of sources = 4
                                   .- Number of sample points in measurement set.
                                  /    .- Number of residual runs with same sign.
                                 |    /    .- Length of measurement set (time).
                                 |   |    /      .- Est. clock freq error (ppm).
                                 |   |   |      /           .- Est. error in freq.
                                 |   |   |     |           /         .- Est. offset.
                                 |   |   |     |          |          |   On the -.
                                 |   |   |     |          |          |   samples. \
                                 |   |   |     |          |          |             |
      Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
      ==============================================================================
      200.160.0.8                15   7   718      2.197     10.669  -7221us  2248us
      201.73.152.122             13   7   721     -0.203      2.476    -10ms   463us
      200.189.40.8               15   8   718      0.729      9.211    +36ms  2088us
      200.160.7.186               8   6   522     -1.902      7.489  -6332us   606us
    
  • The chronyc -n tracking command provides detailed information about the system's current synchronization status with an NTP server.

      # chronyc -n tracking
      Reference ID    : 200.160.7.186 (200.160.7.186)
      Stratum         : 2
      Ref time (UTC)  : Wed Nov  5 18:45:23 2014
      System time     : 0.000799330 seconds fast of NTP time
      Last offset     : 0.000525316 seconds
      RMS offset      : 0.015844775 seconds
      Frequency       : 0.971 ppm fast
      Residual freq   : -0.002 ppm
      Skew            : 2.283 ppm
      Root delay      : 0.349031 seconds
      Root dispersion : 0.000644 seconds
      Update interval : 65.6 seconds
      Leap status     : Normal
    
  • The command chronyd -Q 'server <ntp.server.ip> iburst' is used to query the Chrony NTP daemon for synchronization status without making any adjustments to the system clock.

    • The -Q option allows you to check the system clock's accuracy without modifying it.
    • This is useful for testing or diagnostics when you don't want Chrony to adjust the clock.
      # chronyd -Q 'server 200.160.0.8 iburst'
      2024-10-24T03:02:19Z chronyd version 4.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +DEBUG)
      2024-10-24T03:02:19Z Disabled control of system clock
      2024-10-24T03:02:23Z System clock wrong by -0.021074 seconds (ignored)
      2024-10-24T03:02:23Z chronyd exiting
    
  • The output of the timedatectl command provides a comprehensive overview of the system's time and date settings.

         # timedatectl
          Local time: Wed 2014-11-05 13:48:16 EST
      Universal time: Wed 2014-11-05 18:48:16 UTC
            RTC time: Wed 2014-11-05 18:48:15
            Timezone: America/New_York (EST, -0500)
         NTP enabled: yes
    NTP synchronized: no
     RTC in local TZ: no
          DST active: no
     Last DST change: DST ended at
                      Sun 2014-11-02 01:59:59 EDT
                      Sun 2014-11-02 01:00:00 EST
     Next DST change: DST begins (the clock jumps one hour forward) at
                      Sun 2015-03-08 01:59:59 EST
                      Sun 2015-03-08 03:00:00 EDT
    
  • Additional References:

  • RHEL 7 Configuring NTP Using the chrony Suite

  • RHEL 8 Configuring basic system settings

  • RHEL 9 Configuring time synchronization

  • RHEL 10 Configuring time synchronization

  • /usr/share/doc/chrony*/

Diagnostic Steps

  • Verify that chronyd is running and active:

     # systemctl status -l chronyd
    
  • If chronyd is unable to communicate with any servers, gather detailed logs for further analysis:

      # systemctl stop chronyd
      # strace -fttTvyys 4096 -o chronyd.strace chronyd -d -d -q > chronyd.output 2>&1
      # systemctl start chronyd
    
  • If you suspect network issues, capture NTP traffic packets for additional investigation:

     # tcpdump -s0 port 123 -vvv -i <NIC> -w port123.pcap
    
SBR
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.