How to troubleshoot chrony issues?
Environment
- Red Hat Enterprise Linux (RHEL) 7
- Red Hat Enterprise Linux (RHEL) 8
- Red Hat Enterprise Linux (RHEL) 9
- Red Hat Enterprise Linux (RHEL) 10
- Chrony
Issue
- How to troubleshoot
chronyissues. - chrony
NTPtroubleshooting techniques for accurate and reliabletimesync. - How to check if
chronyclients aresynchronizingcorrectly withNTPservers.
Resolution
-
Chrony is composed of two main components:
-
chronyd: A daemon that operates in user space to
synchronizeand maintain theaccuracyof the system clock. -
chronyc: A command-line utility used to monitor the performance of
chronydand modify its operating parameters while it is running.
Thechronyddaemon can be monitored and managed using thechronycutility. This utility provides an interactive command prompt, allowing users to execute various commands to:- Query the current state of
chronyd. - Adjust its configuration dynamically.
- Query the current state of
-
-
By default, chronyd only accepts commands from a local instance of
chronyc. However, it can be configured to allow monitoring commands from remote hosts. When enabling remote access, it is crucial to restrict access to authorized hosts to ensure security. -
Key Points
- Ensure you configure at least 4 functional NTP servers: 3 for verification and 1 as a backup.
- Chrony provides several commands to monitor and troubleshoot time synchronization. Below is a list of commonly used commands
-
To get the status of
NTPsources (online/offline).# chronyc activity -
To get detailed information about configured
NTPservers.# chronyc ntpdata -
To get a list of
NTPsources and theirsynchronizationstates.# chronyc -n sources -v -
To get statistics for each
NTPsource.# chronyc -n sourcestats -v -
To get the system
clocksynchronization status.# chronyc -n tracking -
To test
NTPserver reachability.# chronyd -Q 'server <NTPserver-IP> iburst' -
To get the status of the
chronydservice.# systemctl status chronyd -
To manage
timesettings and checkNTPsynchronization.# timedatectl -
This command provides an
overviewof the activity of thechronyddaemon, including the number ofNTPsources that areonline,offline, orexperiencingissues. It helps determine whetherchronydis actively communicating with its configuredNTPservers.# chronyc activity 200 OK 4 sources online 0 sources offline 0 sources doing burst (return to online) 0 sources doing burst (return to offline) 0 sources with unknown address -
This is the output of the
chronyc -n sources -vcommand, which provides detailed information about the configuredNTPsources.# chronyc -n sources -v 210 Number of sources = 4 .-- Source mode '^' = server, '=' = peer, '#' = local clock. / .- Source state '*' = current synced, '+' = combined , '-' = not combined, | / '?' = unreachable, 'x' = time may be in error, '~' = time too variable. || .- xxxx [ yyyy ] +/- zzzz || Reachability register (octal) -. | xxxx = adjusted offset, || Log2(Polling interval) --. | | yyyy = measured offset, || \ | | zzzz = estimated error. || | | \ MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^+ 200.160.0.8 2 6 377 60 -7418us[-6893us] +/- 21ms ^+ 201.73.152.122 2 6 377 58 -9712us[-9187us] +/- 169ms ^+ 200.189.40.8 2 6 377 60 +36ms[ +37ms] +/- 83ms ^* 200.160.7.186 1 6 377 56 -6725us[-6200us] +/- 176ms -
The columns are as follows:
M This indicates the mode of the source. ^ means a server, = means a peer and # indicates a locally connected reference clock. S This column indicates the selection state of the source. • * indicates the best source which is currently selected for synchronisation. • + indicates other sources selected for synchronisation, which are combined with the best source. • - indicates a source which is considered to be selectable for synchronisation, but not currently selected. • x indicates a source which chronyd thinks is a falseticker (i.e. its time is inconsistent with a majority of other sources, or sources specified with the trust option). • ~ indicates a source whose time appears to have too much variability. • ? indicates a source which is not considered to be selectable for synchronisation for other reasons (e.g. unreachable, not synchronised, or does not have enough measurements).- The second character on the server lines should always be a '-', '+', or '*' after running for a while, otherwise the server is having trouble.
- If Reach is not 377 and chrony has been running for a while then access to that server is having connection issues.
- If LastRx is higher than Poll time (2^X seconds) then the server is having connection issues and it should be reflected in
Reachas well.
-
The
chronyc -n sourcestats -vcommand providesdetailedstatistics about theperformanceof the timesourcesbeing used byChrony.# chronyc -n sourcestats -v 210 Number of sources = 4 .- Number of sample points in measurement set. / .- Number of residual runs with same sign. | / .- Length of measurement set (time). | | / .- Est. clock freq error (ppm). | | | / .- Est. error in freq. | | | | / .- Est. offset. | | | | | | On the -. | | | | | | samples. \ | | | | | | | Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev ============================================================================== 200.160.0.8 15 7 718 2.197 10.669 -7221us 2248us 201.73.152.122 13 7 721 -0.203 2.476 -10ms 463us 200.189.40.8 15 8 718 0.729 9.211 +36ms 2088us 200.160.7.186 8 6 522 -1.902 7.489 -6332us 606us -
The
chronyc -n trackingcommand provides detailed information about thesystem'scurrentsynchronizationstatus with anNTPserver.# chronyc -n tracking Reference ID : 200.160.7.186 (200.160.7.186) Stratum : 2 Ref time (UTC) : Wed Nov 5 18:45:23 2014 System time : 0.000799330 seconds fast of NTP time Last offset : 0.000525316 seconds RMS offset : 0.015844775 seconds Frequency : 0.971 ppm fast Residual freq : -0.002 ppm Skew : 2.283 ppm Root delay : 0.349031 seconds Root dispersion : 0.000644 seconds Update interval : 65.6 seconds Leap status : Normal -
The command
chronyd -Q 'server <ntp.server.ip> iburst'is used to query the Chrony NTP daemon for synchronization status without making any adjustments to the system clock.- The
-Qoption allows you to check the systemclock'saccuracy withoutmodifyingit. - This is useful for
testingordiagnosticswhen you don't wantChronyto adjust the clock.
# chronyd -Q 'server 200.160.0.8 iburst' 2024-10-24T03:02:19Z chronyd version 4.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +DEBUG) 2024-10-24T03:02:19Z Disabled control of system clock 2024-10-24T03:02:23Z System clock wrong by -0.021074 seconds (ignored) 2024-10-24T03:02:23Z chronyd exiting - The
-
The output of the
timedatectlcommand provides acomprehensiveoverview of thesystem'stimeanddatesettings.# timedatectl Local time: Wed 2014-11-05 13:48:16 EST Universal time: Wed 2014-11-05 18:48:16 UTC RTC time: Wed 2014-11-05 18:48:15 Timezone: America/New_York (EST, -0500) NTP enabled: yes NTP synchronized: no RTC in local TZ: no DST active: no Last DST change: DST ended at Sun 2014-11-02 01:59:59 EDT Sun 2014-11-02 01:00:00 EST Next DST change: DST begins (the clock jumps one hour forward) at Sun 2015-03-08 01:59:59 EST Sun 2015-03-08 03:00:00 EDT -
Additional References:
-
/usr/share/doc/chrony*/
Diagnostic Steps
-
Verify that
chronydis running and active:# systemctl status -l chronyd -
If
chronydis unable to communicate with any servers, gather detailed logs for further analysis:# systemctl stop chronyd # strace -fttTvyys 4096 -o chronyd.strace chronyd -d -d -q > chronyd.output 2>&1 # systemctl start chronyd -
If you suspect
networkissues, captureNTPtraffic packets foradditionalinvestigation:# tcpdump -s0 port 123 -vvv -i <NIC> -w port123.pcap
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.