Erratic time issues on VMware based RHEL guests running ntpd/chronyd

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) or Red Hat Core OS (RHCOS) running as a virtual machine (VM) on a VMware host.
  • ntp
  • chrony
  • NTP (Network Time Protocol)

Issue

  • Erratic time issues on VMware based RHEL/RHCOS guests running ntpd/chronyd
  • Why is time on my RHEL guests under VMware behaving erratically?
  • Why is time on my RHEL guest under VMware skipping randomly?
  • How can I check if VMware time sync is disabled?
  • VMware has a functionality to enable time synchronization from the host system to VMs. This is typically undesirable and not recommended as it can interfere with ntpd/chronyd. What are the steps to ensure this functionality is disabled?
  • We have some virtual machines (running on VMware ESXi hosts) having problem with time synchronization by using ntpd/chronyd. How do we fix it?
  • VMWare recommends keeping one-off time synchronization enabled. Should we keep it enabled/disabled if ntpd/chronyd is running?
  • In Openshift nodes running on top of VMWare, the NodeClockNotSynchronising alert is firing after performing VMotion on clusters configured with NTP/chrony.

Resolution

If the VMware based RHEL guests are experiencing erratic time issues our suggestion to troubleshoot the problem is to disable the two types of VMware time synchronization (periodic and one-off time sync) to isolate the issue and use only a single time synchronization program (chronyd is preferred).

Disabling periodic time synchronization

Periodic time synchronization can be checked by using the command vmware-toolbox-cmd timesync status and disabled in one of three ways:

  • Run the command vmware-toolbox-cmd timesync disable from the guest
  • Deselect "Time synchronization between the virtual machine and the host operating system" in the VMware Tools toolbox GUI of the guest operating system.
  • Select or by add the line tools.syncTime = "0" to the .vmx file

Disabling one-off time synchronization

The following options need to be added to the .vmx file for each system:

time.synchronize.continue = "FALSE"
time.synchronize.restore = "FALSE"
time.synchronize.resume.disk = "FALSE"
time.synchronize.shrink = "FALSE"
time.synchronize.tools.startup = "FALSE"
time.synchronize.tools.enable = "FALSE"
time.synchronize.resume.host = "FALSE"

For more details, please contact the vendor.

Root Cause

VMware has two types of time synchronization:

Periodic time synchronization
Only a single time synchronization program should be disciplining the time of an operating system. Therefore, when using NTP in the guest, you must ensure that VMware Tools periodic time synchronization is disabled (the default setting for VMware virtual machines).

One-off time synchronization
Certain virtual machine life-cycle events can cause guest clock to become incorrect (typically lag behind real wall clock time). VMware Tools recognizes the lag, and synchronizes guest operating system time to that of the host. This capability is turned on by default.
Time is re-synchronized when you migrate the virtual machine using vMotion, take a snapshot, restore to a snapshot, shrink the virtual disk, resume a virtual machine after being suspended, or restart the tools service in the virtual machine (including rebooting the virtual machine).

IMPORTANT: Since one-off time synchronization relies on the time in the host operating system as a reference, it is important that host system time is kept accurate using time synchronization software according to best practices for that host.

NOTE: When a VM is migrated from one host to another and one-off sync is enabled, the VM will change the system clock according to the time from the new host. It's possible to see messages like "Time has been changed" or "Forward time jump detected". In order to avoid time jumps that may cause problems to the applications, it's recommended to disable one-off time sync and use only chronyd, which will adjust the system clock gradually without clock jumps.

For more information, see:
Content from kb.vmware.com is not included."Migrating virtual machine may cause guest operating system clock to fall behind real time"

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Diagnostic Steps

  • Review the output of vmware-toolbox-cmd timesync status
  • Review the .vmx file for this system
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.