Time drift issues in Red Hat Enterprise Linux guests in Microsoft Windows Hyper-V virtualization environment

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 5.4, 5.5 (64-bit and 32-bit) virtualization guest, or
  • Red Hat Enterprise Linux (RHEL) 4.8 (64-bit and 32-bit) virtualization guest
  • Microsoft Windows Hyper-V virtualization host environment

Issue

  • Time is skipping or drifting on Red Hat Enterprise Linux 5 guests in Microsoft Windows Hyper-V.
  • Since kernel 2.6.18-194.8.1.el5 we observe large clock drift on RHEL 5.5  / x86_64 guests running on Hyper-V hosts. This clock drift is not  happening on older kernels (up to 2.6.18-194.3.1.el5), nor on i386 machines.
  • Since kernel 2.6.9-89.0.26.EL we observe large clock drift on RHEL 4.8 / x86_64 guests running on Hyper-V hosts.
  • We observed that guests running in RHEV 2.x that was exported and then imported in to RHEV 3.1.1 had an incorrect time by 5 to 10 hours. We had to set date manually to get NTP to adjust it correctly. Is this an expected behavior or a bug?

Resolution

Product documentation

Minimum kernel versions

  • For RHEL5 virtualization guests, ensure that you are running kernel 2.6.18-194.26.1.el5 or later.
  • For RHEL4 virtualization guests, ensure that you are running kernel 2.6.9-89.33.1 or later.
  • Older kernel versions have known issues in their time keeping code that can be exposed when run in a Hyper-V virtualization guest.

Adjust kernel parameters

The following kernel parameter settings have been reported to reduce time drift:

  • Red Hat Enterprise Linux 5 64bit guest: "divider=10 clock=pmtmr"
  • Red Hat Enterprise Linux 5 32bit guest: "divider=10 notsc"
  • Red Hat Enterprise Linux 4 64bit guest: "divider=10 notsc"
  • Red Hat Enterprise Linux 4 32bit guest: "divider=10 notsc"
  • Other kernel parameters that may bring relief are documented in the KVM guest timing management chapter of the Red Hat Enterprise Linux 5 Virtualization Guide. Note that these settings were tested with the Kernel Virtual Machine hypervisor and may not always work with the Hyper-V hypervisor.

Adjust NTP configuration

If running ntpd, in /etc/ntp.conf :

  1. Ensure the top line is "tinker panic 0". This causes nptd to sync time even when it would otherwise consider the time difference too large to correct.

  2. Comment out any lines that enable the use of a local clock source:

    # server 127.127.1.0  
# fudge 127.127.1.0 stratum 10

Adjust Hyper-V configuration

Root Cause

  • The default kernel parameters are not appropriate for proper time handling of Red Hat Enterprise Linux guests in Microsoft Windows Hyper-V virtualization environments.
  • Accurate time keeping within virtualization guest is generally challenging as virtualization guests are not in (full) control of the hardware, and the hypervisor can preempt guest code and certain instructions take longer in a virtualization environment than on hardware as they require emulation under virtualization.

Diagnostic Steps

Establish basic context of the issue

  • Is the time drift consistent in one direction, or does it vary?
  • If the time drift's direction is consistent,
    • What direction is it in: the guest time is running ahead of or behind host/wall time?
    • At what rate is it drifting? E.g. how many seconds additional drift in guest time in an hour of host/wall time?
  • Does the drift/rate depend on the guest's workload? E.g. does it drift only when the system is busy or also when it is idle?
  • Does the drift/rate depend on the host's workload?

Comments

  • The time drift issues discussed here occur regardless of whether the Microsoft Integration Components are installed in the RHEL Guests.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.