How to measure OS and hardware latency on isolated CPUs?
Environment
- Red Hat Enterprise Linux 8
- Red Hat Enterprise Linux 7
Issue
- When the low-latency environment is set and/or CPUs are isolated, the question is how good the isolation is?
- Are there any tools to measure the isolation and how to collect statistics of undesired "interruptions", latency, unwanted timer interrupts on isolated CPU cores?
Resolution
Disclaimer:Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.
There are set of testing tools that can measure latency that useful workload can experience when being run on isolated CPUs.
oslat (OS latency detector)
Content from github.com is not included.Content from github.com is not included.https://github.com/xzpeter/oslat
Oslat spawns working threads that spin in busy loops with empty or non-empty workload performing RDTSC (or another free running counter if implementation is available for architecture) sequence on specified cores.
Interruptions of RDTSC sequence are collected and reported.
Example of running oslat for 300 seconds on isolated cores under RHEL8 system tuned for low-latency:
[root@machine oslat]# ./oslat --cpu-list 1,2,3,5,6,7 --rtprio 1 --runtime 300
Version: v0.1.7
Total runtime: 300 seconds
Thread priority: SCHED_FIFO:1
CPU list: 1,2,3,5,6,7
CPU for main thread: 0
Workload: no
Workload mem: 0 (KiB)
Preheat cores: 6
Pre-heat for 1 seconds...
Test starts...
Test completed.
Core: 1 2 3 5 6 7
CPU Freq: 3300 3300 3289 3289 3289 3289 (Mhz)
001 (us): 4419469990 4417661547 4417661784 4430052740 4417662676 4417662676
002 (us): 0 0 0 0 0 0
003 (us): 0 0 0 0 0 0
004 (us): 1 0 0 0 0 0
005 (us): 0 0 0 0 0 0
006 (us): 0 0 0 0 0 0
007 (us): 1 4 1 0 0 0
008 (us): 3 0 1 0 0 0
009 (us): 0 0 0 0 0 0
010 (us): 0 0 0 0 0 0
011 (us): 0 0 0 0 0 0
012 (us): 0 0 0 0 0 0
013 (us): 0 0 0 0 0 0
014 (us): 0 0 0 0 0 0
015 (us): 0 0 0 0 0 0
016 (us): 0 0 0 0 0 0
017 (us): 0 0 0 0 0 0
018 (us): 0 0 0 0 0 0
019 (us): 0 0 0 0 0 0
020 (us): 0 0 0 0 0 0
021 (us): 0 0 0 0 0 0
022 (us): 0 0 0 0 0 0
023 (us): 0 0 0 0 0 0
024 (us): 0 0 0 0 0 0
025 (us): 0 0 0 0 0 0
026 (us): 0 0 0 0 0 0
027 (us): 0 0 0 0 0 0
028 (us): 0 0 0 0 0 0
029 (us): 0 0 0 0 0 0
030 (us): 0 0 0 0 0 0
031 (us): 0 0 0 0 0 0
032 (us): 0 0 0 0 0 0 (including overflows)
Minimum: 1 1 1 1 1 1 (us)
Average: 1.000 1.000 1.000 1.000 1.000 1.000 (us)
Maximum: 8 7 8 1 1 1 (us)
Max-Min: 7 6 7 0 0 0 (us)
Duration: 299.275 299.275 300.276 300.276 300.276 300.276 (sec)
The same under RHEL7:
[root@machine oslat]# ./oslat --cpu-list 1,2,3,5,6,7 --rtprio 1 --runtime 300
Version: v0.1.7
Total runtime: 300 seconds
Thread priority: SCHED_FIFO:1
CPU list: 1,2,3,5,6,7
CPU for main thread: 0
Workload: no
Workload mem: 0 (KiB)
Preheat cores: 6
Pre-heat for 1 seconds...
Test starts...
Test completed.
Core: 1 2 3 5 6 7
CPU Freq: 3289 3300 3289 3289 3289 3289 (Mhz)
001 (us): 4294238136 4294919999 4296176306 4295246720 4296200531 4295384992
002 (us): 0 0 0 0 0 0
003 (us): 0 0 0 0 0 0
004 (us): 384 87 93 1 3 3
005 (us): 70 11 11 6 9 5
006 (us): 70 138 123 0 0 0
007 (us): 104 42 41 97 99 95
008 (us): 190 65 85 183 187 192
009 (us): 17 8 10 19 14 12
010 (us): 78 23 30 5 8 8
011 (us): 62 76 62 56 60 49
012 (us): 41 133 131 205 198 216
013 (us): 46 10 6 2 7 5
014 (us): 96 3 3 4 3 2
015 (us): 15 0 0 1 4 4
016 (us): 0 0 0 0 1 1
017 (us): 1 0 0 3 0 0
018 (us): 4 4 5 3 3 3
019 (us): 6 1 1 1 0 1
020 (us): 2 0 0 8 2 2
021 (us): 7 0 0 3 1 1
022 (us): 4 1 0 2 0 0
023 (us): 5 0 1 0 0 0
024 (us): 0 0 0 0 0 0
025 (us): 0 0 0 0 0 0
026 (us): 0 0 0 1 0 0
027 (us): 0 0 0 0 1 1
028 (us): 0 0 0 0 0 0
029 (us): 0 0 0 0 0 0
030 (us): 0 0 0 0 0 0
031 (us): 0 0 0 0 0 0
032 (us): 0 0 0 0 0 0 (including overflows)
Minimum: 1 1 1 1 1 1 (us)
Average: 1.000 1.000 1.000 1.000 1.000 1.000 (us)
Maximum: 23 22 23 26 27 27 (us)
Max-Min: 22 21 22 25 26 26 (us)
Duration: 300.276 299.275 300.276 300.276 300.276 300.276 (sec)
sysjitter
Content from gitlab.com is not included.Content from gitlab.com is not included.https://gitlab.com/0xeafffffe/sysjitter_mod
It is the predecessor of oslat and operates in a similar way. Sysjitter reports stats differently, can save some detailed stats in files but overall has less features than oslat.
Example of running sysjitter on isolated cores for 300 seconds with threshold of 700 ns (rdtsc sequence times higher than threshold will be be considered as interruption/latency) under RHEL8 system tuned for low-latency:
[root@machine sysjitter_mod]# ./sysjitter --verbose --cores 1,2,3,5,6,7 --runtime 300 700
core_i: 1 2 3 5 6 7
threshold(ns): 700 700 700 700 700 700
cpu_mhz: 3289 3289 3289 3289 3289 3300
runtime(ns): 300278293628 300278293702 300278293772 300278293839 300278293719 299277375983
runtime(s): 300.278 300.278 300.278 300.278 300.278 299.277
int_n: 3 3 3 1 0 1
int_n_per_sec: 0.010 0.010 0.010 0.003 0.000 0.003
int_min(ns): 869 876 859 21561 0 7936
int_median(ns): 2250 2210 2284 21561 0 7936
int_mean(ns): 3722 3698 3729 21561 0 7936
int_90(ns): 8046 8010 8043 21561 0 7936
int_99(ns): 8046 8010 8043 21561 0 7936
int_999(ns): 8046 8010 8043 21561 0 7936
int_9999(ns): 8046 8010 8043 21561 0 7936
int_99999(ns): 8046 8010 8043 21561 0 7936
int_max(ns): 8046 8010 8043 21561 0 7936
int_total(ns): 11167 11096 11187 21561 0 7936
int_min(us): 0.87 0.88 0.86 21.56 0.00 7.94
int_max(us): 8.05 8.01 8.04 21.56 0.00 7.94
int_total(us): 11.17 11.10 11.19 21.56 0.00 7.94
int_total(%): 0.000 0.000 0.000 0.000 0.000 0.000
frc_start: 25f075bb046f 25f075bb0401 25f075bb0388 25f075bb0162 25f075bb02ee 25f075bb02e3
frc_stop: 26d66830ac50 26d66830acd4 26d66830ad42 26d66830abf8 26d66830abf8 26d668312bac
The same under RHEL7:
[root@localhost sysjitter_mod]# ./sysjitter --verbose --cores 1,2,3,5,6,7 --runtime 300 700
core_i: 1 2 3 5 6 7
threshold(ns): 700 700 700 700 700 700
cpu_mhz: 3289 3289 3289 3300 3289 3300
runtime(ns): 300277330100 300277330163 300277330066 299276405700 300277330240 299276415586
runtime(s): 300.277 300.277 300.277 299.276 300.277 299.276
int_n: 301 301 301 301 301 302
int_n_per_sec: 1.002 1.002 1.002 1.006 1.002 1.009
int_min(ns): 4404 4411 4397 5863 6505 4050
int_median(ns): 4745 4678 4775 13126 13408 13116
int_mean(ns): 8115 8005 8254 13019 13451 13103
int_90(ns): 13494 13488 13658 13446 13729 13446
int_99(ns): 18444 17909 17769 15020 17100 18513
int_999(ns): 21036 20220 20153 22900 30287 26536
int_9999(ns): 21036 20220 20153 22900 30287 26536
int_99999(ns): 21036 20220 20153 22900 30287 26536
int_max(ns): 21036 20220 20153 22900 30287 26536
int_total(ns): 2442816 2409688 2484672 3918983 4048849 3957330
int_min(us): 4.40 4.41 4.40 5.86 6.51 4.05
int_max(us): 21.04 20.22 20.15 22.90 30.29 26.54
int_total(us): 2442.82 2409.69 2484.67 3918.98 4048.85 3957.33
int_total(%): 0.001 0.001 0.001 0.001 0.001 0.001
frc_start: 5f3bb7aabce 5f3bb7aaa9a 5f3bb7aab3f 5f3bb7aa9f5 5f3bb7aa832 5f3bb7aaa2c
frc_stop: 6d9adbff89a 6d9adbff837 6d9adbff79d 6d9adbff72f 6d9adbff6cc 6d9adc076d8
SBR
Product(s)
Components
Category
Tags
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.