SUSE Linux Real Time Hardware Testing and Latency

Install the real-time kernel package:

> sudo zypper install kernel-rt

Reboot your system.

Select the kernel-rt kernel from the boot menu to activate the real-time capabilities.

Access your server BIOS or firmware settings.

Disable power-management features that introduce variable latency, since these are common sources of SMIs and scheduling jitter. Menu labels vary by vendor, but look for and disable:

Reduce memory patrol scrubbing to its lowest available rate or disable it, since scrubbing cycles can introduce firmware-level latency spikes.

Start the detector with a specified threshold:

> sudo hwlatdetect --threshold=10

This command monitors system management interrupts (SMIs) with a minimum latency threshold of 10 microseconds.

Check the measured latencies.

A run against a tuned, real-time-capable system produces output similar to the following:

> sudo hwlatdetect --threshold=10 --duration=120
hwlatdetect:  test duration 120 seconds
detector: hwlat
parameters:
Latency threshold: 10us
Sample window:     1000000us
Sample width:      500000us
Non-sampling period:  500000us
Output File:       None
                
Starting test
test finished
Max Latency: 3us
Samples recorded: 0
Samples exceeding threshold: 0

Max Latency reports the largest gap observed between the CPU Time Stamp Counter reads, in microseconds. Samples exceeding threshold counts how many of those gaps were larger than the value you passed with --threshold.

A hardware platform suitable for real-time workloads should report Samples exceeding threshold: 0 over the full test duration, with Max Latency below the threshold you set. If Samples exceeding threshold is greater than zero, or Max Latency approaches or exceeds your threshold, the platform firmware or hardware is introducing delays your application deadline cannot tolerate. Treat this as a hardware or firmware-level failure regardless of what cyclictest reports later, and use a different hardware platform.

Install the rt-tests package if it is not already present:

> sudo zypper install rt-tests

Determine your test parameters from the cyclictest options listed in Section 6, “hwlatdetect and cyclictest command-line options”.

Start cyclictest. For example, to run 1000 cycles with a latency histogram tracked up to 1000 microseconds, execute:

> sudo cyclictest --loops=1000 --histogram=1000

By default, cyclictest writes the histogram data to standard output after the run. To save it to a file for later analysis instead, redirect the output:

> sudo cyclictest --loops=1000 --histogram=1000 > histogram

Analyze the recorded minimal, actual, average and maximum latencies.

A typical result on a multi-core system looks like:

cyclictest --loops=1000 --histogram=1000
policy: fifo: loadavg: 0.12 0.08 0.03 1/342 4821
            
T: 0 ( 4822) P:99 I:1000 C:  1000 Min:      4 Act:    6 Avg:    7 Max:      31
T: 1 ( 4823) P:99 I:1500 C:   667 Min:      3 Act:    5 Avg:    6 Max:      24

Each line reports one measurement thread (T:), its process ID, scheduling priority (P:) and interval in microseconds (I:). Four values matter for evaluation: Min, Act (the latency of the most recent cycle), Avg and Max. All four are reported in microseconds by default. Use --nsec if you need nanosecond resolution.

Max is the value to compare against the deadline of your application: it is the single worst-case scheduling latency observed during the run. As a rough order of magnitude, well-tuned real-time systems generally report Max values in the tens of microseconds. Results climbing into the hundreds of microseconds suggest an unresolved source of interference, and results in the thousands of microseconds indicate the system is not meeting real-time deadlines at all. These figures vary by hardware, workload and kernel configuration. Compare this Max value against the Max Latency reported by hwlatdetect in Section 4, “Evaluating hardware latencies with the hardware latency detector”. Per Section 1, “Understanding real-time hardware latency testing”, cyclictest is expected to report somewhat higher latencies, since it includes scheduling and interrupt handling overhead that hwlatdetect does not measure.

The Max value from cyclictest includes hardware latency plus kernel scheduling overhead. The Max Latency value from hwlatdetect does not. So cyclictest is expected to report a higher number. The two tools are not measuring the same thing.

What matters is the size of the gap. A small gap is normal software overhead. A gap of orders of magnitude points to a real problem in the operating system, not the hardware.