1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
5 Low Latency Tunning Suggestion
6 ==============================
8 The correct configuration is critical for improving the NFV
9 performance/latency.Even working on the same codebase, configurations can cause
10 wildly different performance/latency results.
12 There are many combinations of configurations, from hardware configuration to
13 Operating System configuration and application level configuration. And there
14 is no one simple configuration that works for every case. To tune a specific
15 scenario, it's important to know the behaviors of different configurations and
18 Platform Configuration
19 ----------------------
21 Some hardware features can be configured through firmware interface(like BIOS)
22 but others may not be configurable (e.g. SMI on most platforms).
24 * **Power management:**
25 Most power management related features save power at the
26 expensive of latency. These features include: IntelĀ®Turbo Boost Technology,
27 Enhanced IntelĀ®SpeedStep, Processor C state and P state. Normally they
28 should be disabled but, depending on the real-time application design and
29 latency requirements, there might be some features that can be enabled if
30 the impact on deterministic execution of the workload is small.
32 * **Hyper-Threading:**
33 The logic cores that share resource with other logic cores can introduce
34 latency so the recommendation is to disable this feature for realtime use
37 * **Legacy USB Support/Port 60/64 Emulation:**
38 These features involve some emulation in firmware and can introduce random
39 latency. It is recommended that they are disabled.
41 * **SMI (System Management Interrupt):**
42 SMI runs outside of the kernel code and can potentially cause
43 latency. It is a pity there is no simple way to disable it. Some vendors may
44 provide related switches in BIOS but most machines do not have this
47 Operating System Configuration
48 ------------------------------
51 To achieve deterministic latency, dedicated CPUs should be allocated for
52 realtime application. This can be achieved by isolating cpus from kernel
53 scheduler. Please refer to
54 http://lxr.free-electrons.com/source/Documentation/kernel-parameters.txt#L1608
57 * **Memory allocation:**
58 Memory shoud be reserved for realtime applications and usually hugepage
59 should be used to reduce page fauts/TLB misses.
62 All the non-realtime IRQs should be affinitized to non realtime CPUs to
63 reduce the impact on realtime CPUs. Some OS distributions contain an
64 irqbalance daemon which balances the IRQs among all the cores dynamically.
65 It should be disabled as well.
67 * **Device assignment for VM:**
68 If a device is used in a VM, then device passthrough is desirable. In this
69 case,the IOMMU should be enabled.
72 Frequent clock ticks cause latency. CONFIG_NOHZ_FULL should be enabled in
73 the linux kernel. With CONFIG_NOHZ_FULL, the physical CPU will trigger many
74 fewer clock tick interrupts(currently, 1 tick per second). This can reduce
75 latency because each host timer interrupt triggers a VM exit from guest to
76 host which causes performance/latency impacts.
79 Mark TSC clock source as reliable. A TSC clock source that seems to be
80 unreliable causes the kernel to continuously enable the clock source
81 watchdog to check if TSC frequency is still correct. On recent Intel
82 platforms with Constant TSC/Invariant TSC/Synchronized TSC, the TSC is
83 reliable so the watchdog is useless but cause latency.
86 The poll option forces a polling idle loop that can slightly improve the
87 performance of waking up an idle CPU.
90 RCU is a kernel synchronization mechanism. Refer to
91 http://lxr.free-electrons.com/source/Documentation/RCU/whatisRCU.txt for more
92 information. With RCU_NOCB, the impact from RCU to the VNF will be reduced.
94 * **Disable the RT throttling:**
95 RT Throttling is a Linux kernel mechanism that
96 occurs when a process or thread uses 100% of the core, leaving no resources
97 for the Linux scheduler to execute the kernel/housekeeping tasks. RT
98 Throttling increases the latency so should be disabled.
100 * **NUMA configuration:**
101 To achieve the best latency. CPU/Memory and device allocated for realtime
102 application/VM should be in the same NUMA node.