1 Low Latency Environment
2 =======================
4 Achieving low latency with the KVM4NFV project requires setting up a special
5 test environment. This environment includes the BIOS settings, kernel
6 configuration, kernel parameters and the run-time environment.
8 Hardware Environment Description
9 --------------------------------
11 BIOS setup plays an important role in achieving real-time latency. A collection
12 of relevant settings, used on the platform where the baseline performance data
13 was collected, is detailed below:
18 Some special CPU features like TSC-deadline timer, invariant TSC and Process posted
19 interrupts etc are helpful for latency reduction.
21 Below is the CPU information on the baseline test platform.
24 vendor_id : GenuineIntel
27 model name : Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
42 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
43 mca cmov pat pse36 clflush dts acpi mmx fxsr sse
44 sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm
45 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
46 aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
47 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt
48 tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
49 pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase
50 tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc
55 address sizes : 46 bits physical, 48 bits virtual
61 NUMA topology is also important for latency reducation.
63 Below is the CPU topology on the baseline test platform.
65 [nfv@otcnfv02 ~]$ lscpu
67 CPU op-mode(s): 32-bit, 64-bit
68 Byte Order: Little Endian
70 On-line CPU(s) list: 0-35
72 Core(s) per socket: 18
75 Vendor ID: GenuineIntel
78 Model name: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
87 NUMA node0 CPU(s): 0-17
88 NUMA node1 CPU(s): 18-35
93 Careful BIOS setup is important in achieving real time latency. Different
94 platforms have different BIOS setups, below are the important BIOS settings on
95 the platform used to collect the baseline performance data.
97 CPU Power and Performance <Performance>
98 CPU C-State <Disabled>
99 C1E Autopromote <Disabled>
100 Processor C3 <Disabled>
101 Processor C6 <Disabled>
102 Select Memory RAS <Maximum Performance>
103 NUMA Optimized <Enabled>
104 Cluster-on-Die <Disabled>
105 Patrol Scrub <Disabled>
106 Demand Scrub <Disabled>
107 Correctable Error <10>
108 Intel(R) Hyper-Threading <Disabled>
109 Active Processor Cores <All>
110 Execute Disable Bit <Enabled>
111 Intel(R) Virtualization Technology <Enabled>
112 Intel(R) TXT <Disabled>
113 Enhanced Error Containment Mode <Disabled>
114 USB Controller <Enabled>
115 USB 3.0 Controller <Auto>
116 Legacy USB Support <Disabled>
117 Port 60/64 Emulation <Disabled>
119 Software Environment Setup
120 --------------------------
121 Both the host and the guest environment need to be configured properly to
122 reduce latency variations. Below are some suggested kernel configurations.
123 The ci/envs/ directory gives etailed implementation on how to setup the
129 Please check the default kernel configuration in the source code at:
130 kernel/arch/x86/configs/opnfv.config.
132 Below is host kernel boot line example:
134 isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35 iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable
136 Below is guest kernel boot line example
138 isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G hugepagesz=1G
140 Please refer to :doc:`tunning` for more explanation.
142 Run-time Environment Setup
143 ~~~~~~~~~~~~~~~~~~~~~~~~~~
145 Not only are special kernel parameters needed but a special run-time
146 environment is also required. Please refer to :doc:`tunning` for more