1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) <optionally add copywriters name>
5 Low Latency Environment
6 =======================
8 Achieving low latency with the KVM4NFV project requires setting up a special
9 test environment. This environment includes the BIOS settings, kernel
10 configuration, kernel parameters and the run-time environment.
12 Hardware Environment Description
13 --------------------------------
15 BIOS setup plays an important role in achieving real-time latency. A collection
16 of relevant settings, used on the platform where the baseline performance data
17 was collected, is detailed below:
22 Some special CPU features like TSC-deadline timer, invariant TSC and Process posted
23 interrupts, etc, are helpful for latency reduction.
25 Below is the CPU information on the baseline test platform.
28 vendor_id : GenuineIntel
31 model name : Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
46 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
47 mca cmov pat pse36 clflush dts acpi mmx fxsr sse
48 sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm
49 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc
50 aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2
51 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt
52 tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm arat epb
53 pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase
54 tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc
59 address sizes : 46 bits physical, 48 bits virtual
65 NUMA topology is also important for latency reduction.
67 Below is the CPU topology on the baseline test platform.
69 [nfv@otcnfv02 ~]$ lscpu
71 CPU op-mode(s): 32-bit, 64-bit
72 Byte Order: Little Endian
74 On-line CPU(s) list: 0-35
76 Core(s) per socket: 18
79 Vendor ID: GenuineIntel
82 Model name: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
91 NUMA node0 CPU(s): 0-17
92 NUMA node1 CPU(s): 18-35
97 Careful BIOS setup is important in achieving real time latency. Different
98 platforms have different BIOS setups, below are the important BIOS settings on
99 the platform used to collect the baseline performance data.
101 CPU Power and Performance <Performance>
102 CPU C-State <Disabled>
103 C1E Autopromote <Disabled>
104 Processor C3 <Disabled>
105 Processor C6 <Disabled>
106 Select Memory RAS <Maximum Performance>
107 NUMA Optimized <Enabled>
108 Cluster-on-Die <Disabled>
109 Patrol Scrub <Disabled>
110 Demand Scrub <Disabled>
111 Correctable Error <10>
112 Intel(R) Hyper-Threading <Disabled>
113 Active Processor Cores <All>
114 Execute Disable Bit <Enabled>
115 Intel(R) Virtualization Technology <Enabled>
116 Intel(R) TXT <Disabled>
117 Enhanced Error Containment Mode <Disabled>
118 USB Controller <Enabled>
119 USB 3.0 Controller <Auto>
120 Legacy USB Support <Disabled>
121 Port 60/64 Emulation <Disabled>
123 Software Environment Setup
124 --------------------------
125 Both the host and the guest environment need to be configured properly to
126 reduce latency variations. Below are some suggested kernel configurations.
127 The ci/envs/ directory gives detailed implementation on how to setup the
133 Please check the default kernel configuration in the source code at:
134 kernel/arch/x86/configs/opnfv.config.
136 Below is host kernel boot line example:
138 isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35 iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable
140 Below is guest kernel boot line example
142 isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G hugepagesz=1G
144 Please refer to :doc:`tunning` for more explanation.
146 Run-time Environment Setup
147 ~~~~~~~~~~~~~~~~~~~~~~~~~~
149 Not only are special kernel parameters needed but a special run-time
150 environment is also required. Please refer to :doc:`tunning` for more