1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
5 Low Latency Environment
6 =======================
8 Achieving low latency with the KVM4NFV project requires setting up a special
9 test environment. This environment includes the BIOS settings, kernel
10 configuration, kernel parameters and the run-time environment.
12 Hardware Environment Description
13 --------------------------------
15 BIOS setup plays an important role in achieving real-time latency. A collection
16 of relevant settings, used on the platform where the baseline performance data
17 was collected, is detailed below:
22 Some special CPU features like TSC-deadline timer, invariant TSC and Process
23 posted interrupts, etc, are helpful for latency reduction.
28 NUMA topology is also important for latency reduction.
33 Careful BIOS setup is important in achieving real time latency. Different
34 platforms have different BIOS setups, below are the important BIOS settings on
35 the platform used to collect the baseline performance data.
37 Software Environment Setup
38 --------------------------
39 Both the host and the guest environment need to be configured properly to
40 reduce latency variations. Below are some suggested kernel configurations.
41 The ci/envs/ directory gives detailed implementation on how to setup the
47 Please check the default kernel configuration in the source code at:
48 kernel/arch/x86/configs/opnfv.config.
50 Below is host kernel boot line example:
54 isolcpus=11-15,31-35 nohz_full=11-15,31-35 rcu_nocbs=11-15,31-35
55 iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G mce=off idle=poll
56 intel_pstate=disable processor.max_cstate=1 pcie_asmp=off tsc=reliable
58 Below is guest kernel boot line example
62 isolcpus=1 nohz_full=1 rcu_nocbs=1 mce=off idle=poll default_hugepagesz=1G
65 Please refer to `tuning.userguide` for more explanation.
67 Run-time Environment Setup
68 ~~~~~~~~~~~~~~~~~~~~~~~~~~
70 Not only are special kernel parameters needed but a special run-time
71 environment is also required. Please refer to `tunning.userguide` for
74 Test cases to measure Latency
75 -----------------------------
76 The performance of the kvm4nfv is assesed by the latency values. Cyclictest and Packet forwarding
77 Test cases result in real time latency values of average, minimum and maximum.
81 * Packet Forwarding test
85 Cyclictest results are the most frequently cited real-time Linux metric. The core concept of Cyclictest is very simple.
86 In KVM4NFV cyclictest is implemented on the Guest-VM with 4.4-Kernel RPM installed. It generated Max,Min and Avg
87 values which help in assesing the kernel used. Cyclictest in currently divided into the following test types,
94 Future scope of work may include the below test-types,
96 * CPU_stress-CPU_stress
97 * Memory_stress-Memory_stress
100 Understanding the naming convention
101 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
105 [Host-Type ] - [Guest-Type]
107 * **Host-Type :** Mentions the type of stress applied on the kernel of the Host
108 * **Guest-Type :** Mentions the type of stress applied on the kernel of the Guest
116 The above name signifies that,
118 - No Stress is applied on the Host kernel
120 - CPU Stress is applied on the Guest kernel
124 - Stress is applied using the stress which is installed as part of the deployment.
125 Stress can be applied on CPU, Memory and Input-Output (Read/Write) operations using the stress tool.
130 +-----------------------+------------------+-----------------+
131 | **Test Name** | **Colorado** | **Danube** |
133 +-----------------------+------------------+-----------------+
134 | - Idle - Idle | ``Y`` | ``Y`` |
135 +-----------------------+------------------+-----------------+
136 | - Cpustress - Idle | | ``Y`` |
137 +-----------------------+------------------+-----------------+
138 | - Memorystress - Idle | | ``Y`` |
139 +-----------------------+------------------+-----------------+
140 | - IOstress - Idle | | ``Y`` |
141 +-----------------------+------------------+-----------------+
146 Cyclictest in run on the Guest VM when Host,Guest are not under any kind of stress. This is the basic
147 cyclictest of the KVM4NFV project. Outputs Avg, Min and Max latency values.
149 .. figure:: images/idle-idle-test-type.png
150 :name: idle-idle test type
154 CPU_Stress-Idle test-type
155 ~~~~~~~~~~~~~~~~~~~~~~~~~
156 Here, the host is under CPU stress, where multiple times sqrt() function is called on kernel which
157 results increased CPU load. The cyclictest will run on the guest, where the guest is under no stress.
158 Outputs Avg, Min and Max latency values.
160 .. figure:: images/cpu-stress-idle-test-type.png
161 :name: cpu-stress-idle test type
165 Memory_Stress-Idle test-type
166 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
167 In this type, the host is under memory stress where continuos memory operations are implemented to
168 increase the Memory stress (Buffer stress).The cyclictest will run on the guest, where the guest is under
169 no stress. It outputs Avg, Min and Max latency values.
171 .. figure:: images/memory-stress-idle-test-type.png
172 :name: memory-stress-idle test type
176 IO_Stress-Idle test-type
177 ~~~~~~~~~~~~~~~~~~~~~~~~
178 The host is under constant Input/Output stress .i.e., multiple read-write operations are invoked to
179 increase stress. Cyclictest will run on the guest VM that is launched on the same host, where the guest
180 is under no stress. It outputs Avg, Min and Max latency values.
182 .. figure:: images/io-stress-idle-test-type.png
183 :name: io-stress-idle test type
187 CPU_Stress-CPU_Stress test-type
188 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
189 Not implemented for Danube release.
191 Memory_Stress-Memory_Stress test-type
192 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
193 Not implemented for Danube release.
195 IO_Stress-IO_Stress test type
196 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
197 Not implemented for Danube release.
199 2. Packet Forwarding Test cases
200 -------------------------------
201 Packet forwarding is an other test case of Kvm4nfv. It measures the time taken by a packet to return
202 to source after reaching its destination. This test case uses automated test-framework provided by
203 OPNFV VSWITCHPERF project and a traffic generator (IXIA is used for kvm4nfv). Only latency results
204 generating test cases are triggered as a part of kvm4nfv daily job.
206 Latency test measures the time required for a frame to travel from the originating device through the
207 network to the destination device. Please note that RFC2544 Latency measurement will be superseded with
208 a measurement of average latency over all successfully transferred packets or frames.
210 Packet forwarding test cases currently supports the following test types:
212 * Packet forwarding to Host
214 * Packet forwarding to Guest
216 * Packet forwarding to Guest using SRIOV
218 The testing approach adoped is black box testing, meaning the test inputs can be generated and the
219 outputs captured and completely evaluated from the outside of the System Under Test(SUT).
221 Packet forwarding to Host
222 ~~~~~~~~~~~~~~~~~~~~~~~~~
223 This is also known as Physical port → vSwitch → physical port deployment.
224 This test measures the time taken by the packet/frame generated by traffic generator(phy) to travel
225 through the network to the destination device(phy). This test results min,avg and max latency values.
226 This value signifies the performance of the installed kernel.
230 .. figure:: images/host_pk_fw.png
231 :name: packet forwarding to host
235 Packet forwarding to Guest
236 ~~~~~~~~~~~~~~~~~~~~~~~~~~
237 This is also known as Physical port → vSwitch → VNF → vSwitch → physical port deployment.
239 This test measures the time taken by the packet/frame generated by traffic generator(phy) to travel
240 through the network involving a guest to the destination device(phy). This test results min,avg and
241 max latency values. This value signifies the performance of the installed kernel.
245 .. figure:: images/guest_pk_fw.png
246 :name: packet forwarding to guest
250 Packet forwarding to Guest using SRIOV
251 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
252 This test is used to verify the VNF and measure the base performance (maximum forwarding rate in
253 fps and latency) that can be achieved by the VNF without a vSwitch. The performance metrics
254 collected by this test will serve as a key comparison point for NIC passthrough technologies and
255 vSwitches. VNF in this context refers to the hypervisor and the VM.
257 **Note:** The Vsperf running on the host is still required.
261 .. figure:: images/sriov_pk_fw.png
262 :name: packet forwarding to guest using sriov