1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
3 .. (c) Cisco Systems, Inc
9 NFVbench supports SR-IOV with the PVP and PVVP packet flow. Most use cases for SR-IOV only require single VNF chains (NxPVP).
10 Daisy chaining VNFs with SR-IOV (PVVP) requires selecting either SR-IOV for the middle network or a fast vswitch (using the
11 standard OVS for that purpose works but would be a serious bottleneck)
13 Instructions below refer to the PVP or PVVP use cases.
14 For external chains using SR-IOV, select the VLAN tagging option that corresponds to the external chains SR-IOV setting.
18 To test SR-IOV you need to have compute nodes configured to support one or more SR-IOV interfaces (also knows as PF or physical function)
19 and you need OpenStack to be configured to support SR-IOV.
20 You will also need to know:
21 - the name of the physical networks associated to the SR-IOV interfaces (this is a configuration in Nova compute)
22 - the VLAN range that can be used on the switch ports that are wired to the SR-IOV ports. Such switch ports are normally configured in trunk mode with a range of VLAN ids enabled on that port
24 For example, in the case of 2 SR-IOV ports per compute node, 2 physical networks are generally configured in OpenStack with a distinct name.
25 The VLAN range to use is is also allocated and reserved by the network administrator and in coordination with the corresponding top of rack switch port configuration.
30 To enable SR-IOV test, you will need to provide the following configuration options to NFVbench (in the configuration file).
31 This example instructs NFVbench to create the left and right networks of a PVP packet flow to run on 2 SRIOV ports named "phys_sriov0" and "phys_sriov1" using resp. segmentation_id 2000 and 2001:
39 physical_network: phys_sriov0
42 physical_network: phys_sriov1
44 The segmentation ID fields must be different.
45 In the case of PVVP, the middle network also needs to be provisioned properly.
46 The same physical network can also be shared by the virtual networks but with different segmentation IDs.
50 The above configuration works for multi-chaining and shared network ("service_chain_shared_net" set to true).
51 In that case all VNFs will share the same left and right network/VLAN.
53 In the case of non shared network ("service_chain_shared_net" set to false), the segmentation_id fields must
54 contain a list of distinct VLANs to use for each chain. Example of configuration for 3 chains:
61 segmentation_id: [2000, 2001, 2002]
62 physical_network: phys_sriov0
64 segmentation_id: [2100, 2101, 2102]
65 physical_network: phys_sriov1
67 Alternatively it is also possible to specify different physnets per chain:
74 segmentation_id: [2000, 2001, 2002]
75 physical_network: [phys_sriov0, phys_sriov2, phys_sriov4]
77 segmentation_id: [2100, 2101, 2102]
78 physical_network: [phys_sriov1, phys_srviov3, phys_sriov5]
81 NFVbench cores with SR-IOV
82 --------------------------
83 The default core count for NFVbench/TRex may not be sufficient for higher throughput line cards (greater than 10Gbps).
84 This will result in warning messages such as:
88 INFO WARNING: There is a significant difference between requested TX rate (119047618) and actual TX rate (38897379).
89 The traffic generator may not have sufficient CPU to achieve the requested TX rate.
91 In that case it is recommended to try allocating more cores to TRex using the cores property in the configuration
92 file, for example to set to 8 cores:
98 It is also advisable to increase the number of vcpus in the VMs:
101 VM Flavor for SR-IOV and NIC NUMA socket placement
102 --------------------------------------------------
104 Because SR-IOV throughput uses a lot of CPU in the VM, it is recommended to increase the
105 vcpu count, for example to 4 vcpus:
110 # Number of vCPUs for the flavor
112 # Memory for the flavor in MB
114 # Size of local disk in GB
117 "hw:cpu_policy": dedicated
119 If the 2 selected ports reside on NICs that are on different NUMA sockets, you will need to explicitly tell Nova to use 2 numa nodes in the flavor used for the VMs in order to satisfy the filters, for example:
124 # Number of vCPUs for the flavor
126 # Memory for the flavor in MB
128 # Size of local disk in GB
131 "hw:cpu_policy": dedicated
134 Failure to do so might cause the VM creation to fail with the Nova error
135 "Instance creation error: Insufficient compute resources:
136 Requested instance NUMA topology together with requested PCI devices cannot fit the given host NUMA topology."
138 Example of configuration file (shared network)
139 ----------------------------------------------
141 Single chain or multi-chain with shared network (only requires 2 segmentation ID for all chains):
146 # Number of vCPUs for the flavor
148 # Memory for the flavor in MB
150 # Size of local disk in GB
153 "hw:cpu_policy": dedicated
158 segmentation_id: 3830
159 physical_network: phys_sriov0
161 segmentation_id: 3831
162 physical_network: phys_sriov1
164 Example of full run 2xPVP shared network SR-IOV:
168 2018-12-03 18:24:07,419 INFO Loading configuration file: /tmp/nfvbench/sriov.yaml
169 2018-12-03 18:24:07,423 INFO -c /tmp/nfvbench/sriov.yaml --rate 10Mpps --duration 1 -scc 2 --no-cleanup
170 2018-12-03 18:24:07,426 INFO Connecting to TRex (127.0.0.1)...
171 2018-12-03 18:24:07,575 INFO Connected to TRex
172 2018-12-03 18:24:07,575 INFO Port 0: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:70 pci=0000:5e:00.0 driver=net_i40e
173 2018-12-03 18:24:07,575 INFO Port 1: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:71 pci=0000:5e:00.1 driver=net_i40e
174 2018-12-03 18:24:07,626 INFO Found built-in VM image file nfvbenchvm-0.6.qcow2
175 2018-12-03 18:24:09,072 INFO Created flavor 'nfvbench.medium'
176 2018-12-03 18:24:10,004 INFO Created network: nfvbench-lnet.
177 2018-12-03 18:24:10,837 INFO Created network: nfvbench-rnet.
178 2018-12-03 18:24:12,065 INFO Security disabled on port nfvbench-loop-vm0-0
179 2018-12-03 18:24:13,425 INFO Security disabled on port nfvbench-loop-vm0-1
180 2018-12-03 18:24:13,425 INFO Creating instance nfvbench-loop-vm0 with AZ
181 2018-12-03 18:24:16,052 INFO Created instance nfvbench-loop-vm0 - waiting for placement resolution...
182 2018-12-03 18:24:16,240 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 1/101)...
184 2018-12-03 18:24:59,266 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 21/101)...
185 2018-12-03 18:25:01,427 INFO Instance nfvbench-loop-vm0 is active and has been placed on nova:charter-compute-5
186 2018-12-03 18:25:02,819 INFO Security disabled on port nfvbench-loop-vm1-0
187 2018-12-03 18:25:04,198 INFO Security disabled on port nfvbench-loop-vm1-1
188 2018-12-03 18:25:04,199 INFO Creating instance nfvbench-loop-vm1 with AZ nova:charter-compute-5
189 2018-12-03 18:25:05,032 INFO Created instance nfvbench-loop-vm1 on nova:charter-compute-5
190 2018-12-03 18:25:05,033 INFO Instance nfvbench-loop-vm0 is ACTIVE on nova:charter-compute-5
191 2018-12-03 18:25:05,212 INFO Waiting for 1/2 instance to become active (retry 1/100)...
193 2018-12-03 18:25:48,531 INFO Waiting for 1/2 instance to become active (retry 21/100)...
194 2018-12-03 18:25:50,677 INFO Instance nfvbench-loop-vm1 is ACTIVE on nova:charter-compute-5
195 2018-12-03 18:25:50,677 INFO All instances are active
196 2018-12-03 18:25:50,677 INFO Port 0: VLANs [3830, 3830]
197 2018-12-03 18:25:50,677 INFO Port 1: VLANs [3831, 3831]
198 2018-12-03 18:25:50,677 INFO Port 0: dst MAC ['fa:16:3e:de:4e:54', 'fa:16:3e:7a:26:2b']
199 2018-12-03 18:25:50,677 INFO Port 1: dst MAC ['fa:16:3e:6c:bb:cd', 'fa:16:3e:e0:48:45']
200 2018-12-03 18:25:50,678 INFO ChainRunner initialized
201 2018-12-03 18:25:50,678 INFO Starting 2xPVP benchmark...
202 2018-12-03 18:25:50,683 INFO Starting traffic generator to ensure end-to-end connectivity
203 2018-12-03 18:25:50,698 INFO Created 2 traffic streams for port 0.
204 2018-12-03 18:25:50,700 INFO Created 2 traffic streams for port 1.
205 2018-12-03 18:25:50,821 INFO Captured unique src mac 0/4, capturing return packets (retry 1/100)...
206 2018-12-03 18:25:52,944 INFO Received packet from mac: fa:16:3e:de:4e:54 (chain=0, port=0)
207 2018-12-03 18:25:52,945 INFO Received packet from mac: fa:16:3e:6c:bb:cd (chain=0, port=1)
208 2018-12-03 18:25:53,077 INFO Captured unique src mac 2/4, capturing return packets (retry 2/100)...
210 2018-12-03 18:26:10,798 INFO End-to-end connectivity established
211 2018-12-03 18:26:10,816 INFO Cleared all existing streams
212 2018-12-03 18:26:10,846 INFO Created 4 traffic streams for port 0.
213 2018-12-03 18:26:10,849 INFO Created 4 traffic streams for port 1.
214 2018-12-03 18:26:10,849 INFO Starting to generate traffic...
215 2018-12-03 18:26:10,850 INFO Running traffic generator
216 2018-12-03 18:26:11,877 INFO TX: 10000004; RX: 9999999; Est. Dropped: 5; Est. Drop rate: 0.0000%
217 2018-12-03 18:26:11,877 INFO ...traffic generating ended.
218 2018-12-03 18:26:11,882 INFO Service chain 'PVP' run completed.
219 2018-12-03 18:26:11,936 INFO Clean up skipped.
220 2018-12-03 18:26:11,969 INFO
221 ========== NFVBench Summary ==========
222 Date: 2018-12-03 18:25:50
223 NFVBench version 3.0.3.dev1
235 build_date: Nov 13 2017
240 > CiscoVIM: 2.9.7-17036
244 Profile: traffic_profile_64B
247 Service chains count: 2
248 Compute nodes: [u'nova:charter-compute-5']
252 +-----------------+-------------+----------------------+----------------------+----------------------+
253 | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) |
254 +=================+=============+======================+======================+======================+
255 | 64 | 0.0000% | 13 | 10 | 141 |
256 +-----------------+-------------+----------------------+----------------------+----------------------+
263 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
264 | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) |
265 +=============+===========================+========================+=================+===========================+========================+=================+
266 | Forward | 336.0000 Mbps | 336.0000 Mbps | 336.0000 Mbps | 500,000 pps | 500,000 pps | 500,000 pps |
267 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
268 | Reverse | 336.0000 Mbps | 336.0000 Mbps | 336.0000 Mbps | 500,000 pps | 500,000 pps | 500,000 pps |
269 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
270 | Total | 672.0000 Mbps | 672.0000 Mbps | 672.0000 Mbps | 1,000,000 pps | 1,000,000 pps | 1,000,000 pps |
271 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
273 Forward Chain Packet Counters and Latency:
275 +---------+--------------+--------------+------------+------------+------------+
276 | Chain | TRex.TX.p0 | TRex.RX.p1 | Avg lat. | Min lat. | Max lat. |
277 +=========+==============+==============+============+============+============+
278 | 0 | 250,000 | 250,000 | 17 usec | 10 usec | 138 usec |
279 +---------+--------------+--------------+------------+------------+------------+
280 | 1 | 250,000 | 250,000 | 17 usec | 10 usec | 139 usec |
281 +---------+--------------+--------------+------------+------------+------------+
282 | total | 500,000 | 500,000 | 17 usec | 10 usec | 139 usec |
283 +---------+--------------+--------------+------------+------------+------------+
285 Reverse Chain Packet Counters and Latency:
287 +---------+--------------+--------------+------------+------------+------------+
288 | Chain | TRex.TX.p1 | TRex.RX.p0 | Avg lat. | Min lat. | Max lat. |
289 +=========+==============+==============+============+============+============+
290 | 0 | 250,000 | 250,000 | 12 usec | 10 usec | 141 usec |
291 +---------+--------------+--------------+------------+------------+------------+
292 | 1 | 250,000 | 250,000 | 11 usec | 10 usec | 132 usec |
293 +---------+--------------+--------------+------------+------------+------------+
294 | total | 500,000 | 500,000 | 12 usec | 10 usec | 141 usec |
295 +---------+--------------+--------------+------------+------------+------------+
297 Example of configuration file (non shared network)
298 --------------------------------------------------
300 Multi-chain with non shared network (requires 2 segmentation ID per chain), example with 2 chains
301 sharing the same 2 SRIOV ports (or PF):
306 # Number of vCPUs for the flavor
308 # Memory for the flavor in MB
310 # Size of local disk in GB
313 "hw:cpu_policy": dedicated
318 segmentation_id: [3830, 3831]
319 physical_network: phys_sriov0
321 segmentation_id: [3832, 3833]
322 physical_network: phys_sriov1
324 Example of full run 2xPVP non-shared network SR-IOV:
328 2018-12-04 17:15:25,284 INFO -c /tmp/nfvbench/sriov.yaml --rate 1Mpps --duration 1 -scc 2 --no-cleanup
329 2018-12-04 17:15:25,287 INFO Connecting to TRex (127.0.0.1)...
330 2018-12-04 17:15:25,463 INFO Connected to TRex
331 2018-12-04 17:15:25,464 INFO Port 0: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:70 pci=0000:5e:00.0 driver=net_i40e
332 2018-12-04 17:15:25,464 INFO Port 1: Ethernet Controller XL710 for 40GbE QSFP+ speed=40Gbps mac=3c:fd:fe:b5:3d:71 pci=0000:5e:00.1 driver=net_i40e
333 2018-12-04 17:15:25,515 INFO Found built-in VM image file nfvbenchvm-0.6.qcow2
334 2018-12-04 17:15:26,457 INFO Created flavor 'nfvbench.medium'
335 2018-12-04 17:15:27,449 INFO Created network: nfvbench-lnet0.
336 2018-12-04 17:15:28,368 INFO Created network: nfvbench-rnet0.
337 2018-12-04 17:15:29,143 INFO Created port nfvbench-loop-vm0-0
338 2018-12-04 17:15:29,626 INFO Security disabled on port nfvbench-loop-vm0-0
339 2018-12-04 17:15:30,636 INFO Created port nfvbench-loop-vm0-1
340 2018-12-04 17:15:31,139 INFO Security disabled on port nfvbench-loop-vm0-1
341 2018-12-04 17:15:31,140 INFO Creating instance nfvbench-loop-vm0 with AZ
342 2018-12-04 17:15:34,893 INFO Created instance nfvbench-loop-vm0 - waiting for placement resolution...
343 2018-12-04 17:15:35,068 INFO Waiting for instance nfvbench-loop-vm0 to become active (retry 1/101)...
345 2018-12-04 17:16:22,253 INFO Instance nfvbench-loop-vm0 is active and has been placed on nova:charter-compute-4
346 2018-12-04 17:16:23,154 INFO Created network: nfvbench-lnet1.
347 2018-12-04 17:16:23,863 INFO Created network: nfvbench-rnet1.
348 2018-12-04 17:16:24,799 INFO Created port nfvbench-loop-vm1-0
349 2018-12-04 17:16:25,267 INFO Security disabled on port nfvbench-loop-vm1-0
350 2018-12-04 17:16:26,006 INFO Created port nfvbench-loop-vm1-1
351 2018-12-04 17:16:26,612 INFO Security disabled on port nfvbench-loop-vm1-1
352 2018-12-04 17:16:26,612 INFO Creating instance nfvbench-loop-vm1 with AZ nova:charter-compute-4
353 2018-12-04 17:16:27,610 INFO Created instance nfvbench-loop-vm1 on nova:charter-compute-4
354 2018-12-04 17:16:27,610 INFO Instance nfvbench-loop-vm0 is ACTIVE on nova:charter-compute-4
355 2018-12-04 17:16:27,788 INFO Waiting for 1/2 instance to become active (retry 1/100)...
358 2018-12-04 17:17:04,258 INFO Instance nfvbench-loop-vm1 is ACTIVE on nova:charter-compute-4
359 2018-12-04 17:17:04,258 INFO All instances are active
360 2018-12-04 17:17:04,259 INFO Port 0: VLANs [3830, 3831]
361 2018-12-04 17:17:04,259 INFO Port 1: VLANs [3832, 3833]
362 2018-12-04 17:17:04,259 INFO Port 0: dst MAC ['fa:16:3e:ef:f4:b0', 'fa:16:3e:e5:74:cd']
363 2018-12-04 17:17:04,259 INFO Port 1: dst MAC ['fa:16:3e:d6:dc:84', 'fa:16:3e:8e:d9:30']
364 2018-12-04 17:17:04,259 INFO ChainRunner initialized
365 2018-12-04 17:17:04,260 INFO Starting 2xPVP benchmark...
366 2018-12-04 17:17:04,266 INFO Starting traffic generator to ensure end-to-end connectivity
367 2018-12-04 17:17:04,297 INFO Created 2 traffic streams for port 0.
368 2018-12-04 17:17:04,300 INFO Created 2 traffic streams for port 1.
369 2018-12-04 17:17:04,420 INFO Captured unique src mac 0/4, capturing return packets (retry 1/100)...
370 2018-12-04 17:17:06,532 INFO Received packet from mac: fa:16:3e:d6:dc:84 (chain=0, port=1)
371 2018-12-04 17:17:06,532 INFO Received packet from mac: fa:16:3e:ef:f4:b0 (chain=0, port=0)
372 2018-12-04 17:17:06,644 INFO Captured unique src mac 2/4, capturing return packets (retry 2/100)...
375 2018-12-04 17:17:24,337 INFO Received packet from mac: fa:16:3e:8e:d9:30 (chain=1, port=1)
376 2018-12-04 17:17:24,338 INFO Received packet from mac: fa:16:3e:e5:74:cd (chain=1, port=0)
377 2018-12-04 17:17:24,338 INFO End-to-end connectivity established
378 2018-12-04 17:17:24,355 INFO Cleared all existing streams
379 2018-12-04 17:17:24,383 INFO Created 4 traffic streams for port 0.
380 2018-12-04 17:17:24,386 INFO Created 4 traffic streams for port 1.
381 2018-12-04 17:17:24,386 INFO Starting to generate traffic...
382 2018-12-04 17:17:24,386 INFO Running traffic generator
383 2018-12-04 17:17:25,415 INFO TX: 1000004; RX: 1000004; Est. Dropped: 0; Est. Drop rate: 0.0000%
384 2018-12-04 17:17:25,415 INFO ...traffic generating ended.
385 2018-12-04 17:17:25,420 INFO Service chain 'PVP' run completed.
386 2018-12-04 17:17:25,471 INFO Clean up skipped.
387 2018-12-04 17:17:25,508 INFO
388 ========== NFVBench Summary ==========
389 Date: 2018-12-04 17:17:04
390 NFVBench version 3.0.3.dev1
402 build_date: Nov 13 2017
407 > CiscoVIM: 2.9.7-17036
411 Profile: traffic_profile_64B
414 Service chains count: 2
415 Compute nodes: [u'nova:charter-compute-4']
419 +-----------------+-------------+----------------------+----------------------+----------------------+
420 | L2 Frame Size | Drop Rate | Avg Latency (usec) | Min Latency (usec) | Max Latency (usec) |
421 +=================+=============+======================+======================+======================+
422 | 64 | 0.0000% | 18 | 10 | 120 |
423 +-----------------+-------------+----------------------+----------------------+----------------------+
430 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
431 | Direction | Requested TX Rate (bps) | Actual TX Rate (bps) | RX Rate (bps) | Requested TX Rate (pps) | Actual TX Rate (pps) | RX Rate (pps) |
432 +=============+===========================+========================+=================+===========================+========================+=================+
433 | Forward | 336.0000 Mbps | 336.0013 Mbps | 336.0013 Mbps | 500,000 pps | 500,002 pps | 500,002 pps |
434 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
435 | Reverse | 336.0000 Mbps | 336.0013 Mbps | 336.0013 Mbps | 500,000 pps | 500,002 pps | 500,002 pps |
436 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
437 | Total | 672.0000 Mbps | 672.0027 Mbps | 672.0027 Mbps | 1,000,000 pps | 1,000,004 pps | 1,000,004 pps |
438 +-------------+---------------------------+------------------------+-----------------+---------------------------+------------------------+-----------------+
440 Forward Chain Packet Counters and Latency:
442 +---------+--------------+--------------+------------+------------+------------+
443 | Chain | TRex.TX.p0 | TRex.RX.p1 | Avg lat. | Min lat. | Max lat. |
444 +=========+==============+==============+============+============+============+
445 | 0 | 250,001 | 250,001 | 26 usec | 10 usec | 70 usec |
446 +---------+--------------+--------------+------------+------------+------------+
447 | 1 | 250,001 | 250,001 | 11 usec | 10 usec | 39 usec |
448 +---------+--------------+--------------+------------+------------+------------+
449 | total | 500,002 | 500,002 | 19 usec | 10 usec | 70 usec |
450 +---------+--------------+--------------+------------+------------+------------+
452 Reverse Chain Packet Counters and Latency:
454 +---------+--------------+--------------+------------+------------+------------+
455 | Chain | TRex.TX.p1 | TRex.RX.p0 | Avg lat. | Min lat. | Max lat. |
456 +=========+==============+==============+============+============+============+
457 | 0 | 250,001 | 250,001 | 19 usec | 10 usec | 119 usec |
458 +---------+--------------+--------------+------------+------------+------------+
459 | 1 | 250,001 | 250,001 | 19 usec | 10 usec | 120 usec |
460 +---------+--------------+--------------+------------+------------+------------+
461 | total | 500,002 | 500,002 | 19 usec | 10 usec | 120 usec |
462 +---------+--------------+--------------+------------+------------+------------+