1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) OPNFV, Intel Corporation, AT&T and others.
5 vSwitchPerf test suites userguide
6 ---------------------------------
11 VSPERF requires a traffic generators to run tests, automated traffic gen
12 support in VSPERF includes:
14 - IXIA traffic generator (IxNetwork hardware) and a machine that runs the IXIA
16 - Spirent traffic generator (TestCenter hardware chassis or TestCenter virtual
17 in a VM) and a VM to run the Spirent Virtual Deployment Service image,
18 formerly known as "Spirent LabServer".
19 - Xena Network traffic generator (Xena hardware chassis) that houses the Xena
20 Traffic generator modules.
21 - Moongen software traffic generator. Requires a separate machine running
22 moongen to execute packet generation.
24 If you want to use another traffic generator, please select the Dummy generator
25 option as shown in `Traffic generator instructions
26 <http://artifacts.opnfv.org/vswitchperf/docs/configguide/trafficgen.html>`__
31 To see the supported Operating Systems, vSwitches and system requirements,
32 please follow the `installation instructions
33 <http://artifacts.opnfv.org/vswitchperf/docs/configguide/installation.html>`__ to
36 Traffic Generator Setup
37 ^^^^^^^^^^^^^^^^^^^^^^^
39 Follow the `Traffic generator instructions
40 <http://artifacts.opnfv.org/vswitchperf/docs/configguide/trafficgen.html>`__ to
41 install and configure a suitable traffic generator.
43 Cloning and building src dependencies
44 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
46 In order to run VSPERF, you will need to download DPDK and OVS. You can
47 do this manually and build them in a preferred location, OR you could
48 use vswitchperf/src. The vswitchperf/src directory contains makefiles
49 that will allow you to clone and build the libraries that VSPERF depends
50 on, such as DPDK and OVS. To clone and build simply:
52 .. code-block:: console
57 VSPERF can be used with stock OVS (without DPDK support). When build
58 is finished, the libraries are stored in src_vanilla directory.
60 The 'make' builds all options in src:
63 * OVS with vhost_user as the guest access method (with DPDK support)
64 * OVS with vhost_cuse s the guest access method (with DPDK support)
66 The vhost_user build will reside in src/ovs/
67 The vhost_cuse build will reside in vswitchperf/src_cuse
68 The Vanilla OVS build will reside in vswitchperf/src_vanilla
70 To delete a src subdirectory and its contents to allow you to re-clone simply
73 .. code-block:: console
77 Configure the ``./conf/10_custom.conf`` file
78 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
80 The ``10_custom.conf`` file is the configuration file that overrides
81 default configurations in all the other configuration files in ``./conf``
82 The supplied ``10_custom.conf`` file **MUST** be modified, as it contains
83 configuration items for which there are no reasonable default values.
85 The configuration items that can be added is not limited to the initial
86 contents. Any configuration item mentioned in any .conf file in
87 ``./conf`` directory can be added and that item will be overridden by
88 the custom configuration value.
90 Using a custom settings file
91 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
93 If your ``10_custom.conf`` doesn't reside in the ``./conf`` directory
94 of if you want to use an alternative configuration file, the file can
95 be passed to ``vsperf`` via the ``--conf-file`` argument.
97 .. code-block:: console
99 $ ./vsperf --conf-file <path_to_custom_conf> ...
101 Note that configuration passed in via the environment (``--load-env``)
102 or via another command line argument will override both the default and
103 your custom configuration files. This "priority hierarchy" can be
104 described like so (1 = max priority):
106 1. Command line arguments
107 2. Environment variables
108 3. Configuration file(s)
113 vsperf uses a VM called vloop_vnf for looping traffic in the PVP and PVVP
114 deployment scenarios. The image can be downloaded from
115 `<http://artifacts.opnfv.org/>`__.
117 .. code-block:: console
119 $ wget http://artifacts.opnfv.org/vswitchperf/vloop-vnf-ubuntu-14.04_20151216.qcow2
121 Newer vloop_vnf images are available. Please reference the
122 installation instructions for information on these images
123 `installation instructions
124 <http://artifacts.opnfv.org/vswitchperf/docs/configguide/installation.html>`__
127 vloop_vnf forwards traffic through a VM using one of:
130 * l2fwd kernel Module.
132 Alternatively you can use your own QEMU image.
137 A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with
138 support for Destination Network Address Translation (DNAT) for both the MAC and
139 IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd
144 Before running any tests make sure you have root permissions by adding
145 the following line to /etc/sudoers:
147 .. code-block:: console
149 username ALL=(ALL) NOPASSWD: ALL
151 username in the example above should be replaced with a real username.
153 To list the available tests:
155 .. code-block:: console
159 To run a single test:
161 .. code-block:: console
165 Where $TESTNAME is the name of the vsperf test you would like to run.
167 To run a group of tests, for example all tests with a name containing
170 .. code-block:: console
172 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests="RFC2544"
176 .. code-block:: console
178 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
180 Some tests allow for configurable parameters, including test duration
181 (in seconds) as well as packet sizes (in bytes).
185 $ ./vsperf --conf-file user_settings.py
187 --test-params "duration=10;pkt_sizes=128"
189 For all available options, check out the help dialog:
191 .. code-block:: console
195 Executing Vanilla OVS tests
196 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
198 1. If needed, recompile src for all OVS variants
200 .. code-block:: console
206 2. Update your ''10_custom.conf'' file to use the appropriate variables
209 .. code-block:: console
211 VSWITCH = 'OvsVanilla'
213 Where $PORT1 and $PORT2 are the Linux interfaces you'd like to bind
218 .. code-block:: console
220 $ ./vsperf --conf-file=<path_to_custom_conf>
222 Please note if you don't want to configure Vanilla OVS through the
223 configuration file, you can pass it as a CLI argument; BUT you must
226 .. code-block:: console
228 $ ./vsperf --vswitch OvsVanilla
231 Executing PVP and PVVP tests
232 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
234 To run tests using vhost-user as guest access method:
236 1. Set VHOST_METHOD and VNF of your settings file to:
238 .. code-block:: console
241 VNF = 'QemuDpdkVhost'
243 2. If needed, recompile src for all OVS variants
245 .. code-block:: console
253 .. code-block:: console
255 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
257 To run tests using vhost-cuse as guest access method:
259 1. Set VHOST_METHOD and VNF of your settings file to:
261 .. code-block:: console
264 VNF = 'QemuDpdkVhostCuse'
266 2. If needed, recompile src for all OVS variants
268 .. code-block:: console
276 .. code-block:: console
278 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
280 Executing PVP tests using Vanilla OVS
281 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
283 To run tests using Vanilla OVS:
285 1. Set the following variables:
287 .. code-block:: console
289 VSWITCH = 'OvsVanilla'
290 VNF = 'QemuVirtioNet'
292 VANILLA_TGEN_PORT1_IP = n.n.n.n
293 VANILLA_TGEN_PORT1_MAC = nn:nn:nn:nn:nn:nn
295 VANILLA_TGEN_PORT2_IP = n.n.n.n
296 VANILLA_TGEN_PORT2_MAC = nn:nn:nn:nn:nn:nn
298 VANILLA_BRIDGE_IP = n.n.n.n
302 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
303 --test-params "vanilla_tgen_tx_ip=n.n.n.n;
304 vanilla_tgen_tx_mac=nn:nn:nn:nn:nn:nn"
307 2. If needed, recompile src for all OVS variants
309 .. code-block:: console
317 .. code-block:: console
319 $ ./vsperf --conf-file<path_to_custom_conf>/10_custom.conf
323 Using vfio_pci with DPDK
324 ^^^^^^^^^^^^^^^^^^^^^^^^^
326 To use vfio with DPDK instead of igb_uio edit 'conf/02_vswitch.conf'
327 with the following parameters:
329 .. code-block:: console
334 SYS_MODULES = ['cuse']
336 **NOTE:** Please ensure that Intel VT-d is enabled in BIOS.
338 **NOTE:** Please ensure your boot/grub parameters include
341 .. code-block:: console
343 iommu=pt intel_iommu=on
345 To check that IOMMU is enabled on your platform:
347 .. code-block:: console
350 [ 0.000000] Intel-IOMMU: enabled
351 [ 0.139882] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
352 [ 0.139888] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
353 [ 0.139893] IOAPIC id 2 under DRHD base 0xfbffe000 IOMMU 0
354 [ 0.139894] IOAPIC id 0 under DRHD base 0xebffc000 IOMMU 1
355 [ 0.139895] IOAPIC id 1 under DRHD base 0xebffc000 IOMMU 1
356 [ 3.335744] IOMMU: dmar0 using Queued invalidation
357 [ 3.335746] IOMMU: dmar1 using Queued invalidation
365 To use virtual functions of NIC with SRIOV support, use extended form
366 of NIC PCI slot definition:
368 .. code-block:: python
370 WHITELIST_NICS = ['0000:05:00.0|vf0', '0000:05:00.1|vf3']
372 Where 'vf' is an indication of virtual function usage and following
373 number defines a VF to be used. In case that VF usage is detected,
374 then vswitchperf will enable SRIOV support for given card and it will
375 detect PCI slot numbers of selected VFs.
377 So in example above, one VF will be configured for NIC '0000:05:00.0'
378 and four VFs will be configured for NIC '0000:05:00.1'. Vswitchperf
379 will detect PCI addresses of selected VFs and it will use them during
382 At the end of vswitchperf execution, SRIOV support will be disabled.
384 SRIOV support is generic and it can be used in different testing scenarios.
387 * vSwitch tests with DPDK or without DPDK support to verify impact
388 of VF usage on vSwitch performance
389 * tests without vSwitch, where traffic is forwared directly
390 between VF interfaces by packet forwarder (e.g. testpmd application)
391 * tests without vSwitch, where VM accesses VF interfaces directly
392 by PCI-passthrough_ to measure raw VM throughput performance.
396 Using QEMU with PCI passthrough support
397 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
399 Raw virtual machine throughput performance can be measured by execution of PVP
400 test with direct access to NICs by PCI passthrough. To execute VM with direct
401 access to PCI devices, enable vfio-pci_. In order to use virtual functions,
402 SRIOV-support_ must be enabled.
404 Execution of test with PCI passthrough with vswitch disabled:
406 .. code-block:: console
408 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
409 --vswitch none --vnf QemuPciPassthrough pvp_tput
411 Any of supported guest-loopback-application_ can be used inside VM with
412 PCI passthrough support.
414 Note: Qemu with PCI passthrough support can be used only with PVP test
417 .. _guest-loopback-application:
419 Selection of loopback application for PVP and PVVP tests
420 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
422 To select loopback application, which will perform traffic forwarding
423 inside VM, following configuration parameter should be configured:
425 .. code-block:: console
427 GUEST_LOOPBACK = ['testpmd', 'testpmd']
431 .. code-block:: console
433 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
434 --test-params "guest_loopback=testpmd"
436 Supported loopback applications are:
438 .. code-block:: console
440 'testpmd' - testpmd from dpdk will be built and used
441 'l2fwd' - l2fwd module provided by Huawei will be built and used
442 'linux_bridge' - linux bridge will be configured
443 'buildin' - nothing will be configured by vsperf; VM image must
444 ensure traffic forwarding between its interfaces
446 Guest loopback application must be configured, otherwise traffic
447 will not be forwarded by VM and testcases with PVP and PVVP deployments
448 will fail. Guest loopback application is set to 'testpmd' by default.
450 Multi-Queue Configuration
451 ^^^^^^^^^^^^^^^^^^^^^^^^^
453 VSPerf currently supports multi-queue with the following limitations:
455 1. Execution of pvp/pvvp tests require testpmd as the loopback if multi-queue
456 is enabled at the guest.
458 2. Requires QemuDpdkVhostUser as the vnf.
460 3. Requires switch to be set to OvsDpdkVhost.
462 4. Requires QEMU 2.5 or greater and any OVS version higher than 2.5. The
463 default upstream package versions installed by VSPerf satisfy this
466 To enable multi-queue modify the ''02_vswitch.conf'' file to enable multi-queue
469 .. code-block:: console
471 VSWITCH_MULTI_QUEUES = 2
473 **NOTE:** you should consider using the switch affinity to set a pmd cpu mask
474 that can optimize your performance. Consider the numa of the NIC in use if this
475 applies by checking /sys/class/net/<eth_name>/device/numa_node and setting an
476 appropriate mask to create PMD threads on the same numa node.
478 When multi-queue is enabled, each dpdk or dpdkvhostuser port that is created
479 on the switch will set the option for multiple queues.
481 To enable multi-queue on the guest modify the ''04_vnf.conf'' file.
483 .. code-block:: console
487 Enabling multi-queue at the guest will add multiple queues to each NIC port when
488 qemu launches the guest.
490 Testpmd should be configured to take advantage of multi-queue on the guest. This
491 can be done by modifying the ''04_vnf.conf'' file.
493 .. code-block:: console
495 GUEST_TESTPMD_CPU_MASK = '-l 0,1,2,3,4'
497 GUEST_TESTPMD_NB_CORES = 4
498 GUEST_TESTPMD_TXQ = 2
499 GUEST_TESTPMD_RXQ = 2
501 **NOTE:** The guest SMP cores must be configured to allow for testpmd to use the
502 optimal number of cores to take advantage of the multiple guest queues.
504 **NOTE:** For optimal performance guest SMPs should be on the same numa as the
505 NIC in use if possible/applicable. Testpmd should be assigned at least
506 (nb_cores +1) total cores with the cpu mask.
508 Executing Packet Forwarding tests
509 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
511 To select application, which will perform packet forwarding,
512 following configuration parameter should be configured:
514 .. code-block:: console
519 or use --vswitch and --fwdapp
521 $ ./vsperf --conf-file user_settings.py
525 Supported Packet Forwarding applications are:
527 .. code-block:: console
529 'testpmd' - testpmd from dpdk
532 1. Update your ''10_custom.conf'' file to use the appropriate variables
533 for selected Packet Forwarder:
535 .. code-block:: console
537 # testpmd configuration
539 # packet forwarding mode supported by testpmd; Please see DPDK documentation
540 # for comprehensive list of modes supported by your version.
541 # e.g. io|mac|mac_retry|macswap|flowgen|rxonly|txonly|csum|icmpecho|...
542 # Note: Option "mac_retry" has been changed to "mac retry" since DPDK v16.07
543 TESTPMD_FWD_MODE = 'csum'
544 # checksum calculation layer: ip|udp|tcp|sctp|outer-ip
545 TESTPMD_CSUM_LAYER = 'ip'
546 # checksum calculation place: hw (hardware) | sw (software)
547 TESTPMD_CSUM_CALC = 'sw'
548 # recognize tunnel headers: on|off
549 TESTPMD_CSUM_PARSE_TUNNEL = 'off'
553 .. code-block:: console
555 $ ./vsperf --conf-file <path_to_settings_py>
557 VSPERF modes of operation
558 ^^^^^^^^^^^^^^^^^^^^^^^^^
560 VSPERF can be run in different modes. By default it will configure vSwitch,
561 traffic generator and VNF. However it can be used just for configuration
562 and execution of traffic generator. Another option is execution of all
563 components except traffic generator itself.
565 Mode of operation is driven by configuration parameter -m or --mode
567 .. code-block:: console
569 -m MODE, --mode MODE vsperf mode of operation;
571 "normal" - execute vSwitch, VNF and traffic generator
572 "trafficgen" - execute only traffic generator
573 "trafficgen-off" - execute vSwitch and VNF
574 "trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission
576 In case, that VSPERF is executed in "trafficgen" mode, then configuration
577 of traffic generator should be configured through --test-params option.
578 Supported CLI options useful for traffic generator configuration are:
580 .. code-block:: console
582 'traffic_type' - One of the supported traffic types. E.g. rfc2544,
583 back2back or continuous
584 Default value is "rfc2544".
585 'bidirectional' - Specifies if generated traffic will be full-duplex (true)
586 or half-duplex (false)
587 Default value is "false".
588 'iload' - Defines desired percentage of frame rate used during
589 continuous stream tests.
590 Default value is 100.
591 'multistream' - Defines number of flows simulated by traffic generator.
592 Value 0 disables MultiStream feature
594 'stream_type' - Stream Type is an extension of the "MultiStream" feature.
595 If MultiStream is disabled, then Stream Type will be
596 ignored. Stream Type defines ISO OSI network layer used
597 for simulation of multiple streams.
598 Default value is "L4".
600 Example of execution of VSPERF in "trafficgen" mode:
602 .. code-block:: console
604 $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf
605 --test-params "traffic_type=continuous;bidirectional=True;iload=60"
607 Code change verification by pylint
608 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
610 Every developer participating in VSPERF project should run
611 pylint before his python code is submitted for review. Project
612 specific configuration for pylint is available at 'pylint.rc'.
614 Example of manual pylint invocation:
616 .. code-block:: console
618 $ pylint --rcfile ./pylintrc ./vsperf
623 OVS with DPDK and QEMU
624 ~~~~~~~~~~~~~~~~~~~~~~~
626 If you encounter the following error: "before (last 100 chars):
627 '-path=/dev/hugepages,share=on: unable to map backing store for
628 hugepages: Cannot allocate memory\r\n\r\n" with the PVP or PVVP
629 deployment scenario, check the amount of hugepages on your system:
631 .. code-block:: console
633 $ cat /proc/meminfo | grep HugePages
636 By default the vswitchd is launched with 1Gb of memory, to change
637 this, modify --socket-mem parameter in conf/02_vswitch.conf to allocate
638 an appropriate amount of memory:
640 .. code-block:: console
642 VSWITCHD_DPDK_ARGS = ['-c', '0x4', '-n', '4', '--socket-mem 1024,0']
643 VSWITCHD_DPDK_CONFIG = {
644 'dpdk-init' : 'true',
645 'dpdk-lcore-mask' : '0x4',
646 'dpdk-socket-mem' : '1024,0',
649 Note: Option VSWITCHD_DPDK_ARGS is used for vswitchd, which supports --dpdk
650 parameter. In recent vswitchd versions, option VSWITCHD_DPDK_CONFIG will be
651 used to configure vswitchd via ovs-vsctl calls.
657 For more information and details refer to the vSwitchPerf user guide at:
658 http://artifacts.opnfv.org/vswitchperf/docs/userguide/index.html