1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) OPNFV, Intel Corporation, AT&T and others.
5 vSwitchPerf test suites userguide
6 ---------------------------------
11 VSPERF requires a traffic generators to run tests, automated traffic gen
12 support in VSPERF includes:
14 - IXIA traffic generator (IxNetwork hardware) and a machine that runs the IXIA
16 - Spirent traffic generator (TestCenter hardware chassis or TestCenter virtual
17 in a VM) and a VM to run the Spirent Virtual Deployment Service image,
18 formerly known as "Spirent LabServer".
19 - Xena Network traffic generator (Xena hardware chassis) that houses the Xena
20 Traffic generator modules.
21 - Moongen software traffic generator. Requires a separate machine running
22 moongen to execute packet generation.
23 - T-Rex software traffic generator. Requires a separate machine running T-Rex
24 Server to execute packet generation.
26 If you want to use another traffic generator, please select the :ref:`trafficgen-dummy`
32 To see the supported Operating Systems, vSwitches and system requirements,
33 please follow the `installation instructions <vsperf-installation>`.
35 Traffic Generator Setup
36 ^^^^^^^^^^^^^^^^^^^^^^^
38 Follow the `Traffic generator instructions <trafficgen-installation>` to
39 install and configure a suitable traffic generator.
41 Cloning and building src dependencies
42 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
44 In order to run VSPERF, you will need to download DPDK and OVS. You can
45 do this manually and build them in a preferred location, OR you could
46 use vswitchperf/src. The vswitchperf/src directory contains makefiles
47 that will allow you to clone and build the libraries that VSPERF depends
48 on, such as DPDK and OVS. To clone and build simply:
50 .. code-block:: console
55 VSPERF can be used with stock OVS (without DPDK support). When build
56 is finished, the libraries are stored in src_vanilla directory.
58 The 'make' builds all options in src:
61 * OVS with vhost_user as the guest access method (with DPDK support)
63 The vhost_user build will reside in src/ovs/
64 The Vanilla OVS build will reside in vswitchperf/src_vanilla
66 To delete a src subdirectory and its contents to allow you to re-clone simply
69 .. code-block:: console
73 Configure the ``./conf/10_custom.conf`` file
74 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
76 The ``10_custom.conf`` file is the configuration file that overrides
77 default configurations in all the other configuration files in ``./conf``
78 The supplied ``10_custom.conf`` file **MUST** be modified, as it contains
79 configuration items for which there are no reasonable default values.
81 The configuration items that can be added is not limited to the initial
82 contents. Any configuration item mentioned in any .conf file in
83 ``./conf`` directory can be added and that item will be overridden by
84 the custom configuration value.
86 Further details about configuration files evaluation and special behaviour
87 of options with ``GUEST_`` prefix could be found at :ref:`design document
88 <design-configuration>`.
90 Using a custom settings file
91 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
93 If your ``10_custom.conf`` doesn't reside in the ``./conf`` directory
94 or if you want to use an alternative configuration file, the file can
95 be passed to ``vsperf`` via the ``--conf-file`` argument.
97 .. code-block:: console
99 $ ./vsperf --conf-file <path_to_custom_conf> ...
101 Evaluation of configuration parameters
102 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
104 The value of configuration parameter can be specified at various places,
105 e.g. at the test case definition, inside configuration files, by the command
106 line argument, etc. Thus it is important to understand the order of configuration
107 parameter evaluation. This "priority hierarchy" can be described like so
110 1. Testcase definition keywords ``vSwitch``, ``Trafficgen``, ``VNF`` and ``Tunnel Type``
111 2. Parameters inside testcase definition section ``Parameters``
112 3. Command line arguments (e.g. ``--test-params``, ``--vswitch``, ``--trafficgen``, etc.)
113 4. Environment variables (see ``--load-env`` argument)
114 5. Custom configuration file specified via ``--conf-file`` argument
115 6. Standard configuration files, where higher prefix number means higher
118 For example, if the same configuration parameter is defined in custom configuration
119 file (specified via ``--conf-file`` argument), via ``--test-params`` argument
120 and also inside ``Parameters`` section of the testcase definition, then parameter
121 value from the ``Parameters`` section will be used.
123 Further details about order of configuration files evaluation and special behaviour
124 of options with ``GUEST_`` prefix could be found at :ref:`design document
125 <design-configuration>`.
127 .. _overriding-parameters-documentation:
129 Overriding values defined in configuration files
130 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
132 The configuration items can be overridden by command line argument
133 ``--test-params``. In this case, the configuration items and
134 their values should be passed in form of ``item=value`` and separated
141 $ ./vsperf --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,);" \
142 "GUEST_LOOPBACK=['testpmd','l2fwd']" pvvp_tput
144 The ``--test-params`` command line argument can also be used to override default
145 configuration values for multiple tests. Providing a list of parameters will apply each
146 element of the list to the test with the same index. If more tests are run than
147 parameters provided the last element of the list will repeat.
151 $ ./vsperf --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"
152 "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \
155 The second option is to override configuration items by ``Parameters`` section
156 of the test case definition. The configuration items can be added into ``Parameters``
157 dictionary with their new values. These values will override values defined in
158 configuration files or specified by ``--test-params`` command line argument.
164 "Parameters" : {'TRAFFICGEN_PKT_SIZES' : (128,),
165 'TRAFFICGEN_DURATION' : 10,
166 'GUEST_LOOPBACK' : ['testpmd','l2fwd'],
169 **NOTE:** In both cases, configuration item names and their values must be specified
170 in the same form as they are defined inside configuration files. Parameter names
171 must be specified in uppercase and data types of original and new value must match.
172 Python syntax rules related to data types and structures must be followed.
173 For example, parameter ``TRAFFICGEN_PKT_SIZES`` above is defined as a tuple
174 with a single value ``128``. In this case trailing comma is mandatory, otherwise
175 value can be wrongly interpreted as a number instead of a tuple and vsperf
176 execution would fail. Please check configuration files for default values and their
177 types and use them as a basis for any customized values. In case of any doubt, please
178 check official python documentation related to data structures like tuples, lists
181 **NOTE:** Vsperf execution will terminate with runtime error in case, that unknown
182 parameter name is passed via ``--test-params`` CLI argument or defined in ``Parameters``
183 section of test case definition. It is also forbidden to redefine a value of
184 ``TEST_PARAMS`` configuration item via CLI or ``Parameters`` section.
186 **NOTE:** The new definition of the dictionary parameter, specified via ``--test-params``
187 or inside ``Parameters`` section, will not override original dictionary values. Instead
188 the original dictionary will be updated with values from the new dictionary definition.
190 Referencing parameter values
191 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
193 It is possible to use a special macro ``#PARAM()`` to refer to the value of
194 another configuration parameter. This reference is evaluated during
195 access of the parameter value (by ``settings.getValue()`` call), so it
196 can refer to parameters created during VSPERF runtime, e.g. NICS dictionary.
197 It can be used to reflect DUT HW details in the testcase definition.
209 # set destination MAC to the MAC of the first
210 # interface from WHITELIST_NICS list
211 'dstmac' : '#PARAM(NICS[0]["mac"])',
219 VSPERF uses a VM image called vloop_vnf for looping traffic in the deployment
220 scenarios involving VMs. The image can be downloaded from
221 `<http://artifacts.opnfv.org/>`__.
223 Please see the installation instructions for information on :ref:`vloop-vnf`
231 A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with
232 support for Destination Network Address Translation (DNAT) for both the MAC and
233 IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd
235 Additional Tools Setup
236 ^^^^^^^^^^^^^^^^^^^^^^
238 Follow the `Additional tools instructions <additional-tools-configuration>` to
239 install and configure additional tools such as collectors and loadgens.
244 All examples inside these docs assume, that user is inside the VSPERF
245 directory. VSPERF can be executed from any directory.
247 Before running any tests make sure you have root permissions by adding
248 the following line to /etc/sudoers:
250 .. code-block:: console
252 username ALL=(ALL) NOPASSWD: ALL
254 username in the example above should be replaced with a real username.
256 To list the available tests:
258 .. code-block:: console
262 To run a single test:
264 .. code-block:: console
268 Where $TESTNAME is the name of the vsperf test you would like to run.
270 To run a test multiple times, repeat it:
272 .. code-block:: console
274 $ ./vsperf $TESTNAME $TESTNAME $TESTNAME
276 To run a group of tests, for example all tests with a name containing
279 .. code-block:: console
281 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf --tests="RFC2544"
285 .. code-block:: console
287 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
289 Some tests allow for configurable parameters, including test duration
290 (in seconds) as well as packet sizes (in bytes).
294 $ ./vsperf --conf-file user_settings.py \
295 --tests RFC2544Tput \
296 --test-params "TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)"
298 To specify configurable parameters for multiple tests, use a list of
299 parameters. One element for each test.
303 $ ./vsperf --conf-file user_settings.py \
304 --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\
305 "'TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(64,)']" \
306 phy2phy_cont phy2phy_cont
308 If the ``CUMULATIVE_PARAMS`` setting is set to True and there are different parameters
309 provided for each test using ``--test-params``, each test will take the parameters of
310 the previous test before appyling it's own.
311 With ``CUMULATIVE_PARAMS`` set to True the following command will be equivalent to the
316 $ ./vsperf --conf-file user_settings.py \
317 --test-params "['TRAFFICGEN_DURATION=10;TRAFFICGEN_PKT_SIZES=(128,)',"\
318 "'TRAFFICGEN_PKT_SIZES=(64,)']" \
319 phy2phy_cont phy2phy_cont
322 For all available options, check out the help dialog:
324 .. code-block:: console
328 Executing Vanilla OVS tests
329 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
331 1. If needed, recompile src for all OVS variants
333 .. code-block:: console
339 2. Update your ``10_custom.conf`` file to use Vanilla OVS:
341 .. code-block:: python
343 VSWITCH = 'OvsVanilla'
347 .. code-block:: console
349 $ ./vsperf --conf-file=<path_to_custom_conf>
351 Please note if you don't want to configure Vanilla OVS through the
352 configuration file, you can pass it as a CLI argument.
354 .. code-block:: console
356 $ ./vsperf --vswitch OvsVanilla
359 Executing tests with VMs
360 ^^^^^^^^^^^^^^^^^^^^^^^^
362 To run tests using vhost-user as guest access method:
364 1. Set VSWITCH and VNF of your settings file to:
366 .. code-block:: python
368 VSWITCH = 'OvsDpdkVhost'
369 VNF = 'QemuDpdkVhost'
371 2. If needed, recompile src for all OVS variants
373 .. code-block:: console
381 .. code-block:: console
383 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
385 **NOTE:** By default vSwitch is acting as a server for dpdk vhost-user sockets.
386 In case, that QEMU should be a server for vhost-user sockets, then parameter
387 ``VSWITCH_VHOSTUSER_SERVER_MODE`` should be set to ``False``.
389 Executing tests with VMs using Vanilla OVS
390 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
392 To run tests using Vanilla OVS:
394 1. Set the following variables:
396 .. code-block:: python
398 VSWITCH = 'OvsVanilla'
399 VNF = 'QemuVirtioNet'
401 VANILLA_TGEN_PORT1_IP = n.n.n.n
402 VANILLA_TGEN_PORT1_MAC = nn:nn:nn:nn:nn:nn
404 VANILLA_TGEN_PORT2_IP = n.n.n.n
405 VANILLA_TGEN_PORT2_MAC = nn:nn:nn:nn:nn:nn
407 VANILLA_BRIDGE_IP = n.n.n.n
409 or use ``--test-params`` option
411 .. code-block:: console
413 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
414 --test-params "VANILLA_TGEN_PORT1_IP=n.n.n.n;" \
415 "VANILLA_TGEN_PORT1_MAC=nn:nn:nn:nn:nn:nn;" \
416 "VANILLA_TGEN_PORT2_IP=n.n.n.n;" \
417 "VANILLA_TGEN_PORT2_MAC=nn:nn:nn:nn:nn:nn"
419 2. If needed, recompile src for all OVS variants
421 .. code-block:: console
429 .. code-block:: console
431 $ ./vsperf --conf-file<path_to_custom_conf>/10_custom.conf
438 Currently it is not possible to use standard scenario deployments for execution of
439 tests with VPP. It means, that deployments ``p2p``, ``pvp``, ``pvvp`` and in general any
440 :ref:`pxp-deployment` won't work with VPP. However it is possible to use VPP in
441 :ref:`step-driven-tests`. A basic set of VPP testcases covering ``phy2phy``, ``pvp``
442 and ``pvvp`` tests are already prepared.
444 List of performance tests with VPP support follows:
446 * phy2phy_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
447 * phy2phy_cont_vpp: VPP: Phy2Phy Continuous Stream
448 * phy2phy_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames
449 * pvp_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
450 * pvp_cont_vpp: VPP: PVP Continuous Stream
451 * pvp_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames
452 * pvvp_tput_vpp: VPP: LTD.Throughput.RFC2544.PacketLossRatio
453 * pvvp_cont_vpp: VPP: PVP Continuous Stream
454 * pvvp_back2back_vpp: VPP: LTD.Throughput.RFC2544.BackToBackFrames
456 In order to execute testcases with VPP it is required to:
458 * install VPP manually, see :ref:`vpp-installation`
459 * configure ``WHITELIST_NICS``, with two physical NICs connected to the traffic generator
460 * configure traffic generator, see :ref:`trafficgen-installation`
462 After that it is possible to execute VPP testcases listed above.
466 .. code-block:: console
468 $ ./vsperf --conf-file=<path_to_custom_conf> phy2phy_tput_vpp
472 Using vfio_pci with DPDK
473 ^^^^^^^^^^^^^^^^^^^^^^^^^
475 To use vfio with DPDK instead of igb_uio add into your custom configuration
476 file the following parameter:
478 .. code-block:: python
480 PATHS['dpdk']['src']['modules'] = ['uio', 'vfio-pci']
483 **NOTE:** In case, that DPDK is installed from binary package, then please
484 set ``PATHS['dpdk']['bin']['modules']`` instead.
486 **NOTE:** Please ensure that Intel VT-d is enabled in BIOS.
488 **NOTE:** Please ensure your boot/grub parameters include
491 .. code-block:: console
493 iommu=pt intel_iommu=on
495 To check that IOMMU is enabled on your platform:
497 .. code-block:: console
500 [ 0.000000] Intel-IOMMU: enabled
501 [ 0.139882] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
502 [ 0.139888] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
503 [ 0.139893] IOAPIC id 2 under DRHD base 0xfbffe000 IOMMU 0
504 [ 0.139894] IOAPIC id 0 under DRHD base 0xebffc000 IOMMU 1
505 [ 0.139895] IOAPIC id 1 under DRHD base 0xebffc000 IOMMU 1
506 [ 3.335744] IOMMU: dmar0 using Queued invalidation
507 [ 3.335746] IOMMU: dmar1 using Queued invalidation
510 **NOTE:** In case of VPP, it is required to explicitly define, that vfio-pci
511 DPDK driver should be used. It means to update dpdk part of VSWITCH_VPP_ARGS
512 dictionary with uio-driver section, e.g. VSWITCH_VPP_ARGS['dpdk'] = 'uio-driver vfio-pci'
519 To use virtual functions of NIC with SRIOV support, use extended form
520 of NIC PCI slot definition:
522 .. code-block:: python
524 WHITELIST_NICS = ['0000:05:00.0|vf0', '0000:05:00.1|vf3']
526 Where 'vf' is an indication of virtual function usage and following
527 number defines a VF to be used. In case that VF usage is detected,
528 then vswitchperf will enable SRIOV support for given card and it will
529 detect PCI slot numbers of selected VFs.
531 So in example above, one VF will be configured for NIC '0000:05:00.0'
532 and four VFs will be configured for NIC '0000:05:00.1'. Vswitchperf
533 will detect PCI addresses of selected VFs and it will use them during
536 At the end of vswitchperf execution, SRIOV support will be disabled.
538 SRIOV support is generic and it can be used in different testing scenarios.
541 * vSwitch tests with DPDK or without DPDK support to verify impact
542 of VF usage on vSwitch performance
543 * tests without vSwitch, where traffic is forwarded directly
544 between VF interfaces by packet forwarder (e.g. testpmd application)
545 * tests without vSwitch, where VM accesses VF interfaces directly
546 by PCI-passthrough_ to measure raw VM throughput performance.
550 Using QEMU with PCI passthrough support
551 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
553 Raw virtual machine throughput performance can be measured by execution of PVP
554 test with direct access to NICs by PCI pass-through. To execute VM with direct
555 access to PCI devices, enable vfio-pci_. In order to use virtual functions,
556 SRIOV-support_ must be enabled.
558 Execution of test with PCI pass-through with vswitch disabled:
560 .. code-block:: console
562 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
563 --vswitch none --vnf QemuPciPassthrough pvp_tput
565 Any of supported guest-loopback-application_ can be used inside VM with
566 PCI pass-through support.
568 Note: Qemu with PCI pass-through support can be used only with PVP test
571 .. _guest-loopback-application:
573 Selection of loopback application for tests with VMs
574 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
576 To select the loopback applications which will forward packets inside VMs,
577 the following parameter should be configured:
579 .. code-block:: python
581 GUEST_LOOPBACK = ['testpmd']
583 or use ``--test-params`` CLI argument:
585 .. code-block:: console
587 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
588 --test-params "GUEST_LOOPBACK=['testpmd']"
590 Supported loopback applications are:
592 .. code-block:: console
594 'testpmd' - testpmd from dpdk will be built and used
595 'l2fwd' - l2fwd module provided by Huawei will be built and used
596 'linux_bridge' - linux bridge will be configured
597 'buildin' - nothing will be configured by vsperf; VM image must
598 ensure traffic forwarding between its interfaces
600 Guest loopback application must be configured, otherwise traffic
601 will not be forwarded by VM and testcases with VM related deployments
602 will fail. Guest loopback application is set to 'testpmd' by default.
604 **NOTE:** In case that only 1 or more than 2 NICs are configured for VM,
605 then 'testpmd' should be used. As it is able to forward traffic between
606 multiple VM NIC pairs.
608 **NOTE:** In case of linux_bridge, all guest NICs are connected to the same
609 bridge inside the guest.
611 Mergable Buffers Options with QEMU
612 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
614 Mergable buffers can be disabled with VSPerf within QEMU. This option can
615 increase performance significantly when not using jumbo frame sized packets.
616 By default VSPerf disables mergable buffers. If you wish to enable it you
617 can modify the setting in the a custom conf file.
619 .. code-block:: python
621 GUEST_NIC_MERGE_BUFFERS_DISABLE = [False]
623 Then execute using the custom conf file.
625 .. code-block:: console
627 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf
629 Alternatively you can just pass the param during execution.
631 .. code-block:: console
633 $ ./vsperf --test-params "GUEST_NIC_MERGE_BUFFERS_DISABLE=[False]"
636 Selection of dpdk binding driver for tests with VMs
637 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
639 To select dpdk binding driver, which will specify which driver the vm NICs will
640 use for dpdk bind, the following configuration parameter should be configured:
642 .. code-block:: console
644 GUEST_DPDK_BIND_DRIVER = ['igb_uio_from_src']
646 The supported dpdk guest bind drivers are:
648 .. code-block:: console
650 'uio_pci_generic' - Use uio_pci_generic driver
651 'igb_uio_from_src' - Build and use the igb_uio driver from the dpdk src
653 'vfio_no_iommu' - Use vfio with no iommu option. This requires custom
654 guest images that support this option. The default
655 vloop image does not support this driver.
657 Note: uio_pci_generic does not support sr-iov testcases with guests attached.
658 This is because uio_pci_generic only supports legacy interrupts. In case
659 uio_pci_generic is selected with the vnf as QemuPciPassthrough it will be
660 modified to use igb_uio_from_src instead.
662 Note: vfio_no_iommu requires kernels equal to or greater than 4.5 and dpdk
663 16.04 or greater. Using this option will also taint the kernel.
665 Please refer to the dpdk documents at http://dpdk.org/doc/guides for more
666 information on these drivers.
668 Guest Core and Thread Binding
669 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
671 VSPERF provides options to achieve better performance by guest core binding and
672 guest vCPU thread binding as well. Core binding is to bind all the qemu threads.
673 Thread binding is to bind the house keeping threads to some CPU and vCPU thread to
674 some other CPU, this helps to reduce the noise from qemu house keeping threads.
677 .. code-block:: python
679 GUEST_CORE_BINDING = [('#EVAL(6+2*#VMINDEX)', '#EVAL(7+2*#VMINDEX)')]
681 **NOTE** By default the GUEST_THREAD_BINDING will be none, which means same as
682 the GUEST_CORE_BINDING, i.e. the vcpu threads are sharing the physical CPUs with
683 the house keeping threads. Better performance using vCPU thread binding can be
684 achieved by enabling affinity in the custom configuration file.
686 For example, if an environment requires 32,33 to be core binded and 29,30&31 for
687 guest thread binding to achieve better performance.
689 .. code-block:: python
691 VNF_AFFINITIZATION_ON = True
692 GUEST_CORE_BINDING = [('32','33')]
693 GUEST_THREAD_BINDING = [('29', '30', '31')]
698 QEMU default to a compatible subset of performance enhancing cpu features.
699 To pass all available host processor features to the guest.
701 .. code-block:: python
703 GUEST_CPU_OPTIONS = ['host,migratable=off']
705 **NOTE** To enhance the performance, cpu features tsc deadline timer for guest,
706 the guest PMU, the invariant TSC can be provided in the custom configuration file.
708 Multi-Queue Configuration
709 ^^^^^^^^^^^^^^^^^^^^^^^^^
711 VSPerf currently supports multi-queue with the following limitations:
713 1. Requires QEMU 2.5 or greater and any OVS version higher than 2.5. The
714 default upstream package versions installed by VSPerf satisfies this
717 2. Guest image must have ethtool utility installed if using l2fwd or linux
718 bridge inside guest for loopback.
720 3. If using OVS versions 2.5.0 or less enable old style multi-queue as shown
721 in the ''02_vswitch.conf'' file.
723 .. code-block:: python
725 OVS_OLD_STYLE_MQ = True
727 To enable multi-queue for dpdk modify the ''02_vswitch.conf'' file.
729 .. code-block:: python
731 VSWITCH_DPDK_MULTI_QUEUES = 2
733 **NOTE:** you should consider using the switch affinity to set a pmd cpu mask
734 that can optimize your performance. Consider the numa of the NIC in use if this
735 applies by checking /sys/class/net/<eth_name>/device/numa_node and setting an
736 appropriate mask to create PMD threads on the same numa node.
738 When multi-queue is enabled, each dpdk or dpdkvhostuser port that is created
739 on the switch will set the option for multiple queues. If old style multi queue
740 has been enabled a global option for multi queue will be used instead of the
743 To enable multi-queue on the guest modify the ''04_vnf.conf'' file.
745 .. code-block:: python
747 GUEST_NIC_QUEUES = [2]
749 Enabling multi-queue at the guest will add multiple queues to each NIC port when
750 qemu launches the guest.
752 In case of Vanilla OVS, multi-queue is enabled on the tuntap ports and nic
753 queues will be enabled inside the guest with ethtool. Simply enabling the
754 multi-queue on the guest is sufficient for Vanilla OVS multi-queue.
756 Testpmd should be configured to take advantage of multi-queue on the guest if
757 using DPDKVhostUser. This can be done by modifying the ''04_vnf.conf'' file.
759 .. code-block:: python
761 GUEST_TESTPMD_PARAMS = ['-l 0,1,2,3,4 -n 4 --socket-mem 512 -- '
762 '--burst=64 -i --txqflags=0xf00 '
763 '--nb-cores=4 --rxq=2 --txq=2 '
766 **NOTE:** The guest SMP cores must be configured to allow for testpmd to use the
767 optimal number of cores to take advantage of the multiple guest queues.
769 In case of using Vanilla OVS and qemu virtio-net you can increase performance
770 by binding vhost-net threads to cpus. This can be done by enabling the affinity
771 in the ''04_vnf.conf'' file. This can be done to non multi-queue enabled
772 configurations as well as there will be 2 vhost-net threads.
774 .. code-block:: python
776 VSWITCH_VHOST_NET_AFFINITIZATION = True
778 VSWITCH_VHOST_CPU_MAP = [4,5,8,11]
780 **NOTE:** This method of binding would require a custom script in a real
783 **NOTE:** For optimal performance guest SMPs and/or vhost-net threads should be
784 on the same numa as the NIC in use if possible/applicable. Testpmd should be
785 assigned at least (nb_cores +1) total cores with the cpu mask.
790 VSPERF provides options to support jumbo frame testing with a jumbo frame supported
791 NIC and traffic generator for the following vswitches:
797 3. TestPMD loopback with or without a guest
799 **NOTE:** There is currently no support for SR-IOV or VPP at this time with jumbo
802 All packet forwarding applications for pxp testing is supported.
804 To enable jumbo frame testing simply enable the option in the conf files and set the
805 maximum size that will be used.
807 .. code-block:: python
809 VSWITCH_JUMBO_FRAMES_ENABLED = True
810 VSWITCH_JUMBO_FRAMES_SIZE = 9000
812 To enable jumbo frame testing with OVSVanilla the NIC in test on the host must have
813 its mtu size changed manually using ifconfig or applicable tools:
815 .. code-block:: console
817 ifconfig eth1 mtu 9000 up
819 **NOTE:** To make the setting consistent across reboots you should reference the OS
820 documents as it differs from distribution to distribution.
822 To start a test for jumbo frames modify the conf file packet sizes or pass the option
823 through the VSPERF command line.
825 .. code-block:: python
827 TEST_PARAMS = {'TRAFFICGEN_PKT_SIZES':(2000,9000)}
829 .. code-block:: python
831 ./vsperf --test-params "TRAFFICGEN_PKT_SIZES=2000,9000"
833 It is recommended to increase the memory size for OvsDpdkVhostUser testing from the default
834 1024. Your size required may vary depending on the number of guests in your testing. 4096
835 appears to work well for most typical testing scenarios.
837 .. code-block:: python
839 DPDK_SOCKET_MEM = ['4096', '0']
841 **NOTE:** For Jumbo frames to work with DpdkVhostUser, mergable buffers will be enabled by
842 default. If testing with mergable buffers in QEMU is desired, disable Jumbo Frames and only
843 test non jumbo frame sizes. Test Jumbo Frames sizes separately to avoid this collision.
846 Executing Packet Forwarding tests
847 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
849 To select the applications which will forward packets,
850 the following parameters should be configured:
852 .. code-block:: python
857 or use ``--vswitch`` and ``--fwdapp`` CLI arguments:
859 .. code-block:: console
861 $ ./vsperf phy2phy_cont --conf-file user_settings.py \
865 Supported Packet Forwarding applications are:
867 .. code-block:: console
869 'testpmd' - testpmd from dpdk
872 1. Update your ''10_custom.conf'' file to use the appropriate variables
873 for selected Packet Forwarder:
875 .. code-block:: python
877 # testpmd configuration
879 # packet forwarding mode supported by testpmd; Please see DPDK documentation
880 # for comprehensive list of modes supported by your version.
881 # e.g. io|mac|mac_retry|macswap|flowgen|rxonly|txonly|csum|icmpecho|...
882 # Note: Option "mac_retry" has been changed to "mac retry" since DPDK v16.07
883 TESTPMD_FWD_MODE = 'csum'
884 # checksum calculation layer: ip|udp|tcp|sctp|outer-ip
885 TESTPMD_CSUM_LAYER = 'ip'
886 # checksum calculation place: hw (hardware) | sw (software)
887 TESTPMD_CSUM_CALC = 'sw'
888 # recognize tunnel headers: on|off
889 TESTPMD_CSUM_PARSE_TUNNEL = 'off'
893 .. code-block:: console
895 $ ./vsperf phy2phy_tput --conf-file <path_to_settings_py>
897 Executing Packet Forwarding tests with one guest
898 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
900 TestPMD with DPDK 16.11 or greater can be used to forward packets as a switch to a single guest using TestPMD vdev
901 option. To set this configuration the following parameters should be used.
903 .. code-block:: python
908 or use ``--vswitch`` and ``--fwdapp`` CLI arguments:
910 .. code-block:: console
912 $ ./vsperf pvp_tput --conf-file user_settings.py \
916 Guest forwarding application only supports TestPMD in this configuration.
918 .. code-block:: python
920 GUEST_LOOPBACK = ['testpmd']
922 For optimal performance one cpu per port +1 should be used for TestPMD. Also set additional params for packet forwarding
923 application to use the correct number of nb-cores.
925 .. code-block:: python
927 DPDK_SOCKET_MEM = ['1024', '0']
928 VSWITCHD_DPDK_ARGS = ['-l', '46,44,42,40,38', '-n', '4']
929 TESTPMD_ARGS = ['--nb-cores=4', '--txq=1', '--rxq=1']
931 For guest TestPMD 3 VCpus should be assigned with the following TestPMD params.
933 .. code-block:: python
935 GUEST_TESTPMD_PARAMS = ['-l 0,1,2 -n 4 --socket-mem 1024 -- '
936 '--burst=64 -i --txqflags=0xf00 '
937 '--disable-hw-vlan --nb-cores=2 --txq=1 --rxq=1']
939 Execution of TestPMD can be run with the following command line
941 .. code-block:: console
943 ./vsperf pvp_tput --vswitch=none --fwdapp=TestPMD --conf-file <path_to_settings_py>
945 **NOTE:** To achieve the best 0% loss numbers with rfc2544 throughput testing, other tunings should be applied to host
946 and guest such as tuned profiles and CPU tunings to prevent possible interrupts to worker threads.
948 VSPERF modes of operation
949 ^^^^^^^^^^^^^^^^^^^^^^^^^
951 VSPERF can be run in different modes. By default it will configure vSwitch,
952 traffic generator and VNF. However it can be used just for configuration
953 and execution of traffic generator. Another option is execution of all
954 components except traffic generator itself.
956 Mode of operation is driven by configuration parameter -m or --mode
958 .. code-block:: console
960 -m MODE, --mode MODE vsperf mode of operation;
962 "normal" - execute vSwitch, VNF and traffic generator
963 "trafficgen" - execute only traffic generator
964 "trafficgen-off" - execute vSwitch and VNF
965 "trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission
967 In case, that VSPERF is executed in "trafficgen" mode, then configuration
968 of traffic generator can be modified through ``TRAFFIC`` dictionary passed to the
969 ``--test-params`` option. It is not needed to specify all values of ``TRAFFIC``
970 dictionary. It is sufficient to specify only values, which should be changed.
971 Detailed description of ``TRAFFIC`` dictionary can be found at
972 :ref:`configuration-of-traffic-dictionary`.
974 Example of execution of VSPERF in "trafficgen" mode:
976 .. code-block:: console
978 $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \
979 --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}"
984 The ``--matrix`` command line argument analyses and displays the performance of
985 all the tests run. Using the metric specified by ``MATRIX_METRIC`` in the conf-file,
986 the first test is set as the baseline and all the other tests are compared to it.
987 The ``MATRIX_METRIC`` must always refer to a numeric value to enable comparision.
988 A table, with the test ID, metric value, the change of the metric in %, testname
989 and the test parameters used for each test, is printed out as well as saved into the
992 Example of 2 tests being compared using Performance Matrix:
994 .. code-block:: console
996 $ ./vsperf --conf-file user_settings.py \
997 --test-params "['TRAFFICGEN_PKT_SIZES=(64,)',"\
998 "'TRAFFICGEN_PKT_SIZES=(128,)']" \
999 phy2phy_cont phy2phy_cont --matrix
1003 .. code-block:: console
1005 +------+--------------+---------------------+----------+---------------------------------------+
1006 | ID | Name | throughput_rx_fps | Change | Parameters, CUMULATIVE_PARAMS = False |
1007 +======+==============+=====================+==========+=======================================+
1008 | 0 | phy2phy_cont | 23749000.000 | 0 | 'TRAFFICGEN_PKT_SIZES': [64] |
1009 +------+--------------+---------------------+----------+---------------------------------------+
1010 | 1 | phy2phy_cont | 16850500.000 | -29.048 | 'TRAFFICGEN_PKT_SIZES': [128] |
1011 +------+--------------+---------------------+----------+---------------------------------------+
1014 Code change verification by pylint
1015 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1017 Every developer participating in VSPERF project should run
1018 pylint before his python code is submitted for review. Project
1019 specific configuration for pylint is available at 'pylint.rc'.
1021 Example of manual pylint invocation:
1023 .. code-block:: console
1025 $ pylint --rcfile ./pylintrc ./vsperf
1030 Custom image fails to boot
1031 ~~~~~~~~~~~~~~~~~~~~~~~~~~
1033 Using custom VM images may not boot within VSPerf pxp testing because of
1034 the drive boot and shared type which could be caused by a missing scsi
1035 driver inside the image. In case of issues you can try changing the drive
1038 .. code-block:: python
1040 GUEST_BOOT_DRIVE_TYPE = ['ide']
1041 GUEST_SHARED_DRIVE_TYPE = ['ide']
1043 OVS with DPDK and QEMU
1044 ~~~~~~~~~~~~~~~~~~~~~~~
1046 If you encounter the following error: "before (last 100 chars):
1047 '-path=/dev/hugepages,share=on: unable to map backing store for
1048 hugepages: Cannot allocate memory\r\n\r\n" during qemu initialization,
1049 check the amount of hugepages on your system:
1051 .. code-block:: console
1053 $ cat /proc/meminfo | grep HugePages
1056 By default the vswitchd is launched with 1Gb of memory, to change
1057 this, modify --socket-mem parameter in conf/02_vswitch.conf to allocate
1058 an appropriate amount of memory:
1060 .. code-block:: python
1062 DPDK_SOCKET_MEM = ['1024', '0']
1063 VSWITCHD_DPDK_ARGS = ['-c', '0x4', '-n', '4']
1064 VSWITCHD_DPDK_CONFIG = {
1065 'dpdk-init' : 'true',
1066 'dpdk-lcore-mask' : '0x4',
1067 'dpdk-socket-mem' : '1024,0',
1070 Note: Option ``VSWITCHD_DPDK_ARGS`` is used for vswitchd, which supports ``--dpdk``
1071 parameter. In recent vswitchd versions, option ``VSWITCHD_DPDK_CONFIG`` will be
1072 used to configure vswitchd via ``ovs-vsctl`` calls.
1078 For more information and details refer to the rest of vSwitchPerfuser documentation.