1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
3 .. http://creativecommons.org/licenses/by/4.0
9 About Packet Forwarding
10 -----------------------
12 Packet Forwarding is a test suite of KVMFORNFV. These latency tests measures the time taken by a
13 **Packet** generated by the traffic generator to travel from the originating device through the
14 network to the destination device. Packet Forwarding is implemented using test framework
15 implemented by OPNFV VSWITCHPERF project and an ``IXIA Traffic Generator``.
20 +-----------------------------+---------------------------------------------------+
22 | **Release** | **Features** |
24 +=============================+===================================================+
25 | | - Packet Forwarding is not part of Colorado |
26 | Colorado | release of KVMFORNFV |
28 +-----------------------------+---------------------------------------------------+
29 | | - Packet Forwarding is a testcase in KVMFORNFV |
30 | | - Implements three scenarios (Host/Guest/SRIOV) |
31 | | as part of testing in KVMFORNFV |
32 | Danube | - Uses automated test framework of OPNFV |
33 | | VSWITCHPERF software (PVP/PVVP) |
35 | | - Works with IXIA Traffic Generator |
36 +-----------------------------+---------------------------------------------------+
41 VSPerf is an OPNFV testing project.
42 VSPerf will develop a generic and architecture agnostic vSwitch testing framework and associated
43 tests, that will serve as a basis for validating the suitability of different vSwitch
44 implementations in a Telco NFV deployment environment. The output of this project will be utilized
45 by the OPNFV Performance and Test group and its associated projects, as part of OPNFV Platform and
46 VNF level testing and validation.
48 For complete VSPERF documentation go to `link.`_
50 .. _link.: http://artifacts.opnfv.org/vswitchperf/colorado/index.html
56 Guidelines of installating `VSPERF`_.
58 .. _VSPERF: http://artifacts.opnfv.org/vswitchperf/colorado/configguide/index.html
60 Supported Operating Systems
61 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
73 The vSwitch must support Open Flow 1.3 or greater.
75 * OVS (built from source).
76 * OVS with DPDK (built from source).
86 The test suite requires Python 3.3 and relies on a number of other
87 packages. These need to be installed for the test suite to function.
89 Installation of required packages, preparation of Python 3 virtual
90 environment and compilation of OVS, DPDK and QEMU is performed by
91 script **systems/build_base_machine.sh**. It should be executed under
92 user account, which will be used for vsperf execution.
94 **Please Note:** Password-less sudo access must be configured for given user
95 before script is executed.
97 Execution of installation script:
103 $ ./build_base_machine.sh
105 Script **build_base_machine.sh** will install all the vsperf dependencies
106 in terms of system packages, Python 3.x and required Python modules.
107 In case of CentOS 7 it will install Python 3.3 from an additional repository
108 provided by Software Collections (`a link`_). In case of RedHat 7 it will
109 install Python 3.4 as an alternate installation in /usr/local/bin. Installation
110 script will also use `virtualenv`_ to create a vsperf virtual environment,
111 which is isolated from the default Python environment. This environment will
112 reside in a directory called **vsperfenv** in $HOME.
114 You will need to activate the virtual environment every time you start a
115 new shell session. Its activation is specific to your OS:
117 For running testcases VSPERF is installed on Intel pod1-node2 in which centos
118 operating system is installed. Only VSPERF installion on Centos is discussed here.
119 For installation steps on other operating systems please refer to `here`_.
121 .. _here: http://artifacts.opnfv.org/vswitchperf/colorado/configguide/index.html
128 To avoid file permission errors and Python version issues, use virtualenv to create an isolated environment with Python3.
129 The required Python 3 packages can be found in the `requirements.txt` file in the root of the test suite.
130 They can be installed in your virtual environment like so:
134 scl enable python33 bash
135 # Create virtual environment
139 pip install -r requirements.txt
142 You need to activate the virtual environment every time you start a new shell session.
143 To activate, simple run:
147 scl enable python33 bash
152 Working Behind a Proxy
153 ~~~~~~~~~~~~~~~~~~~~~~
155 If you're behind a proxy, you'll likely want to configure this before running any of the above. For example:
159 export http_proxy="http://<username>:<password>@<proxy>:<port>/";
160 export https_proxy="https://<username>:<password>@<proxy>:<port>/";
161 export ftp_proxy="ftp://<username>:<password>@<proxy>:<port>/";
162 export socks_proxy="socks://<username>:<password>@<proxy>:<port>/";
164 .. _a link: http://www.softwarecollections.org/en/scls/rhscl/python33/
165 .. _virtualenv: https://virtualenv.readthedocs.org/en/latest/
167 For other OS specific activation click `this link`_:
169 .. _this link: http://artifacts.opnfv.org/vswitchperf/colorado/configguide/installation.html#other-requirements
174 VSPERF supports many Traffic-generators. For configuring VSPERF to work with the available traffic-generator go through `this`_.
176 .. _this: http://artifacts.opnfv.org/vswitchperf/colorado/configguide/trafficgen.html
178 VSPERF supports the following traffic generators:
180 * Dummy (DEFAULT): Allows you to use your own external
182 * IXIA (IxNet and IxOS)
187 To see the list of traffic gens from the cli:
189 .. code-block:: console
191 $ ./vsperf --list-trafficgens
193 This guide provides the details of how to install
194 and configure the various traffic generators.
196 As KVM4NFV uses only IXIA traffic generator, it is discussed here. For complete documentation regarding traffic generators please follow this `link`_.
198 .. _link: https://gerrit.opnfv.org/gerrit/gitweb?p=vswitchperf.git;a=blob;f=docs/configguide/trafficgen.rst;h=85fc35b886d30db3b92a6b7dcce7ca742b70cbdc;hb=HEAD
203 Hardware Requirements
204 ~~~~~~~~~~~~~~~~~~~~~
206 VSPERF requires the following hardware to run tests: IXIA traffic generator (IxNetwork), a machine that
207 runs the IXIA client software and a CentOS Linux release 7.1.1503 (Core) host.
212 Follow the [installation instructions] to install.
214 On the CentOS 7 system
215 ~~~~~~~~~~~~~~~~~~~~~~
217 You need to install IxNetworkTclClient$(VER_NUM)Linux.bin.tgz.
219 On the IXIA client software system
220 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
222 Find the IxNetwork TCL server app (start -> All Programs -> IXIA -> IxNetwork -> IxNetwork_$(VER_NUM) -> IxNetwork TCL Server)
223 - Right click on IxNetwork TCL Server, select properties
224 - Under shortcut tab in the Target dialogue box make sure there is the argument "-tclport xxxx"
226 where xxxx is your port number (take note of this port number you will need it for the 10_custom.conf file).
228 .. figure:: images/IXIA1.png
233 - Hit Ok and start the TCL server application
238 There are several configuration options specific to the IxNetworks traffic generator
239 from IXIA. It is essential to set them correctly, before the VSPERF is executed
242 Detailed description of options follows:
244 * TRAFFICGEN_IXNET_MACHINE - IP address of server, where IxNetwork TCL Server is running
245 * TRAFFICGEN_IXNET_PORT - PORT, where IxNetwork TCL Server is accepting connections from
247 * TRAFFICGEN_IXNET_USER - username, which will be used during communication with IxNetwork
248 TCL Server and IXIA chassis
249 * TRAFFICGEN_IXIA_HOST - IP address of IXIA traffic generator chassis
250 * TRAFFICGEN_IXIA_CARD - identification of card with dedicated ports at IXIA chassis
251 * TRAFFICGEN_IXIA_PORT1 - identification of the first dedicated port at TRAFFICGEN_IXIA_CARD
252 at IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of
253 unidirectional traffic, it is essential to correctly connect 1st IXIA port to the 1st NIC
254 at DUT, i.e. to the first PCI handle from WHITELIST_NICS list. Otherwise traffic may not
255 be able to pass through the vSwitch.
256 * TRAFFICGEN_IXIA_PORT2 - identification of the second dedicated port at TRAFFICGEN_IXIA_CARD
257 at IXIA chassis; VSPERF uses two separated ports for traffic generation. In case of
258 unidirectional traffic, it is essential to correctly connect 2nd IXIA port to the 2nd NIC
259 at DUT, i.e. to the second PCI handle from WHITELIST_NICS list. Otherwise traffic may not
260 be able to pass through the vSwitch.
261 * TRAFFICGEN_IXNET_LIB_PATH - path to the DUT specific installation of IxNetwork TCL API
262 * TRAFFICGEN_IXNET_TCL_SCRIPT - name of the TCL script, which VSPERF will use for
263 communication with IXIA TCL server
264 * TRAFFICGEN_IXNET_TESTER_RESULT_DIR - folder accessible from IxNetwork TCL server,
265 where test results are stored, e.g. ``c:/ixia_results``; see test-results-share_
266 * TRAFFICGEN_IXNET_DUT_RESULT_DIR - directory accessible from the DUT, where test
267 results from IxNetwork TCL server are stored, e.g. ``/mnt/ixia_results``; see
270 .. _test-results-share:
275 VSPERF is not able to retrieve test results via TCL API directly. Instead, all test
276 results are stored at IxNetwork TCL server. Results are stored at folder defined by
277 ``TRAFFICGEN_IXNET_TESTER_RESULT_DIR`` configuration parameter. Content of this
278 folder must be shared (e.g. via samba protocol) between TCL Server and DUT, where
279 VSPERF is executed. VSPERF expects, that test results will be available at directory
280 configured by ``TRAFFICGEN_IXNET_DUT_RESULT_DIR`` configuration parameter.
282 Example of sharing configuration:
284 * Create a new folder at IxNetwork TCL server machine, e.g. ``c:\ixia_results``
285 * Modify sharing options of ``ixia_results`` folder to share it with everybody
286 * Create a new directory at DUT, where shared directory with results
287 will be mounted, e.g. ``/mnt/ixia_results``
288 * Update your custom VSPERF configuration file as follows:
290 .. code-block:: python
292 TRAFFICGEN_IXNET_TESTER_RESULT_DIR = 'c:/ixia_results'
293 TRAFFICGEN_IXNET_DUT_RESULT_DIR = '/mnt/ixia_results'
295 Note: It is essential to use slashes '/' also in path
296 configured by ``TRAFFICGEN_IXNET_TESTER_RESULT_DIR`` parameter.
298 * Install cifs-utils package.
300 e.g. at rpm based Linux distribution:
302 .. code-block:: console
304 yum install cifs-utils
306 * Mount shared directory, so VSPERF can access test results.
308 e.g. by adding new record into ``/etc/fstab``
310 .. code-block:: console
312 mount -t cifs //_TCL_SERVER_IP_OR_FQDN_/ixia_results /mnt/ixia_results
313 -o file_mode=0777,dir_mode=0777,nounix
315 It is recommended to verify, that any new file inserted into ``c:/ixia_results`` folder
316 is visible at DUT inside ``/mnt/ixia_results`` directory.
319 Cloning and building src dependencies
320 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
322 In order to run VSPERF, you will need to download DPDK and OVS. You can do this manually and build
323 them in a preferred location, or you could use vswitchperf/src. The vswitchperf/src directory
324 contains makefiles that will allow you to clone and build the libraries that VSPERF depends on,
325 such as DPDK and OVS. To clone and build simply:
332 To delete a src subdirectory and its contents to allow you to re-clone simply use:
338 Configure the `./conf/10_custom.conf` file
339 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
341 The supplied `10_custom.conf` file must be modified, as it contains configuration items for which there are no reasonable default values.
343 The configuration items that can be added is not limited to the initial contents. Any configuration item
344 mentioned in any .conf file in `./conf` directory can be added and that item will be overridden by the custom
347 Using a custom settings file
348 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
350 Alternatively a custom settings file can be passed to `vsperf` via the `--conf-file` argument.
354 ./vsperf --conf-file <path_to_settings_py> ...
356 Note that configuration passed in via the environment (`--load-env`) or via another command line
357 argument will override both the default and your custom configuration files. This
358 "priority hierarchy" can be described like so (1 = max priority):
360 1. Command line arguments
361 2. Environment variables
362 3. Configuration file(s)
367 VSPERF uses a VM image called vloop_vnf for looping traffic in the deployment
368 scenarios involving VMs. The image can be downloaded from
369 `<http://artifacts.opnfv.org/>`__.
371 Please see the installation instructions for information on :ref:`vloop-vnf`
379 A Kernel Module that provides OSI Layer 2 Ipv4 termination or forwarding with
380 support for Destination Network Address Translation (DNAT) for both the MAC and
381 IP addresses. l2fwd can be found in <vswitchperf_dir>/src/l2fwd
383 .. figure:: images/Guest_Scenario.png
384 :name: Guest_Scenario
392 Before running any tests make sure you have root permissions by adding the following line to /etc/sudoers:
395 username ALL=(ALL) NOPASSWD: ALL
397 username in the example above should be replaced with a real username.
399 To list the available tests:
403 ./vsperf --list-tests
406 To run a group of tests, for example all tests with a name containing
411 ./vsperf --conf-file=user_settings.py --tests="RFC2544"
417 ./vsperf --conf-file=user_settings.py
419 Some tests allow for configurable parameters, including test duration (in seconds) as well as packet sizes (in bytes).
423 ./vsperf --conf-file user_settings.py
425 --test-param` "rfc2544_duration=10;packet_sizes=128"
427 For all available options, check out the help dialog:
437 Available Tests in VSPERF are:
442 * phy2phy_tput_mod_vlan
447 * phy2phy_scalability
455 VSPERF modes of operation
456 --------------------------
458 VSPERF can be run in different modes. By default it will configure vSwitch,
459 traffic generator and VNF. However it can be used just for configuration
460 and execution of traffic generator. Another option is execution of all
461 components except traffic generator itself.
463 Mode of operation is driven by configuration parameter -m or --mode
465 .. code-block:: console
467 -m MODE, --mode MODE vsperf mode of operation;
469 "normal" - execute vSwitch, VNF and traffic generator
470 "trafficgen" - execute only traffic generator
471 "trafficgen-off" - execute vSwitch and VNF
472 "trafficgen-pause" - execute vSwitch and VNF but wait before traffic transmission
474 In case, that VSPERF is executed in "trafficgen" mode, then configuration
475 of traffic generator can be modified through ``TRAFFIC`` dictionary passed to the
476 ``--test-params`` option. It is not needed to specify all values of ``TRAFFIC``
477 dictionary. It is sufficient to specify only values, which should be changed.
478 Detailed description of ``TRAFFIC`` dictionary can be found at: ref:`configuration-of-traffic-dictionary`.
480 Example of execution of VSPERF in "trafficgen" mode:
482 .. code-block:: console
484 $ ./vsperf -m trafficgen --trafficgen IxNet --conf-file vsperf.conf \
485 --test-params "TRAFFIC={'traffic_type':'rfc2544_continuous','bidir':'False','framerate':60}"
488 Packet Forwarding Test Scenarios
489 --------------------------------
491 KVMFORNFV currently implements three scenarios as part of testing:
498 Packet Forwarding Host Scenario
499 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
501 Here host DUT has VSPERF installed in it and is properly configured to use IXIA Traffic-generator
502 by providing IXIA CARD, PORTS and Lib paths along with IP.
503 please refer to figure.2
505 .. figure:: images/Host_Scenario.png
510 Packet Forwarding Guest Scenario
511 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
513 Here the guest is a Virtual Machine (VM) launched by using vloop_vnf provided by vsperf project
514 on host/DUT using Qemu. In this latency test the time taken by the frame/packet to travel from the
515 originating device through network involving a guest to destination device is calculated.
516 The resulting latency values will define the performance of installed kernel.
518 .. figure:: images/Guest_Scenario.png
519 :name: Guest_Scenario
523 Packet Forwarding SRIOV Scenario
524 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
526 In this test the packet generated at the IXIA is forwarded to the Guest VM launched on Host by
527 implementing SR-IOV interface at NIC level of host .i.e., DUT. The time taken by the packet to
528 travel through the network to the destination the IXIA traffic-generator is calculated and
529 published as a test result for this scenario.
531 SRIOV-support_ is given below, it details how to use SR-IOV.
533 .. figure:: images/SRIOV_Scenario.png
534 :name: SRIOV_Scenario
538 Using vfio_pci with DPDK
539 ~~~~~~~~~~~~~~~~~~~~~~~~~
541 To use vfio with DPDK instead of igb_uio add into your custom configuration
542 file the following parameter:
544 .. code-block:: python
546 PATHS['dpdk']['src']['modules'] = ['uio', 'vfio-pci']
549 **NOTE:** In case, that DPDK is installed from binary package, then please
551 set ``PATHS['dpdk']['bin']['modules']`` instead.
553 **NOTE:** Please ensure that Intel VT-d is enabled in BIOS.
555 **NOTE:** Please ensure your boot/grub parameters include
558 .. code-block:: console
560 iommu=pt intel_iommu=on
562 To check that IOMMU is enabled on your platform:
564 .. code-block:: console
567 [ 0.000000] Intel-IOMMU: enabled
568 [ 0.139882] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
569 [ 0.139888] dmar: IOMMU 1: reg_base_addr ebffc000 ver 1:0 cap d2078c106f0466 ecap f020de
570 [ 0.139893] IOAPIC id 2 under DRHD base 0xfbffe000 IOMMU 0
571 [ 0.139894] IOAPIC id 0 under DRHD base 0xebffc000 IOMMU 1
572 [ 0.139895] IOAPIC id 1 under DRHD base 0xebffc000 IOMMU 1
573 [ 3.335744] IOMMU: dmar0 using Queued invalidation
574 [ 3.335746] IOMMU: dmar1 using Queued invalidation
582 To use virtual functions of NIC with SRIOV support, use extended form
583 of NIC PCI slot definition:
585 .. code-block:: python
587 WHITELIST_NICS = ['0000:03:00.0|vf0', '0000:03:00.1|vf3']
589 Where ``vf`` is an indication of virtual function usage and following
590 number defines a VF to be used. In case that VF usage is detected,
591 then vswitchperf will enable SRIOV support for given card and it will
592 detect PCI slot numbers of selected VFs.
594 So in example above, one VF will be configured for NIC '0000:05:00.0'
595 and four VFs will be configured for NIC '0000:05:00.1'. Vswitchperf
596 will detect PCI addresses of selected VFs and it will use them during
599 At the end of vswitchperf execution, SRIOV support will be disabled.
601 SRIOV support is generic and it can be used in different testing scenarios.
605 * vSwitch tests with DPDK or without DPDK support to verify impact
606 of VF usage on vSwitch performance
607 * tests without vSwitch, where traffic is forwared directly
608 between VF interfaces by packet forwarder (e.g. testpmd application)
609 * tests without vSwitch, where VM accesses VF interfaces directly
610 by PCI-passthrough to measure raw VM throughput performance.
612 Using QEMU with PCI passthrough support
613 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
615 Raw virtual machine throughput performance can be measured by execution of PVP
616 test with direct access to NICs by PCI passthrough. To execute VM with direct
617 access to PCI devices, enable vfio-pci_. In order to use virtual functions,
618 SRIOV-support_ must be enabled.
620 Execution of test with PCI passthrough with vswitch disabled:
622 .. code-block:: console
624 $ ./vsperf --conf-file=<path_to_custom_conf>/10_custom.conf \
625 --vswitch none --vnf QemuPciPassthrough pvp_tput
627 Any of supported guest-loopback-application_ can be used inside VM with
628 PCI passthrough support.
630 Note: Qemu with PCI passthrough support can be used only with PVP test
633 .. _guest-loopback-application: