1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, 2016-2019 Intel Corporation.
6 Convention for heading levels in Yardstick documentation:
8 ======= Heading 0 (reserved for the title in a document)
14 Avoid deeper levels because they do not render well.
16 Yardstick - NSB Testing - Operation
17 ===================================
22 NSB test configuration and OpenStack setup requirements
25 OpenStack Network Configuration
26 -------------------------------
28 NSB requires certain OpenStack deployment configurations.
29 For optimal VNF characterization using external traffic generators NSB requires
30 provider/external networks.
36 The VNFs require a clear L2 connect to the external network in order to
37 generate realistic traffic from multiple address ranges and ports.
39 In order to prevent Neutron from filtering traffic we have to disable Neutron
40 Port Security. We also disable DHCP on the data ports because we are binding
41 the ports to DPDK and do not need DHCP addresses. We also disable gateways
42 because multiple default gateways can prevent SSH access to the VNF from the
43 floating IP. We only want a gateway on the mgmt network
50 port_security_enabled: False
56 By default Heat will attach every node to every Neutron network that is
57 created. For scale-out tests we do not want to attach every node to every
60 For each node you can specify which ports are on which network using the
61 network_ports dictionary.
63 In this example we have ``TRex xe0 <-> xe0 VNF xe1 <-> xe0 UDP_Replay``
85 # Trex always needs two ports
101 The configuration of the availability zone is requred in cases where location
102 of exact compute host/group of compute hosts needs to be specified for
103 :term:`SampleVNF` or traffic generator in the heat test case. If this is the
104 case, please follow the instructions below.
106 .. _`Create a host aggregate`:
108 1. Create a host aggregate in the OpenStack and add the available compute hosts
109 into the aggregate group.
111 .. note:: Change the ``<AZ_NAME>`` (availability zone name), ``<AGG_NAME>``
112 (host aggregate name) and ``<HOST>`` (host name of one of the compute) in the
117 # create host aggregate
118 openstack aggregate create --zone <AZ_NAME> \
119 --property availability_zone=<AZ_NAME> <AGG_NAME>
120 # show available hosts
121 openstack compute service list --service nova-compute
122 # add selected host into the host aggregate
123 openstack aggregate add host <AGG_NAME> <HOST>
125 2. To specify the OpenStack location (the exact compute host or group of the hosts)
126 of SampleVNF or traffic generator in the heat test case, the ``availability_zone`` server
127 configuration option should be used. For example:
129 .. note:: The ``<AZ_NAME>`` (availability zone name) should be changed according
130 to the name used during the host aggregate creation steps above.
136 image: yardstick-samplevnfs
141 availability_zone: <AZ_NAME>
145 availability_zone: <AZ_NAME>
150 There are two example of SampleVNF scale out test case which use the
151 ``availability zone`` feature to specify the exact location of scaled VNFs and
156 .. code-block:: console
158 <repo>/samples/vnf_samples/nsut/prox/tc_prox_heat_context_l2fwd_multiflow-2-scale-out.yaml
159 <repo>/samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale_out.yaml
161 .. note:: This section describes the PROX scale-out testcase, but the same
162 procedure is used for the vFW test case.
164 1. Before running the scale-out test case, make sure the host aggregates are
165 configured in the OpenStack environment. To check this, run the following
168 .. code-block:: console
170 # show configured host aggregates (example)
171 openstack aggregate list
172 +----+------+-------------------+
173 | ID | Name | Availability Zone |
174 +----+------+-------------------+
175 | 4 | agg0 | AZ_NAME_0 |
176 | 5 | agg1 | AZ_NAME_1 |
177 +----+------+-------------------+
179 2. If no host aggregates are configured, please follow the instructions to
180 `Create a host aggregate`_
183 3. Run the SampleVNF PROX scale-out test case, specifying the
184 ``availability zone`` of each VNF and traffic generator as task arguments.
186 .. note:: The ``az_0`` and ``az_1`` should be changed according to the host
187 aggregates created in the OpenStack.
189 .. code-block:: console
191 yardstick -d task start \
192 <repo>/samples/vnf_samples/nsut/prox/tc_prox_heat_context_l2fwd_multiflow-2-scale-out.yaml\
194 "num_vnfs": 4, "availability_zone": {
195 "vnf_0": "az_0", "tg_0": "az_1",
196 "vnf_1": "az_0", "tg_1": "az_1",
197 "vnf_2": "az_0", "tg_2": "az_1",
198 "vnf_3": "az_0", "tg_3": "az_1"
202 ``num_vnfs`` specifies how many VNFs are going to be deployed in the
203 ``heat`` contexts. ``vnf_X`` and ``tg_X`` arguments configure the
204 availability zone where the VNF and traffic generator is going to be deployed.
210 NSB can collect KPIs from collected. We have support for various plugins
211 enabled by the :term:`Barometer` project.
213 The default yardstick-samplevnf has collectd installed. This allows for
214 collecting KPIs from the VNF.
216 Collecting KPIs from the NFVi is more complicated and requires manual setup.
217 We assume that collectd is not installed on the compute nodes.
219 To collectd KPIs from the NFVi compute nodes:
221 * install_collectd on the compute nodes
222 * create pod.yaml for the compute nodes
223 * enable specific plugins depending on the vswitch and DPDK
225 example ``pod.yaml`` section for Compute node running collectd.
244 ovs_socket_path: /var/run/openvswitch/db.sock
252 VNFs performance data with scale-up
254 * Helps to figure out optimal number of cores specification in the Virtual
255 Machine template creation or VNF
256 * Helps in comparison between different VNF vendor offerings
257 * Better the scale-up index, indicates the performance scalability of a
262 For VNF scale-up tests we increase the number for VNF worker threads. In the
263 case of VNFs we also need to increase the number of VCPUs and memory allocated
266 An example scale-up Heat testcase is:
268 .. literalinclude:: /../samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_trex_scale-up.yaml
269 :language: yaml+jinja
271 This testcase template requires specifying the number of VCPUs, Memory and Ports.
272 We set the VCPUs and memory using the ``--task-args`` options
274 .. code-block:: console
276 yardstick task start --task-args='{"mem": 10480, "vcpus": 4, "vports": 2}' \
277 samples/vnf_samples/nsut/vfw/tc_heat_rfc2544_ipv4_1rule_1flow_64B_trex_scale-up.yaml
279 In order to support ports scale-up, traffic and topology templates need to be used in testcase.
281 A example topology template is:
283 .. literalinclude:: /../samples/vnf_samples/nsut/vfw/vfw-tg-topology-scale-up.yaml
284 :language: yaml+jinja
286 This template has ``vports`` as an argument. To pass this argument it needs to
287 be configured in ``extra_args`` scenario definition. Please note that more
288 argument can be defined in that section. All of them will be passed to topology
289 and traffic profile templates
295 schema: yardstick:task:0.1
298 traffic_profile: ../../traffic_profiles/ipv4_throughput-scale-up.yaml
301 topology: vfw-tg-topology-scale-up.yaml
303 A example traffic profile template is:
305 .. literalinclude:: /../samples/vnf_samples/traffic_profiles/ipv4_throughput-scale-up.yaml
306 :language: yaml+jinja
308 There is an option to provide predefined config for SampleVNFs. Path to config
309 file may by specified in ``vnf_config`` scenario section.
314 rules: acl_1rule.yaml
315 vnf_config: {lb_config: 'SW', file: vfw_vnf_pipeline_cores_4_ports_2_lb_1_sw.conf }
320 1. Follow above traffic generator section to setup.
321 2. Edit num of threads in
322 ``<repo>/samples/vnf_samples/nsut/vfw/tc_baremetal_rfc2544_ipv4_trex_scale_up.yaml``
323 e.g, 6 Threads for given VNF
325 .. code-block:: yaml+jinja
327 schema: yardstick:task:0.1
329 {% for worker_thread in [1, 2 ,3 , 4, 5, 6] %}
331 traffic_profile: ../../traffic_profiles/ipv4_throughput.yaml
332 topology: vfw-tg-topology.yaml
334 tg__0: trafficgen_0.yardstick
335 vnf__0: vnf_0.yardstick
341 src_ip: [{'tg__0': 'xe0'}]
342 dst_ip: [{'tg__0': 'xe1'}]
346 allowed_drop_rate: 0.0001 - 0.0001
348 rules: acl_1rule.yaml
349 vnf_config: {lb_config: 'HW', lb_count: 1, worker_config: '1C/1T', worker_threads: {{worker_thread}}}
360 file: /etc/yardstick/nodes/pod.yaml
365 VNFs performance data with scale-out helps
367 * capacity planning to meet the given network node requirements
368 * comparison between different VNF vendor offerings
369 * better the scale-out index, provides the flexibility in meeting future
370 capacity requirements
376 Scale-out not supported on Baremetal.
378 1. Follow above traffic generator section to setup.
379 2. Generate testcase for standalone virtualization using ansible scripts
381 .. code-block:: console
384 trex: standalone_ovs_scale_out_test.yaml or standalone_sriov_scale_out_test.yaml
385 ixia: standalone_ovs_scale_out_ixia_test.yaml or standalone_sriov_scale_out_ixia_test.yaml
386 ixia_correlated: standalone_ovs_scale_out_ixia_correlated_test.yaml or standalone_sriov_scale_out_ixia_correlated_test.yaml
388 update the ovs_dpdk or sriov above Ansible scripts reflect the setup
392 .. code-block:: console
394 <repo>/samples/vnf_samples/nsut/tc_sriov_vfw_udp_ixia_correlated_scale_out-1.yaml
395 <repo>/samples/vnf_samples/nsut/tc_sriov_vfw_udp_ixia_correlated_scale_out-2.yaml
400 There are sample scale-out all-VM Heat tests. These tests only use VMs and
401 don't use external traffic.
403 The tests use UDP_Replay and correlated traffic.
405 .. code-block:: console
407 <repo>/samples/vnf_samples/nsut/cgnapt/tc_heat_rfc2544_ipv4_1flow_64B_trex_correlated_scale_4.yaml
409 To run the test you need to increase OpenStack CPU, Memory and Port quotas.
412 Traffic Generator tuning
413 ------------------------
415 The TRex traffic generator can be setup to use multiple threads per core, this
416 is for multiqueue testing.
418 TRex does not automatically enable multiple threads because we currently cannot
419 detect the number of queues on a device.
421 To enable multiple queue set the ``queues_per_port`` value in the TG VNF
429 tg__0: trafficgen_0.yardstick
436 Standalone configuration
437 ------------------------
439 NSB supports certain Standalone deployment configurations.
440 Standalone supports provisioning a VM in a standalone visualised environment using kvm/qemu.
441 There two types of Standalone contexts available: OVS-DPDK and SRIOV.
442 OVS-DPDK uses OVS network with DPDK drivers.
443 SRIOV enables network traffic to bypass the software switch layer of the Hyper-V stack.
445 Emulated machine type
446 ^^^^^^^^^^^^^^^^^^^^^
448 For better performance test results of emulated VM spawned by Yardstick SA
449 context (OvS-DPDK/SRIOV), it may be important to control the emulated machine
450 type used by QEMU emulator. This attribute can be configured via TC definition
451 in ``contexts`` section under ``extra_specs`` configuration.
459 - type: StandaloneSriov
465 machine_type: pc-i440fx-bionic
467 Where, ``machine_type`` can be set to one of the emulated machine type
468 supported by QEMU running on SUT platform. To get full list of supported
469 emulated machine types, the following command can be used on the target SUT
474 # qemu-system-x86_64 -machine ?
476 By default, the ``machine_type`` option is set to ``pc-i440fx-xenial`` which is
477 suitable for running Ubuntu 16.04 VM image. So, if this type is not supported
478 by the target platform or another VM image is used for stand alone (SA) context
479 VM (e.g.: ``bionic`` image for Ubuntu 18.04), this configuration should be
482 Standalone with OVS-DPDK
483 ^^^^^^^^^^^^^^^^^^^^^^^^
485 SampleVNF image is spawned in a VM on a baremetal server.
486 OVS with DPDK is installed on the baremetal server.
488 .. note:: Ubuntu 17.10 requires DPDK v.17.05 and higher, DPDK v.17.05 requires OVS v.2.8.0.
490 Default values for OVS-DPDK:
494 * pmd_cpu_mask: "0x6"
496 Sample test case file
497 ^^^^^^^^^^^^^^^^^^^^^
499 1. Prepare SampleVNF image and copy it to ``flavor/images``.
500 2. Prepare context files for TREX and SampleVNF under ``contexts/file``.
501 3. Add bridge named ``br-int`` to the baremetal where SampleVNF image is deployed.
502 4. Modify ``networks/phy_port`` accordingly to the baremetal setup.
505 .. literalinclude:: /../samples/vnf_samples/nsut/acl/tc_ovs_rfc2544_ipv4_1rule_1flow_trex.yaml
506 :language: yaml+jinja
508 Preparing test run of vEPC test case
509 ------------------------------------
511 Provided vEPC test cases are examples of emulation of vEPC infrastructure
512 components, such as UE, eNodeB, MME, SGW, PGW.
514 Location of vEPC test cases: ``samples/vnf_samples/nsut/vepc/``.
516 Before running a specific vEPC test case using NSB, some preconfiguration
519 Update Spirent Landslide TG configuration in pod file
520 =====================================================
522 Examples of ``pod.yaml`` files could be found in
523 :file:`etc/yardstick/nodes/standalone`.
524 The name of related pod file could be checked in the context section of NSB
527 The ``pod.yaml`` related to vEPC test case uses some sub-structures that hold the
528 details of accessing the Spirent Landslide traffic generator.
529 These subsections and the changes to be done in provided example pod file are
532 1. ``tas_manager``: data under this key holds the information required to
533 access Landslide TAS (Test Administration Server) and perform needed
534 configurations on it.
536 * ``ip``: IP address of TAS Manager node; should be updated according to test
538 * ``super_user``: superuser name; could be retrieved from Landslide documentation
539 * ``super_user_password``: superuser password; could be retrieved from
540 Landslide documentation
541 * ``cfguser_password``: password of predefined user named 'cfguser'; default
542 password could be retrieved from Landslide documentation
543 * ``test_user``: username to be used during test run as a Landslide library
544 name; to be defined by test run operator
545 * ``test_user_password``: password of test user; to be defined by test run
547 * ``proto``: *http* or *https*; to be defined by test run operator
548 * ``license``: Landslide license number installed on TAS
550 2. The ``config`` section holds information about test servers (TSs) and
551 systems under test (SUTs). Data is represented as a list of entries.
552 Each such entry contains:
554 * ``test_server``: this subsection represents data related to test server
555 configuration, such as:
557 * ``name``: test server name; unique custom name to be defined by test
559 * ``role``: this value is used as a key to bind specific Test Server and
560 TestCase; should be set to one of test types supported by TAS license
561 * ``ip``: Test Server IP address
562 * ``thread_model``: parameter related to Test Server performance mode.
563 The value should be one of the following: "Legacy" | "Max" | "Fireball".
564 Refer to Landslide documentation for details.
565 * ``phySubnets``: a structure used to specify IP ranges reservations on
566 specific network interfaces of related Test Server. Structure fields are:
568 * ``base``: start of IP address range
569 * ``mask``: IP range mask in CIDR format
570 * ``name``: network interface name, e.g. *eth1*
571 * ``numIps``: size of IP address range
573 * ``preResolvedArpAddress``: a structure used to specify the range of IP
574 addresses for which the ARP responses will be emulated
576 * ``StartingAddress``: IP address specifying the start of IP address range
577 * ``NumNodes``: size of the IP address range
579 * ``suts``: a structure that contains definitions of each specific SUT
580 (represents a vEPC component). SUT structure contains following key/value
583 * ``name``: unique custom string specifying SUT name
584 * ``role``: string value corresponding with an SUT role specified in the
585 session profile (test session template) file
586 * ``managementIp``: SUT management IP adress
587 * ``phy``: network interface name, e.g. *eth1*
588 * ``ip``: vEPC component IP address used in test case topology
589 * ``nextHop``: next hop IP address, to allow for vEPC inter-node communication
591 Update NSB test case definitions
592 ================================
593 NSB test case file designated for vEPC testing contains an example of specific
594 test scenario configuration.
595 Test operator may change these definitions as required for the use case that
597 Specifically, following subsections of the vEPC test case (section **scenarios**)
600 1. Subsection ``options``: contains custom parameters used for vEPC testing
602 * subsection ``dmf``: may contain one or more parameters specified in
603 ``traffic_profile`` template file
604 * subsection ``test_cases``: contains re-definitions of parameters specified
605 in ``session_profile`` template file
607 .. note:: All parameters in ``session_profile``, value of which is a
608 placeholder, needs to be re-defined to construct a valid test session.
610 2. Subsection ``runner``: specifies the test duration and the interval of
611 TG and VNF side KPIs polling. For more details, refer to :doc:`03-architecture`.
613 Preparing test run of vPE test case
614 -----------------------------------
615 The vPE (Provider Edge Router) is a :term: `VNF` approximation
616 serving as an Edge Router. The vPE is approximated using the
617 ``ip_pipeline`` dpdk application.
619 .. image:: /../docs/testing/developer/devguide/images/vPE_Diagram.png
621 :alt: NSB vPE Diagram
623 The ``vpe_config`` file must be passed as it is not auto generated.
624 The ``vpe_script`` defines the rules applied to each of the pipelines. This can be
625 auto generated or a file can be passed using the ``script_file`` option in
626 ``vnf_config`` as shown below. The ``full_tm_profile_file`` option must be
627 used if a traffic manager is defined in ``vpe_config``.
631 vnf_config: { file: './vpe_config/vpe_config_2_ports',
632 action_bulk_file: './vpe_config/action_bulk_512.txt',
633 full_tm_profile_file: './vpe_config/full_tm_profile_10G.cfg',
634 script_file: './vpe_config/vpe_script_sample' }
636 Testcases for vPE can be found in the ``vnf_samples/nsut/vpe`` directory.
637 A testcase can be started with the following command as an example:
641 yardstick task start /yardstick/samples/vnf_samples/nsut/vpe/tc_baremetal_rfc2544_ipv4_1flow_64B_ixia.yaml
643 Preparing test run of vIPSEC test case
644 --------------------------------------
646 Location of vIPSEC test cases: ``samples/vnf_samples/nsut/ipsec/``.
648 Before running a specific vIPSEC test case using NSB, some dependencies have to be
649 preinstalled and properly configured.
652 .. code-block:: console
654 export UBUNTU="xenial"
655 export RELEASE=".stable.1810"
656 sudo rm /etc/apt/sources.list.d/99fd.io.list
657 echo "deb [trusted=yes] https://nexus.fd.io/content/repositories/fd.io$RELEASE.ubuntu.$UBUNTU.main/ ./" | sudo tee -a /etc/apt/sources.list.d/99fd.io.list
659 sudo apt-get install vpp vpp-lib vpp-plugin vpp-dbg vpp-dev vpp-api-java vpp-api-python vpp-api-lua
663 VAT templates is required for the VPP API.
665 .. code-block:: console
667 mkdir -p /opt/nsb_bin/vpp/templates/
668 echo 'exec trace add dpdk-input 50' > /opt/nsb_bin/vpp/templates/enable_dpdk_traces.vat
669 echo 'exec trace add vhost-user-input 50' > /opt/nsb_bin/vpp/templates/enable_vhost_user_traces.vat
670 echo 'exec trace add memif-input 50' > /opt/nsb_bin/vpp/templates/enable_memif_traces.vat
671 cat > /opt/nsb_bin/vpp/templates/dump_interfaces.vat << EOL
678 Preparing test run of vCMTS test case
679 -------------------------------------
681 Location of vCMTS test cases: ``samples/vnf_samples/nsut/cmts/``.
683 Before running a specific vIPSEC test case using NSB, some changes must be
684 made to the original vCMTS package.
686 Allow SSH access to the docker images
687 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
688 Follow the documentation at ``https://docs.docker.com/engine/examples/running_ssh_service/``
689 to allow SSH access to the Pktgen/vcmts-d containers located at:
691 * ``$VCMTS_ROOT/pktgen/docker/docker-image-pktgen/Dockerfile`` and
692 * ``$VCMTS_ROOT/vcmtsd/docker/docker-image-vcmtsd/Dockerfile``
695 Deploy the ConfigMaps for Pktgen and vCMTSd
696 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
699 cd $VCMTS_ROOT/kubernetes/helm/pktgen
700 helm template . -x templates/pktgen-configmap.yaml > configmap.yaml
701 kubectl create -f configmap.yaml
703 cd $VCMTS_ROOT/kubernetes/helm/vcmtsd
704 helm template . -x templates/vcmts-configmap.yaml > configmap.yaml
705 kubectl create -f configmap.yaml