Merge "Cleanup Plugin unit tests"

[yardstick.git] / docs / testing / developer / devguide / devguide_nsb_prox.rst

Introduction
=============

This document describes the steps to create a new NSB PROX test based on
existing PROX functionalities. NSB PROX provides is a simple approximation
of an operation and can be used to develop best practices and TCO models
for Telco customers, investigate the impact of new Intel compute,
network and storage technologies, characterize performance, and develop
optimal system architectures and configurations.

.. contents::

Prerequisites
=============

In order to integrate PROX tests into NSB, the following prerequisites are required.

.. _`dpdk wiki page`: http://dpdk.org/
.. _`yardstick wiki page`: https://wiki.opnfv.org/display/yardstick/
.. _`Prox documentation`: https://01.org/intel-data-plane-performance-demonstrators/documentation/prox-documentation
.. _`openstack wiki page`: https://wiki.openstack.org/wiki/Main_Page
.. _`grafana getting started`: http://docs.grafana.org/guides/gettingstarted/
.. _`opnfv grafana dashboard`: https://wiki.opnfv.org/display/yardstick/How+to+work+with+grafana+dashboard
.. _`Prox command line`: https://01.org/intel-data-plane-performance-demonstrators/documentation/prox-documentation#Command_line_options
.. _`grafana deployment`: https://wiki.opnfv.org/display/yardstick/How+to+deploy+InfluxDB+and+Grafana+locally
.. _`Prox options`: https://01.org/intel-data-plane-performance-demonstrators/documentation/prox-documentation#.5Beal_options.5D
.. _`NSB Installation`: http://artifacts.opnfv.org/yardstick/docs/userguide/index.html#document-09-installation

* A working knowledge of Yardstick. See `yardstick wiki page`_.
* A working knowledge of PROX. See `Prox documentation`_.
* Knowledge of Openstack. See `openstack wiki page`_.
* Knowledge of how to use Grafana. See `grafana getting started`_.
* How to Deploy InfluxDB & Grafana. See `grafana deployment`_.
* How to use Grafana in OPNFV/Yardstick. See `opnfv grafana dashboard`_.
* How to install NSB. See `NSB Installation`_

Sample Prox Test Hardware Architecture
======================================

The following is a diagram of a sample NSB PROX Hardware Architecture
for both NSB PROX on Bare metal and on Openstack.

In this example when running yardstick on baremetal, yardstick will
run on the deployment node, the generator will run on the deployment node
and the SUT(SUT) will run on the Controller Node.


.. image:: images/PROX_Hardware_Arch.png
   :width: 800px
   :alt: Sample NSB PROX Hard Architecture

Prox Test Architecture
======================

In order to create a new test, one must understand the architecture of
the test.

A NSB Prox test architecture is composed of:

* A traffic generator. This provides blocks of data on 1 or more ports
  to the SUT.
  The traffic generator also consumes the result packets from the system
  under test.
* A SUT consumes the packets generated by the packet
  generator, and applies one or more tasks to the packets and return the
  modified packets to the traffic generator.

  This is an example of a sample NSB PROX test architecture.

.. image:: images/PROX_Software_Arch.png
   :width: 800px
   :alt: NSB PROX test Architecture

This diagram is of a sample NSB PROX test application.

* Traffic Generator

  * Generator Tasks - Composted of 1 or more tasks (It is possible to
    have multiple tasks sending packets to same port No. See Tasks Ai and Aii
    plus Di and Dii)

    * Task Ai - Generates Packets on Port 0 of Traffic Generator
      and send to Port 0 of SUT Port 0
    * Task Aii - Generates Packets on Port 0 of Traffic Generator
      and send to Port 0 of SUT Port 0
    * Task B - Generates Packets on Port 1 of Traffic Generator
      and send to Port 1 of SUT Port 1
    * Task C - Generates Packets on Port 2 of Traffic Generator
      and send to Port 2 of SUT Port 2
    * Task Di - Generates Packets on Port 3 of Traffic Generator
      and send to Port 3 of SUT Port 3
    * Task Dii - Generates Packets on Port 0 of Traffic Generator
      and send to Port 0 of SUT Port 0

  * Verifier Tasks - Composed of 1 or more tasks which receives
    packets from SUT

    * Task E - Receives packets on Port 0 of Traffic Generator sent
      from Port 0 of SUT Port 0
    * Task F - Receives packets on Port 1 of Traffic Generator sent
      from Port 1 of SUT Port 1
    * Task G - Receives packets on Port 2 of Traffic Generator sent
      from Port 2 of SUT Port 2
    * Task H - Receives packets on Port 3 of Traffic Generator sent
      from Port 3 of SUT Port 3

* SUT

  * Receiver Tasks - Receives packets from generator - Composed on 1 or
    more tasks which consume the packs sent from Traffic Generator

    * Task A - Receives Packets on Port 0 of System-Under-Test from
      Traffic Generator Port 0, and forwards packets to Task E
    * Task B - Receives Packets on Port 1 of System-Under-Test from
      Traffic Generator Port 1, and forwards packets to Task E
    * Task C - Receives Packets on Port 2 of System-Under-Test from
      Traffic Generator Port 2, and forwards packets to Task E
    * Task D - Receives Packets on Port 3 of System-Under-Test from
      Traffic Generator Port 3, and forwards packets to Task E

  * Processing Tasks - Composed of multiple tasks in series which carry
    out some processing on received packets before forwarding to the
    task.

    * Task E - This receives packets from the Receiver Tasks,
      carries out some operation on the data and forwards to result
      packets to the next task in the sequence - Task F
    * Task F - This receives packets from the previous Task - Task
      E, carries out some operation on the data and forwards to result
      packets to the next task in the sequence - Task G
    * Task G - This receives packets from the previous Task - Task F
      and distributes the result packages to the Transmitter tasks

  * Transmitter Tasks - Composed on 1 or more tasks which send the
    processed packets back to the Traffic Generator

    * Task H - Receives Packets from Task G of System-Under-Test and
      sends packets to Traffic Generator Port 0
    * Task I - Receives Packets from Task G of System-Under-Test and
      sends packets to Traffic Generator Port 1
    * Task J - Receives Packets from Task G of System-Under-Test and
      sends packets to Traffic Generator Port 2
    * Task K - Receives Packets From Task G of System-Under-Test and
      sends packets to Traffic Generator Port 3

NSB Prox Test
=============

A NSB Prox test is composed of the following components :-

* Test Description File. Usually called
  ``tc_prox_<context>_<test>-<ports>.yaml`` where

  * <context> is either ``baremetal`` or ``heat_context``
  * <test> is the a one or 2 word description of the test.
  * <ports> is the number of ports used

  Example tests ``tc_prox_baremetal_l2fwd-2.yaml`` or
  ``tc_prox_heat_context_vpe-4.yaml``. This file describes the components
  of the test, in the case of openstack the network description and
  server descriptions, in the case of baremetal the hardware
  description location. It also contains the name of the Traffic Generator, the SUT config file
  and the traffic profile description, all described below. See nsb-test-description-label_

* Traffic Profile file. Example ``prox_binsearch.yaml``. This describes the packet size, tolerated
  loss, initial line rate to start traffic at, test interval etc See nsb-traffic-profile-label_

* Traffic Generator Config file. Usually called ``gen_<test>-<ports>.cfg``.

  This describes the activity of the traffic generator

  * What each core of the traffic generator does,
  * The packet of data sent by a core on a port of the traffic generator
    to the system under test
  * What core is used to wait on what port for data from the system
    under test.

  Example traffic generator config file  ``gen_l2fwd-4.cfg``
  See nsb-traffic-generator-label_

* SUT Config file. Usually called ``handle_<test>-<ports>.cfg``.

  This describes the activity of the SUTs

  * What each core of the  does,
  * What cores receives packets from what ports
  * What cores perform operations on the packets and pass the packets onto
    another core
  * What cores receives packets from what cores and transmit the packets on
    the ports to the Traffic Verifier tasks of the Traffic Generator.

  Example traffic generator config file  ``handle_l2fwd-4.cfg``
  See nsb-sut-generator-label_

* NSB PROX Baremetal Configuration file. Usually called
  ``prox-baremetal-<ports>.yaml``

  * <ports> is the number of ports used

  This is required for baremetal only.  This describes hardware, NICs,
  IP addresses, Network drivers, usernames and passwords.
  See baremetal-config-label_

* Grafana Dashboard. Usually called
  ``Prox_<context>_<test>-<port>-<DateAndTime>.json`` where

  * <context> Is either ``BM`` or ``heat``
  * <test> Is the a one or 2 word description of the test.
  * <port> is the number of ports used express as ``2Port`` or ``4Port``
  * <DateAndTime> is the Date and Time expressed as a string.

  Example grafana dashboard ``Prox_BM_L2FWD-4Port-1507804504588.json``

Other files may be required. These are test specific files and will be
covered later.

.. _nsb-test-description-label:

**Test Description File**

Here we will discuss the test description for both
baremetal and openstack.

*Test Description File for Baremetal*
-------------------------------------

This section will introduce the meaning of the Test case description
file. We will use ``tc_prox_baremetal_l2fwd-2.yaml`` as an example to
show you how to understand the test description file.

.. image:: images/PROX_Test_BM_Script.png
   :width: 800px
   :alt: NSB PROX Test Description File

Now let's examine the components of the file in detail

1. ``traffic_profile`` - This specifies the traffic profile for the
   test. In this case ``prox_binsearch.yaml`` is used. See nsb-traffic-profile-label_

2. ``topology`` - This is either ``prox-tg-topology-1.yaml`` or
    ``prox-tg-topology-2.yaml`` or ``prox-tg-topology-4.yaml``
    depending on number of ports required.

3. ``nodes`` - This names the Traffic Generator and the System
   under Test. Does not need to change.

4. ``interface_speed_gbps`` - This is an optional parameter. If not present
   the system defaults to 10Gbps. This defines the speed of the interfaces.

5. ``prox_path`` - Location of the Prox executable on the traffic
   generator (Either baremetal or Openstack Virtual Machine)

6. ``prox_config`` - This is the ``SUT Config File``.
   In this case it is ``handle_l2fwd-2.cfg``

   A number of additional parameters can be added. This example
   is for VPE::

    options:
      interface_speed_gbps: 10

      vnf__0:
        prox_path: /opt/nsb_bin/prox
        prox_config: ``configs/handle_vpe-4.cfg``
        prox_args:
          ``-t``: ````
        prox_files:
          ``configs/vpe_ipv4.lua`` : ````
          ``configs/vpe_dscp.lua`` : ````
          ``configs/vpe_cpe_table.lua`` : ````
          ``configs/vpe_user_table.lua`` : ````
          ``configs/vpe_rules.lua`` : ````
        prox_generate_parameter: True

   ``interface_speed_gbps`` - this specifies the speed of the interface
   in Gigabits Per Second. This is used to calculate pps(packets per second).
   If the interfaces are of different speeds, then this specifies the speed
   of the slowest interface. This parameter is optional. If omitted the
   interface speed defaults to 10Gbps.

   ``prox_files`` - this specified that a number of addition files
   need to be provided for the test to run correctly. This files
   could provide routing information,hashing information or a
   hashing algorithm and ip/mac information.

   ``prox_generate_parameter`` - this specifies that the NSB application
   is required to provide information to the nsb Prox in the form
   of a file called ``parameters.lua``, which contains information
   retrieved from either the hardware or the openstack configuration.

7. ``prox_args`` - this specifies the command line arguments to start
   prox. See `prox command line`_.

8. ``prox_config`` - This specifies the Traffic Generator config file.

9. ``runner`` - This is set to ``ProxDuration`` - This specifies that the
   test runs for a set duration. Other runner types are available
   but it is recommend to use ``ProxDuration``

   The following parrameters are supported

   ``interval`` - (optional) - This specifies the sampling interval.
   Default is 1 sec

   ``sampled`` - (optional) - This specifies if sampling information is
   required. Default ``no``

   ``duration`` - This is the length of the test in seconds. Default
   is 60 seconds.

   ``confirmation`` - This specifies the number of confirmation retests to
   be made before deciding to increase or decrease line speed. Default 0.

10. ``context`` - This is ``context`` for a 2 port Baremetal configuration.

   If a 4 port configuration was required then file
   ``prox-baremetal-4.yaml`` would be used. This is the NSB Prox
   baremetal configuration file.

.. _nsb-traffic-profile-label:

*Traffic Profile file*
----------------------

This describes the details of the traffic flow. In this case
``prox_binsearch.yaml`` is used.

.. image:: images/PROX_Traffic_profile.png
   :width: 800px
   :alt: NSB PROX Traffic Profile


1. ``name`` - The name of the traffic profile. This name should match the name specified in the
   ``traffic_profile`` field in the Test Description File.

2. ``traffic_type`` - This specifies the type of traffic pattern generated, This name matches
   class name of the traffic generator See::

      network_services/traffic_profile/prox_binsearch.py class ProxBinSearchProfile(ProxProfile)

   In this case it lowers the traffic rate until the number of packets
   sent is equal to the number of packets received (plus a
   tolerated loss). Once it achieves this it increases the traffic
   rate in order to find the highest rate with no traffic loss.

   Custom traffic types can be created by creating a new traffic profile class.

3. ``tolerated_loss`` - This specifies the percentage of packets that
   can be lost/dropped before
   we declare success or failure. Success is Transmitted-Packets from
   Traffic Generator is greater than or equal to
   packets received by Traffic Generator plus tolerated loss.

4. ``test_precision`` - This specifies the precision of the test
   results. For some tests the success criteria may never be
   achieved because the test precision may be greater than the
   successful throughput. For finer results increase the precision
   by making this value smaller.

5. ``packet_sizes`` - This specifies the range of packets size this
   test is run for.

6. ``duration`` - This specifies the sample duration that the test
   uses to check for success or failure.

7. ``lower_bound`` - This specifies the test initial lower bound sample rate.
   On success this value is increased.

8. ``upper_bound`` - This specifies the test initial upper bound sample rate.
   On success this value is decreased.

Other traffic profiles exist eg prox_ACL.yaml which does not
compare what is received with what is transmitted. It just
sends packet at max rate.

It is possible to create custom traffic profiles with by
creating new file in the same folder as prox_binsearch.yaml.
See this prox_vpe.yaml as example::

     schema: ``nsb:traffic_profile:0.1``

     name:            prox_vpe
     description:     Prox vPE traffic profile

     traffic_profile:
       traffic_type: ProxBinSearchProfile
       tolerated_loss: 100.0 #0.001
       test_precision: 0.01
     # The minimum size of the Ethernet frame for the vPE test is 68 bytes.
       packet_sizes: [68]
       duration: 5
       lower_bound: 0.0
       upper_bound: 100.0

*Test Description File for Openstack*
-------------------------------------

We will use ``tc_prox_heat_context_l2fwd-2.yaml`` as a example to show
you how to understand the test description file.

.. image:: images/PROX_Test_HEAT_Script1.png
   :width: 800px
   :alt: NSB PROX Test Description File - Part 1

.. image:: images/PROX_Test_HEAT_Script2.png
   :width: 800px
   :alt: NSB PROX Test Description File - Part 2

Now lets examine the components of the file in detail

Sections 1 to 9 are exactly the same in Baremetal and in Heat. Section
``10`` is replaced with sections A to F. Section 10 was for a baremetal
configuration file. This has no place in a heat configuration.

A. ``image`` - yardstick-samplevnfs. This is the name of the image
   created during the installation of NSB. This is fixed.

B. ``flavor`` - The flavor is created dynamically. However we could
   use an already existing flavor if required. In that case the
   flavor would be named::

    flavor: yardstick-flavor

C. ``extra_specs`` - This allows us to specify the number of
   cores sockets and hyperthreading assigned to it. In this case
   we have 1 socket with 10 codes and no hyperthreading enabled.

D. ``placement_groups`` - default. Do not change for NSB PROX.

E. ``servers`` - ``tg_0`` is the traffic generator and ``vnf_0``
   is the system under test.

F. ``networks`` - is composed of a management network labeled ``mgmt``
   and one uplink network labeled ``uplink_0``  and one downlink
   network labeled ``downlink_0`` for 2 ports. If this was a 4 port
   configuration there would be 2 extra downlink ports. See this
   example from a 4 port l2fwd test.::

    networks:
      mgmt:
        cidr: '10.0.1.0/24'
      uplink_0:
        cidr: '10.0.2.0/24'
        gateway_ip: 'null'
        port_security_enabled: False
        enable_dhcp: 'false'
      downlink_0:
        cidr: '10.0.3.0/24'
        gateway_ip: 'null'
        port_security_enabled: False
        enable_dhcp: 'false'
      uplink_1:
        cidr: '10.0.4.0/24'
        gateway_ip: 'null'
        port_security_enabled: False
        enable_dhcp: 'false'
      downlink_1:
        cidr: '10.0.5.0/24'
        gateway_ip: 'null'
        port_security_enabled: False
        enable_dhcp: 'false'

.. _nsb-traffic-generator-label:

*Traffic Generator Config file*
-------------------------------

This section will describe the traffic generator config file.
This is the same for both baremetal and heat. See this example
of ``gen_l2fwd_multiflow-2.cfg`` to explain the options.

.. image:: images/PROX_Gen_2port_cfg.png
   :width: 1400px
   :alt: NSB PROX Gen Config File

The configuration file is divided into multiple sections, each
of which is used to define some parameters and options.::

  [eal options]
  [variables]
  [port 0]
  [port 1]
  [port .]
  [port Z]
  [defaults]
  [global]
  [core 0]
  [core 1]
  [core 2]
  [core .]
  [core Z]

See `prox options`_ for details

Now let's examine the components of the file in detail

1. ``[eal options]`` - This specified the EAL (Environmental
   Abstraction Layer) options. These are default values and
   are not changed. See `dpdk wiki page`_.

2. ``[variables]`` - This section contains variables, as
   the name suggests. Variables for Core numbers, mac
   addresses, ip addresses etc. They are assigned as a
   ``key = value`` where the key is used in place of the value.

   .. caution::
     A special case for valuables with a value beginning with
     ``@@``. These values are dynamically updated by the NSB
     application at run time. Values like MAC address,
     IP Address etc.

3. ``[port 0]`` - This section describes the DPDK Port. The number
   following the keyword ``port`` usually refers to the DPDK Port
   Id. usually starting from ``0``. Because you can have multiple
   ports this entry usually repeated. Eg. For a 2 port setup
   ``[port0]`` and ``[port 1]`` and for a 4 port setup ``[port 0]``,
   ``[port 1]``, ``[port 2]`` and ``[port 3]``::

      [port 0]
      name=p0
      mac=hardware
      rx desc=2048
      tx desc=2048
      promiscuous=yes

   a. In this example ``name = p0`` assigned the name ``p0`` to the
      port. Any name can be assigned to a port.
   b. ``mac=hardware`` sets the MAC address assigned by the hardware
      to data from this port.
   c. ``rx desc=2048`` sets the number of available descriptors to
      allocate for receive packets. This can be changed and can
      effect performance.
   d. ``tx desc=2048`` sets the number of available descriptors to
      allocate for transmit packets. This can be changed and can
      effect performance.
   e. ``promiscuous=yes`` this enables promiscuous mode for this port.

4. ``[defaults]`` - Here default operations and settings can be over
   written. In this example ``mempool size=4K`` the number of mbufs
   per task is altered. Altering this value could effect
   performance. See `prox options`_ for details.

5. ``[global]`` - Here application wide setting are supported. Things
   like application name, start time, duration and memory
   configurations can be set here. In this example.::

      [global]
      start time=5
      name=Basic Gen

    a. ``start time=5`` Time is seconds after which average
       stats will be started.
    b. ``name=Basic Gen`` Name of the configuration.

6. ``[core 0]`` - This core is designated the master core. Every
   Prox application must have a master core. The master mode must
   be assigned to exactly one task, running alone on one core.::

    [core 0]
    mode=master

7. ``[core 1]`` - This describes the activity on core 1. Cores can
   be configured by means of a set of [core #] sections, where
   # represents either:

   a. an absolute core number: e.g. on a 10-core, dual socket
      system with hyper-threading,
      cores are numbered from 0 to 39.

   b. PROX allows a core to be identified by a core number, the
      letter 's', and a socket number.

      It is possible to write a baremetal and an openstack test which use
      the same traffic generator config file and SUT config file.
      In this case it is advisable not to use physical
      core numbering.

      However it is also possible to write NSB Prox tests that
      have been optimized for a particular hardware configuration.
      In this case it is advisable to use the core numbering.
      It is up to the user to make sure that cores from
      the right sockets are used (i.e. from the socket on which the NIC
      is attached to), to ensure good performance (EPA).

   Each core can be assigned with a set of tasks, each running
   one of the implemented packet processing modes.::

     [core 1]
     name=p0
     task=0
     mode=gen
     tx port=p0
     bps=1250000000
     ; Ethernet + IP + UDP
     pkt inline=${sut_mac0} 70 00 00 00 00 01 08 00 45 00 00 1c 00 01 00 00 40 11 f7 7d 98 10 64 01 98 10 64 02 13 88 13 88 00 08 55 7b
     ; src_ip: 152.16.100.0/8
     random=0000XXX1
     rand_offset=29
     ; dst_ip: 152.16.100.0/8
     random=0000XXX0
     rand_offset=33
     random=0001001110001XXX0001001110001XXX
     rand_offset=34

   a. ``name=p0`` - Name assigned to the core.
   b. ``task=0`` - Each core can run a set of tasks. Starting with ``0``.
      Task 1 can be defined later in this core or
      can be defined in another ``[core 1]`` section with ``task=1``
      later in configuration file. Sometimes running
      multiple task related to the same packet on the same physical
      core improves performance, however sometimes it
      is optimal to move task to a separate core. This is best
      decided by checking performance.
   c. ``mode=gen`` - Specifies the action carried out by this task on
      this core. Supported modes are: classify, drop, gen, lat, genl4, nop, l2fwd, gredecap,
      greencap, lbpos, lbnetwork, lbqinq, lb5tuple, ipv6_decap, ipv6_encap,
      qinqdecapv4, qinqencapv4, qos, routing, impair,
      mirror, unmpls, tagmpls, nat, decapnsh, encapnsh, police, acl
      Which are :-

       * Classify
       * Drop
       * Basic Forwarding (no touch)
       * L2 Forwarding (change MAC)
       * GRE encap/decap
       * Load balance based on packet fields
       * Symmetric load balancing
       * QinQ encap/decap IPv4/IPv6
       * ARP
       * QoS
       * Routing
       * Unmpls
       * Nsh encap/decap
       * Policing
       * ACL

      In the traffic generator we expect a core to generate packets (``gen``)
      and to receive packets & calculate latency (``lat``)
      This core does ``gen`` . ie it is a traffic generator.

      To understand what each of the modes support please see
      `prox documentation`_.

   d. ``tx port=p0`` - This specifies that the packets generated are
      transmitted to port ``p0``
   e. ``bps=1250000000`` - This indicates Bytes Per Second to
      generate packets.
   f. ``; Ethernet + IP + UDP`` - This is a comment. Items starting with
      ``;`` are ignored.
   g. ``pkt inline=${sut_mac0} 70 00 00 00 ...`` - Defines the packet
      format as a sequence of bytes (each
      expressed in hexadecimal notation). This defines the packet
      that is generated. This packets begins
      with the hexadecimal sequence assigned to ``sut_mac`` and the
      remainder of the bytes in the string.
      This packet could now be sent or modified by ``random=..``
      described below before being sent to target.
   h. ``; src_ip: 152.16.100.0/8`` - Comment
   i. ``random=0000XXX1`` - This describes a field of the packet
      containing random data. This string can be
      8,16,24 or 32 character long and represents 1,2,3 or 4
      bytes of data. In this case it describes a byte of
      data. Each character in string can be 0,1 or ``X``. 0 or 1
      are fixed bit values in the data packet and ``X`` is a
      random bit. So random=0000XXX1 generates 00000001(1),
      00000011(3), 00000101(5), 00000111(7),
      00001001(9), 00001011(11), 00001101(13) and 00001111(15)
      combinations.
   j. ``rand_offset=29`` - Defines where to place the previously
      defined random field.
   k. ``; dst_ip: 152.16.100.0/8`` - Comment
   l. ``random=0000XXX0`` - This is another random field which
      generates a byte of 00000000(0), 00000010(2),
      00000100(4), 00000110(6), 00001000(8), 00001010(10),
      00001100(12) and 00001110(14) combinations.
   m. ``rand_offset=33`` - Defines where to place the previously
      defined random field.
   n. ``random=0001001110001XXX0001001110001XXX`` - This is
      another random field which generates 4 bytes.
   o. ``rand_offset=34`` - Defines where to place the previously
      defined 4 byte random field.

   Core 2 executes same scenario as Core 1. The only difference
   in this case is that the packets are generated
   for Port 1.

8. ``[core 3]`` - This defines the activities on core 3. The purpose
   of ``core 3`` and ``core 4`` is to receive packets
   sent by the SUT.::

     [core 3]
     name=rec 0
     task=0
     mode=lat
     rx port=p0
     lat pos=42

   a. ``name=rec 0`` - Name assigned to the core.
   b. ``task=0`` - Each core can run a set of tasks. Starting with
      ``0``. Task 1 can be defined later in this core or
      can be defined in another ``[core 1]`` section with
      ``task=1`` later in configuration file. Sometimes running
      multiple task related to the same packet on the same
      physical core improves performance, however sometimes it
      is optimal to move task to a separate core. This is
      best decided by checking performance.
   c. ``mode=lat`` - Specifies the action carried out by this task on this core. Supported modes are: acl,
      classify, drop, gredecap, greencap, ipv6_decap, ipv6_encap, l2fwd, lbnetwork, lbpos, lbqinq, nop,
      police, qinqdecapv4, qinqencapv4, qos, routing, impair, lb5tuple, mirror, unmpls, tagmpls,
      nat, decapnsh, encapnsh, gen, genl4 and lat. This task(0) per core(3) receives packets on port.
   d. ``rx port=p0`` - The port to receive packets on ``Port 0``. Core 4 will receive packets on ``Port 1``.
   e. ``lat pos=42`` - Describes where to put a 4-byte timestamp in the packet. Note that the packet length should
      be longer than ``lat pos`` + 4 bytes to avoid truncation of the timestamp. It defines where the timestamp is
      to be read from. Note that the SUT workload might cause the position of the timestamp to change
      (i.e. due to encapsulation).

.. _nsb-sut-generator-label:

*SUT Config file*
-------------------------------

This section will describes the SUT(VNF) config file. This is the same for both
baremetal and heat. See this example of ``handle_l2fwd_multiflow-2.cfg`` to explain the options.

.. image:: images/PROX_Handle_2port_cfg.png
   :width: 1400px
   :alt: NSB PROX Handle Config File

See `prox options`_ for details

Now let's examine the components of the file in detail

1. ``[eal options]`` - same as the Generator config file. This specified the EAL (Environmental Abstraction Layer)
   options. These are default values and are not changed.
   See `dpdk wiki page`_.

2. ``[port 0]`` - This section describes the DPDK Port. The number following the keyword ``port`` usually refers to the DPDK Port Id. usually starting from ``0``.
   Because you can have multiple ports this entry usually repeated. Eg. For a 2 port setup ``[port0]`` and ``[port 1]`` and for a 4 port setup ``[port 0]``, ``[port 1]``,
   ``[port 2]`` and ``[port 3]``::

      [port 0]
      name=if0
      mac=hardware
      rx desc=2048
      tx desc=2048
      promiscuous=yes

   a. In this example ``name =if0`` assigned the name ``if0`` to the port. Any name can be assigned to a port.
   b. ``mac=hardware`` sets the MAC address assigned by the hardware to data from this port.
   c. ``rx desc=2048`` sets the number of available descriptors to allocate for receive packets. This can be changed and can effect performance.
   d. ``tx desc=2048`` sets the number of available descriptors to allocate for transmit packets. This can be changed and can effect performance.
   e. ``promiscuous=yes`` this enables promiscuous mode for this port.

3. ``[defaults]`` - Here default operations and settings can be over written.::

     [defaults]
     mempool size=8K
     memcache size=512

   a. In this example ``mempool size=8K`` the number of mbufs per task is altered. Altering this value could effect performance. See `prox options`_ for details.
   b. ``memcache size=512`` - number of mbufs cached per core, default is 256 this is the cache_size. Altering this value could effect performance.

4. ``[global]`` - Here application wide setting are supported. Things like application name, start time, duration and memory configurations can be set here.
   In this example.::

      [global]
      start time=5
      name=Basic Gen

    a. ``start time=5`` Time is seconds after which average stats will be started.
    b. ``name=Handle L2FWD Multiflow (2x)`` Name of the configuration.

5. ``[core 0]`` - This core is designated the master core. Every Prox application must have a master core. The master mode must be assigned to
   exactly one task, running alone on one core.::

    [core 0]
    mode=master

6. ``[core 1]`` - This describes the activity on core 1. Cores can be configured by means of a set of [core #] sections,   where # represents either:

   a. an absolute core number: e.g. on a 10-core, dual socket system with hyper-threading,
      cores are numbered from 0 to 39.

   b. PROX allows a core to be identified by a core number, the letter 's', and a socket number.
      However NSB PROX is hardware agnostic (physical and virtual configurations are the same) it
      is advisable no to use physical core numbering.

   Each core can be assigned with a set of tasks, each running one of the implemented packet processing modes.::

     [core 1]
     name=none
     task=0
     mode=l2fwd
     dst mac=@@tester_mac1
     rx port=if0
     tx port=if1

   a. ``name=none`` - No name assigned to the core.
   b. ``task=0`` - Each core can run a set of tasks. Starting with ``0``. Task 1 can be defined later in this core or
      can be defined in another ``[core 1]`` section with ``task=1`` later in configuration file. Sometimes running
      multiple task related to the same packet on the same physical core improves performance, however sometimes it
      is optimal to move task to a separate core. This is best decided by checking performance.
   c. ``mode=l2fwd`` - Specifies the action carried out by this task on this core. Supported modes are: acl,
      classify, drop, gredecap, greencap, ipv6_decap, ipv6_encap, l2fwd, lbnetwork, lbpos, lbqinq, nop,
      police, qinqdecapv4, qinqencapv4, qos, routing, impair, lb5tuple, mirror, unmpls, tagmpls,
      nat, decapnsh, encapnsh, gen, genl4 and lat. This code does ``l2fwd`` .. ie it does the L2FWD.

   d. ``dst mac=@@tester_mac1`` - The destination mac address of the packet will be set to the MAC address of ``Port 1`` of destination device. (The Traffic Generator/Verifier)
   e. ``rx port=if0`` - This specifies that the packets are received from ``Port 0`` called if0
   f. ``tx port=if1`` - This specifies that the packets are transmitted to ``Port 1``  called if1

   If this example we receive a packet on core on a port, carry out operation on the packet on the core and transmit it on on another port still using the same task on the same core.

   On some implementation you may wish to use multiple tasks, like this.::

     [core 1]
     name=rx_task
     task=0
     mode=l2fwd
     dst mac=@@tester_p0
     rx port=if0
     tx cores=1t1
     drop=no

     name=l2fwd_if0
     task=1
     mode=nop
     rx ring=yes
     tx port=if0
     drop=no

   In this example you can see Core 1/Task 0 called ``rx_task`` receives the packet from if0 and perform the l2fwd. However instead of sending the packet to a
   port it sends it to a core see ``tx cores=1t1``. In this case it sends it to Core 1/Task 1.

   Core 1/Task 1 called ``l2fwd_if0``, receives the packet, not from a port but from the ring. See ``rx ring=yes``. It does not perform any operation on the packet See ``mode=none``
   and sends the packets to ``if0`` see ``tx port=if0``.

   It is also possible to implement more complex operations be chaining multiple operations in sequence and using rings to pass packets from one core to another.

   In thus example we show a Broadband Network Gateway (BNG) with Quality of Service (QoS).  Communication from task to task is via rings.

   .. image:: images/PROX_BNG_QOS.png
      :width: 1000px
      :alt: NSB PROX Config File for BNG_QOS

*Baremetal Configuration file*
------------------------------

.. _baremetal-config-label:

This is required for baremetal testing. It describes the IP address of the various ports, the Network devices drivers and MAC addresses and the network
configuration.

In this example we will describe a 2 port configuration. This file is the same for all 2 port NSB Prox tests on the same platforms/configuration.

  .. image:: images/PROX_Baremetal_config.png
     :width: 1000px
     :alt: NSB PROX Yardstick Config

Now lets describe the sections of the file.

  1. ``TrafficGen`` - This section describes the Traffic Generator node of the test configuration. The name of the node ``trafficgen_1`` must match the node name
     in the ``Test Description File for Baremetal`` mentioned earlier. The password attribute of the test needs to be configured. All other parameters
     can remain as default settings.
  2. ``interfaces`` - This defines the DPDK interfaces on the Traffic Generator.
  3. ``xe0`` is DPDK Port 0. ``lspci`` and `` ./dpdk-devbind.py -s`` can be used to provide the interface information. ``netmask`` and ``local_ip`` should not be changed
  4. ``xe1`` is DPDK Port 1. If more than 2 ports are required then ``xe1`` section needs to be repeated and modified accordingly.
  5. ``vnf`` - This section describes the SUT of the test configuration. The name of the node ``vnf`` must match the node name in the
     ``Test Description File for Baremetal`` mentioned earlier. The password attribute of the test needs to be configured. All other parameters
     can remain as default settings
  6. ``interfaces`` - This defines the DPDK interfaces on the SUT
  7. ``xe0`` - Same as 3 but for the ``SUT``.
  8. ``xe1`` - Same as 4 but for the ``SUT`` also.
  9. ``routing_table`` - All parameters should remain unchanged.
  10. ``nd_route_tbl`` - All parameters should remain unchanged.

*Grafana Dashboard*
-------------------

The grafana dashboard visually displays the results of the tests. The steps required to produce a grafana dashboard are described here.

.. _yardstick-config-label:

  a. Configure ``yardstick`` to use influxDB to store test results. See file ``/etc/yardstick/yardstick.conf``.

     .. image:: images/PROX_Yardstick_config.png
        :width: 1000px
        :alt: NSB PROX Yardstick Config

     1. Specify the dispatcher to use influxDB to store results.
     2. "target = .. " - Specify location of influxDB to store results.
        "db_name = yardstick" - name of database. Do not change
        "username = root" - username to use to store result. (Many tests are run as root)
        "password = ... " - Please set to root user password

  b. Deploy InfludDB & Grafana. See how to Deploy InfluxDB & Grafana. See `grafana deployment`_.
  c. Generate the test data. Run the tests as follows .::

       yardstick --debug task start tc_prox_<context>_<test>-ports.yaml

     eg.::

       yardstick --debug task start tc_prox_heat_context_l2fwd-4.yaml

  d. Now build the dashboard for the test you just ran. The easiest way to do this is to copy an existing dashboard and rename the
     test and the field names. The procedure to do so is described here. See `opnfv grafana dashboard`_.

How to run NSB Prox Test on an baremetal environment
====================================================

In order to run the NSB PROX test.

  1. Install NSB on Traffic Generator node and Prox in SUT. See `NSB Installation`_

  2. To enter container::

       docker exec -it yardstick /bin/bash

  3. Install baremetal configuration file (POD files)

     a. Go to location of PROX tests in container ::

          cd /home/opnfv/repos/yardstick/samples/vnf_samples/nsut/prox

     b. Install prox-baremetal-2.yam and prox-baremetal-4.yaml for that topology
        into this directory as per baremetal-config-label_

     c. Install and configure ``yardstick.conf`` ::

            cd /etc/yardstick/

        Modify /etc/yardstick/yardstick.conf as per yardstick-config-label_

  4. Execute the test. Eg.::

        yardstick --debug task start ./tc_prox_baremetal_l2fwd-4.yaml

How to run NSB Prox Test on an Openstack environment
====================================================

In order to run the NSB PROX test.

  1. Install NSB on Openstack deployment node. See  `NSB Installation`_

  2. To enter container::

       docker exec -it yardstick /bin/bash

  3. Install configuration file

     a. Goto location of PROX tests in container ::

          cd /home/opnfv/repos/yardstick/samples/vnf_samples/nsut/prox

     b. Install and configure ``yardstick.conf`` ::

            cd /etc/yardstick/

        Modify /etc/yardstick/yardstick.conf as per yardstick-config-label_


  4. Execute the test. Eg.::

        yardstick --debug task start ./tc_prox_heat_context_l2fwd-4.yaml

Frequently Asked Questions
==========================

Here is a list of frequently asked questions.

*NSB Prox does not work on Baremetal, How do I resolve this?*
-------------------------------------------------------------

If PROX NSB does not work on baremetal, problem is either in network configuration or test file.

*Solution*

1. Verify network configuration. Execute existing baremetal test.::

       yardstick --debug task start ./tc_prox_baremetal_l2fwd-4.yaml

   If test does not work then error in network configuration.

      a. Check DPDK on Traffic Generator and SUT via:- ::

           /root/dpdk-17./usertools/dpdk-devbind.py

      b. Verify MAC addresses match ``prox-baremetal-<ports>.yaml`` via ``ifconfig`` and ``dpdk-devbind``

      c. Check your eth port is what you expect. You would not be the first person to think that
         the port your cable is plugged into is ethX when in fact it is ethY. Use
         ethtool to visually confirm that the eth is where you expect.::

            ethtool -p ethX

         A led should start blinking on port. (On both System-Under-Test and Traffic Generator)

      d. Check cable.

         Install Linux kernel network driver and ensure your ports are
         ``bound`` to the driver via ``dpdk-devbind``. Bring up port on both
         SUT and Traffic Generator and check connection.

         i) On SUT and on Traffic Generator::

              ifconfig ethX/enoX up

         ii) Check link

               ethtool ethX/enoX

             See ``Link detected`` if ``yes`` .... Cable is good. If ``no`` you have an issue with your cable/port.

2. If existing baremetal works then issue is with your test. Check the traffic generator gen_<test>-<ports>.cfg to ensure
   it is producing a valid packet.

*How do I debug NSB Prox on Baremetal?*
---------------------------------------

*Solution*

1. Execute the test as follows::

     yardstick --debug task start ./tc_prox_baremetal_l2fwd-4.yaml

2. Login to Traffic Generator as ``root``.::

     cd
     /opt/nsb_bin/prox -f /tmp/gen_<test>-<ports>.cfg

3. Login to SUT as ``root``.::

     cd
     /opt/nsb_bin/prox -f /tmp/handle_<test>-<ports>.cfg

4. Now let's examine the Generator Output. In this case the output of gen_l2fwd-4.cfg.

     .. image:: images/PROX_Gen_GUI.png
        :width: 1000px
        :alt: NSB PROX Traffic Generator GUI

   Now let's examine the output

     1. Indicates the amount of data successfully transmitted on Port 0
     2. Indicates the amount of data successfully received on port 1
     3. Indicates the amount of data successfully handled for port 1

   It appears what is transmitted is received.

   .. Caution::
      The number of packets MAY not exactly match because the ports are read in sequence.

   .. Caution::
      What is transmitted on PORT X may not always be received on same port. Please check the Test scenario.

5. Now lets examine the SUT Output

     .. image:: images/PROX_SUT_GUI.png
        :width: 1400px
        :alt: NSB PROX SUT GUI

   Now lets examine the output

     1. What is received on 0 is transmitted on 1, received on 1 transmitted on 0,
        received on 2 transmitted on 3 and received on 3 transmitted on 2.
     2. No packets are Failed.
     3. No packets are discarded.

  We can also dump the packets being received or transmitted via the following commands. ::

       dump                   Arguments: <core id> <task id> <nb packets>
                              Create a hex dump of <nb_packets> from <task_id> on <core_id> showing how
                              packets have changed between RX and TX.
       dump_rx                Arguments: <core id> <task id> <nb packets>
                              Create a hex dump of <nb_packets> from <task_id> on <core_id> at RX
       dump_tx                Arguments: <core id> <task id> <nb packets>
                              Create a hex dump of <nb_packets> from <task_id> on <core_id> at TX

  eg.::

       dump_tx 1 0 1

*NSB Prox works on Baremetal but not in Openstack. How do I resolve this?*
--------------------------------------------------------------------------

NSB Prox on Baremetal is a lot more forgiving than NSB Prox on Openstack. A badly
formed packed may still work with PROX on Baremetal. However on
Openstack the packet must be correct and all fields of the header correct.
Eg A packet with an invalid Protocol ID would still work in Baremetal
but this packet would be rejected by openstack.

*Solution*

 1. Check the validity of the packet.
 2. Use a known good packet in your test
 3. If using ``Random`` fields in the traffic generator, disable them and retry.


*How do I debug NSB Prox on Openstack?*
---------------------------------------

*Solution*

1. Execute the test as follows::

     yardstick --debug task start --keep-deploy ./tc_prox_heat_context_l2fwd-4.yaml

2. Access docker image if required via::

      docker exec -it yardstick /bin/bash

3. Install openstack credentials.

   Depending on your openstack deployment, the location of these credentials may vary.
   On this platform I do this via::

     scp root@10.237.222.55:/etc/kolla/admin-openrc.sh .
     source ./admin-openrc.sh

4. List Stack details

   a. Get the name of the Stack.

         .. image:: images/PROX_Openstack_stack_list.png
            :width: 1000px
            :alt: NSB PROX openstack stack list

   b. Get the Floating IP of the Traffic Generator & SUT

      This generates a lot of information. Please not the floating IP of the VNF and
      the Traffic Generator.

         .. image:: images/PROX_Openstack_stack_show_a.png
            :width: 1000px
            :alt: NSB PROX openstack stack show (Top)

      From here you can see the floating IP Address of the SUT / VNF

         .. image:: images/PROX_Openstack_stack_show_b.png
            :width: 1000px
            :alt: NSB PROX openstack stack show (Top)

      From here you can see the floating IP Address of the Traffic Generator

   c. Get ssh identity file

      In the docker container locate the identity file.::

        cd /home/opnfv/repos/yardstick/yardstick/resources/files
        ls -lt

5. Login to SUT as ``Ubuntu``.::

     ssh -i ./yardstick_key-01029d1d ubuntu@172.16.2.158

   Change to root::

     sudo su

    Now continue as baremetal.

6. Login to SUT as ``Ubuntu``.::

     ssh -i ./yardstick_key-01029d1d ubuntu@172.16.2.156

   Change to root::

     sudo su

    Now continue as baremetal.

*How do I resolve "Quota exceeded for resources"*
-------------------------------------------------

*Solution*

This usually occurs due to 2 reasons when executing an openstack test.

1. One or more stacks already exists and are consuming all resources. To resolve ::

     openstack stack list

   Response::

     +--------------------------------------+--------------------+-----------------+----------------------+--------------+
     | ID                                   | Stack Name         | Stack Status    | Creation Time        | Updated Time |
     +--------------------------------------+--------------------+-----------------+----------------------+--------------+
     | acb559d7-f575-4266-a2d4-67290b556f15 | yardstick-e05ba5a4 | CREATE_COMPLETE | 2017-12-06T15:00:05Z | None         |
     | 7edf21ce-8824-4c86-8edb-f7e23801a01b | yardstick-08bda9e3 | CREATE_COMPLETE | 2017-12-06T14:56:43Z | None         |
     +--------------------------------------+--------------------+-----------------+----------------------+--------------+

   In this case 2 stacks already exist.

   To remove stack::

     openstack stack delete yardstick-08bda9e3
     Are you sure you want to delete this stack(s) [y/N]? y

2. The openstack configuration quotas are too small.

   The solution is to increase the quota. Use below to query existing quotas::

     openstack quota show

   And to set quota::

     openstack quota set <resource>

*Openstack Cli fails or hangs. How do I resolve this?*
------------------------------------------------------

*Solution*

If it fails due to ::

   Missing value auth-url required for auth plugin password

Check your shell environment for Openstack variables. One of them should contain the authentication URL ::


   OS_AUTH_URL=``https://192.168.72.41:5000/v3``

Or similar. Ensure that openstack configurations are exported. ::

   cat  /etc/kolla/admin-openrc.sh

Result ::

   export OS_PROJECT_DOMAIN_NAME=default
   export OS_USER_DOMAIN_NAME=default
   export OS_PROJECT_NAME=admin
   export OS_TENANT_NAME=admin
   export OS_USERNAME=admin
   export OS_PASSWORD=BwwSEZqmUJA676klr9wa052PFjNkz99tOccS9sTc
   export OS_AUTH_URL=http://193.168.72.41:35357/v3
   export OS_INTERFACE=internal
   export OS_IDENTITY_API_VERSION=3
   export EXTERNAL_NETWORK=yardstick-public

and visible.

If the Openstack Cli appears to hang, then verify the proxys and no_proxy are set correctly.
They should be similar to ::

   FTP_PROXY="http://proxy.ir.intel.com:911/"
   HTTPS_PROXY="http://proxy.ir.intel.com:911/"
   HTTP_PROXY="http://proxy.ir.intel.com:911/"
   NO_PROXY="localhost,127.0.0.1,10.237.222.55,10.237.223.80,10.237.222.134,.ir.intel.com"
   ftp_proxy="http://proxy.ir.intel.com:911/"
   http_proxy="http://proxy.ir.intel.com:911/"
   https_proxy="http://proxy.ir.intel.com:911/"
   no_proxy="localhost,127.0.0.1,10.237.222.55,10.237.223.80,10.237.222.134,.ir.intel.com"

Where

    1) 10.237.222.55 = IP Address of deployment node
    2) 10.237.223.80 = IP Address of Controller node
    3) 10.237.222.134 = IP Address of Compute Node
    4) ir.intel.com = local no proxy


*How to Understand the Grafana output?*
---------------------------------------

         .. image:: images/PROX_Grafana_1.png
            :width: 1000px
            :alt: NSB PROX Grafana_1

         .. image:: images/PROX_Grafana_2.png
            :width: 1000px
            :alt: NSB PROX Grafana_2

         .. image:: images/PROX_Grafana_3.png
            :width: 1000px
            :alt: NSB PROX Grafana_3

         .. image:: images/PROX_Grafana_4.png
            :width: 1000px
            :alt: NSB PROX Grafana_4

A. Test Parameters - Test interval, Duartion, Tolerated Loss and Test Precision

B. Overall No of packets send and received during test

C. Generator Stats - packets sent, received and attempted by Generator

D. Packets Size

E. No of packets received by SUT

F. No of packets forwarded by SUT

G. This is the number of packets sent by the generator per port, for each interval.

H. This is the number of packets received by the generator per port, for each interval.

I. This is the number of packets send and received by the generator and lost by the SUT
   that meet the success criteria

J. This is the changes the Percentage of Line Rate used over a test, The MAX and the
   MIN should converge to within the interval specified as the ``test-precision``.

K. This is the packets Size supported during test. If "N/A" appears in any field the result has not been decided.

L. This is the calculated throughput in MPPS(Million Packets Per second) for this line rate.

M. This is the actual No, of packets sent by the generator in MPPS

N. This is the actual No. of packets received by the generator in MPPS

O. This is the total No. of packets sent by SUT.

P. This is the total No. of packets received by the SUT

Q. This is the total No. of packets dropped. (These packets were sent by the generator but not
   received back by the generator, these may be dropped by the SUT or the Generator)

R. This is the tolerated no of packets that can be dropped.

S. This is the test Throughput in Gbps

T. This is the Latencey per Port

U. This is the CPU Utilization