+
+Trex
+----
+
+Installation
+~~~~~~~~~~~~
+
+Trex architecture overview and general installation instructions
+can be found here:
+
+https://trex-tgn.cisco.com/trex/doc/trex_stateless.html
+
+You can directly download from GitHub:
+
+.. code-block:: console
+
+ git clone https://github.com/cisco-system-traffic-generator/trex-core
+
+and use the same Trex version for both server and client API.
+
+**NOTE:** The Trex API version used by VSPERF is defined by variable ``TREX_TAG``
+in file ``src/package-list.mk``.
+
+.. code-block:: console
+
+ git checkout v2.38
+
+or Trex latest release you can download from here:
+
+.. code-block:: console
+
+ wget --no-cache http://trex-tgn.cisco.com/trex/release/latest
+
+After download, Trex repo has to be built:
+
+.. code-block:: console
+
+ cd trex-core/linux_dpdk
+ ./b configure (run only once)
+ ./b build
+
+Next step is to create a minimum configuration file. It can be created by script ``dpdk_setup_ports.py``.
+The script with parameter ``-i`` will run in interactive mode and it will create file ``/etc/trex_cfg.yaml``.
+
+.. code-block:: console
+
+ cd trex-core/scripts
+ sudo ./dpdk_setup_ports.py -i
+
+Or example of configuration file can be found at location below, but it must be updated manually:
+
+.. code-block:: console
+
+ cp trex-core/scripts/cfg/simple_cfg /etc/trex_cfg.yaml
+
+For additional information about configuration file see official documentation (chapter 3.1.2):
+
+https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_creating_minimum_configuration_file
+
+After compilation and configuration it is possible to run trex server in stateless mode.
+It is neccesary for proper connection between Trex server and VSPERF.
+
+.. code-block:: console
+
+ cd trex-core/scripts/
+ ./t-rex-64 -i
+
+**NOTE:** Please check your firewall settings at both DUT and T-Rex server.
+Firewall must allow a connection from DUT (VSPERF) to the T-Rex server running
+at TCP port 4501.
+
+**NOTE:** For high speed cards it may be advantageous to start T-Rex with more transmit queues/cores.
+
+.. code-block:: console
+
+ cd trex-cores/scripts/
+ ./t-rex-64 -i -c 10
+
+For additional information about Trex stateless mode see Trex stateless documentation:
+
+https://trex-tgn.cisco.com/trex/doc/trex_stateless.html
+
+**NOTE:** One will need to set up ssh login to not use passwords between the server
+running Trex and the device under test (running the VSPERF test
+infrastructure). This is because VSPERF on one server uses 'ssh' to
+configure and run Trex upon the other server.
+
+One can set up this ssh access by doing the following on both servers:
+
+.. code-block:: console
+
+ ssh-keygen -b 2048 -t rsa
+ ssh-copy-id <other server>
+
+Configuration
+~~~~~~~~~~~~~
+
+Connection information for Trex must be supplied inside the custom configuration
+file. The following parameters must be set to allow for proper connections to the host with Trex.
+Example of this configuration is in conf/03_traffic.conf or conf/10_custom.conf.
+
+.. code-block:: console
+
+ TRAFFICGEN_TREX_HOST_IP_ADDR = ''
+ TRAFFICGEN_TREX_USER = ''
+ TRAFFICGEN_TREX_BASE_DIR = ''
+
+TRAFFICGEN_TREX_USER has to have sudo permission and password-less access.
+TRAFFICGEN_TREX_BASE_DIR is the place, where is stored 't-rex-64' file.
+
+It is possible to specify the accuracy of RFC2544 Throughput measurement.
+Threshold below defines maximal difference between frame rate of successful
+(i.e. defined frameloss was reached) and unsuccessful (i.e. frameloss was
+exceeded) iterations.
+
+Default value of this parameter is defined in conf/03_traffic.conf as follows:
+
+.. code-block:: console
+
+ TRAFFICGEN_TREX_RFC2544_TPUT_THRESHOLD = ''
+
+T-Rex can have learning packets enabled. For certain tests it may be beneficial
+to send some packets before starting test traffic to allow switch learning to take
+place. This can be adjusted with the following configurations:
+
+.. code-block:: console
+
+ TRAFFICGEN_TREX_LEARNING_MODE=True
+ TRAFFICGEN_TREX_LEARNING_DURATION=5
+
+Latency measurements have impact on T-Rex performance. Thus vswitchperf uses a separate
+latency stream for each direction with limited speed. This workaround is used for RFC2544
+**Throughput** and **Continuous** traffic types. In case of **Burst** traffic type,
+the latency statistics are measured for all frames in the burst. Collection of latency
+statistics is driven by configuration option ``TRAFFICGEN_TREX_LATENCY_PPS`` as follows:
+
+ * value ``0`` - disables latency measurements
+ * non zero integer value - enables latency measurements; In case of Throughput
+ and Continuous traffic types, it specifies a speed of latency specific stream
+ in PPS. In case of burst traffic type, it enables latency measurements for all frames.
+
+.. code-block:: console
+
+ TRAFFICGEN_TREX_LATENCY_PPS = 1000
+
+SR-IOV and Multistream layer 2
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+T-Rex by default only accepts packets on the receive side if the destination mac matches the
+MAC address specified in the /etc/trex-cfg.yaml on the server side. For SR-IOV this creates
+challenges with modifying the MAC address in the traffic profile to correctly flow packets
+through specified VFs. To remove this limitation enable promiscuous mode on T-Rex to allow
+all packets regardless of the destination mac to be accepted.
+
+This also creates problems when doing multistream at layer 2 since the source macs will be
+modified. Enable Promiscuous mode when doing multistream at layer 2 testing with T-Rex.
+
+.. code-block:: console
+
+ TRAFFICGEN_TREX_PROMISCUOUS=True
+
+Card Bandwidth Options
+~~~~~~~~~~~~~~~~~~~~~~
+
+T-Rex API will attempt to retrieve the highest possible speed from the card using internal
+calls to port information. If you are using two separate cards then it will take the lowest
+of the two cards as the max speed. If necessary you can try to force the API to use a
+specific maximum speed per port. The below configurations can be adjusted to enable this.
+
+.. code-block:: console
+
+ TRAFFICGEN_TREX_FORCE_PORT_SPEED = True
+ TRAFFICGEN_TREX_PORT_SPEED = 40000 # 40 gig
+
+**Note::** Setting higher than possible speeds will result in unpredictable behavior when running
+tests such as duration inaccuracy and/or complete test failure.
+
+RFC2544 Validation
+~~~~~~~~~~~~~~~~~~
+
+T-Rex can perform a verification run for a longer duration once the binary search of the
+RFC2544 trials have completed. This duration should be at least 60 seconds. This is similar
+to other traffic generator functionality where a more sustained time can be attempted to
+verify longer runs from the result of the search. This can be configured with the following
+params
+
+.. code-block:: console
+
+ TRAFFICGEN_TREX_VERIFICATION_MODE = False
+ TRAFFICGEN_TREX_VERIFICATION_DURATION = 60
+ TRAFFICGEN_TREX_MAXIMUM_VERIFICATION_TRIALS = 10
+
+The duration and maximum number of attempted verification trials can be set to change the
+behavior of this step. If the verification step fails, it will resume the binary search
+with new values where the maximum output will be the last attempted frame rate minus the
+current set thresh hold.
+
+Scapy frame definition
+~~~~~~~~~~~~~~~~~~~~~~
+
+It is possible to use a SCAPY frame definition to generate various network protocols
+by the **T-Rex** traffic generator. In case that particular network protocol layer
+is disabled by the TRAFFIC dictionary (e.g. TRAFFIC['vlan']['enabled'] = False),
+then disabled layer will be removed from the scapy format definition by VSPERF.
+
+The scapy frame definition can refer to values defined by the TRAFFIC dictionary
+by following keywords. These keywords are used in next examples.
+
+* ``Ether_src`` - refers to ``TRAFFIC['l2']['srcmac']``
+* ``Ether_dst`` - refers to ``TRAFFIC['l2']['dstmac']``
+* ``IP_proto`` - refers to ``TRAFFIC['l3']['proto']``
+* ``IP_PROTO`` - refers to upper case version of ``TRAFFIC['l3']['proto']``
+* ``IP_src`` - refers to ``TRAFFIC['l3']['srcip']``
+* ``IP_dst`` - refers to ``TRAFFIC['l3']['dstip']``
+* ``IP_PROTO_sport`` - refers to ``TRAFFIC['l4']['srcport']``
+* ``IP_PROTO_dport`` - refers to ``TRAFFIC['l4']['dstport']``
+* ``Dot1Q_prio`` - refers to ``TRAFFIC['vlan']['priority']``
+* ``Dot1Q_id`` - refers to ``TRAFFIC['vlan']['cfi']``
+* ``Dot1Q_vlan`` - refers to ``TRAFFIC['vlan']['id']``
+
+In following examples of SCAPY frame definition only relevant parts of TRAFFIC
+dictionary are shown. The rest of the TRAFFIC dictionary is set to default values
+as they are defined in ``conf/03_traffic.conf``.
+
+Please check official documentation of SCAPY project for details about SCAPY frame
+definition and supported network layers at: http://www.secdev.org/projects/scapy
+
+#. Generate ICMP frames:
+
+ .. code-block:: console
+
+ 'scapy': {
+ 'enabled': True,
+ '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/IP(proto={IP_proto}, src={IP_src}, dst={IP_dst})/ICMP()',
+ '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/IP(proto={IP_proto}, src={IP_dst}, dst={IP_src})/ICMP()',
+ }
+
+#. Generate IPv6 ICMP Echo Request
+
+ .. code-block:: console
+
+ 'l3' : {
+ 'srcip': 'feed::01',
+ 'dstip': 'feed::02',
+ },
+ 'scapy': {
+ 'enabled': True,
+ '0' : 'Ether(src={Ether_src}, dst={Ether_dst})/IPv6(src={IP_src}, dst={IP_dst})/ICMPv6EchoRequest()',
+ '1' : 'Ether(src={Ether_dst}, dst={Ether_src})/IPv6(src={IP_dst}, dst={IP_src})/ICMPv6EchoRequest()',
+ }
+
+#. Generate SCTP frames:
+
+ Example uses default SCAPY frame definition, which can reflect ``TRAFFIC['l3']['proto']`` settings. The same
+ approach can be used to generate other protocols, e.g. TCP.
+
+ .. code-block:: console
+
+ 'l3' : {
+ 'proto' : 'sctp',
+ },
+ 'scapy': {
+ 'enabled': True,
+ }
+