+.. 3.2.3.1.13
+
+Test ID: LTD.Throughput.RFC2544.WorstN-BestN
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's maximum forwarding rate with X% traffic
+ loss for a constant load (fixed length frames at a fixed interval time).
+ The default loss percentages to be tested are: X = 0%, X = 10^-7%
+
+ Modified RFC 2544 throughput benchmarking methodology aims to quantify
+ the throughput measurement variations observed during standard RFC 2544
+ benchmarking measurements of virtual switches and VNFs. The RFC2544
+ binary search algorithm is modified to use more samples per test trial
+ to drive the binary search and yield statistically more meaningful
+ results. This keeps the heart of the RFC2544 methodology, still relying
+ on the binary search of throughput at specified loss tolerance, while
+ providing more useful information about the range of results seen in
+ testing. Instead of using a single traffic trial per iteration step,
+ each traffic trial is repeated N times and the success/failure of the
+ iteration step is based on these N traffic trials. Two types of revised
+ tests are defined - *Worst-of-N* and *Best-of-N*.
+
+ **Worst-of-N**
+
+ *Worst-of-N* indicates the lowest expected maximum throughput for (
+ packet size, loss tolerance) when repeating the test.
+
+ 1. Repeat the same test run N times at a set packet rate, record each
+ result.
+ 2. Take the WORST result (highest packet loss) out of N result samples,
+ called the Worst-of-N sample.
+ 3. If Worst-of-N sample has loss less than the set loss tolerance, then
+ the step is successful - increase the test traffic rate.
+ 4. If Worst-of-N sample has loss greater than the set loss tolerance
+ then the step failed - decrease the test traffic rate.
+ 5. Go to step 1.
+
+ **Best-of-N**
+
+ *Best-of-N* indicates the highest expected maximum throughput for (
+ packet size, loss tolerance) when repeating the test.
+
+ 1. Repeat the same traffic run N times at a set packet rate, record
+ each result.
+ 2. Take the BEST result (least packet loss) out of N result samples,
+ called the Best-of-N sample.
+ 3. If Best-of-N sample has loss less than the set loss tolerance, then
+ the step is successful - increase the test traffic rate.
+ 4. If Best-of-N sample has loss greater than the set loss tolerance,
+ then the step failed - decrease the test traffic rate.
+ 5. Go to step 1.
+
+ Performing both Worst-of-N and Best-of-N benchmark tests yields lower
+ and upper bounds of expected maximum throughput under the operating
+ conditions, giving a very good indication to the user of the
+ deterministic performance range for the tested setup.
+
+ **Expected Result**: At the end of each trial series, the presence or
+ absence of loss determines the modification of offered load for the
+ next trial series, converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput
+ with X% loss.
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - Following may also be collected as part of this test, to determine
+ the vSwitch's performance footprint on the system:
+
+ - CPU core utilization.
+ - CPU cache utilization.
+ - Memory footprint.
+ - System bus (QPI, PCI, ...) utilization.
+ - CPU cycles consumed per packet.
+
+.. 3.2.3.1.14
+
+Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+
+ NOTE: Throughout this test, four interchangeable overlay technologies are covered by the
+ same test description. They are: VXLAN, GRE, NVGRE and GENEVE.
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+ This test evaluates standard switch performance benchmarks for the scenario where an
+ Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered
+ (replacing <tech> in the test name) include:
+
+ - VXLAN
+ - GRE
+ - NVGRE
+ - GENEVE
+
+ Performance will be assessed for each of the following overlay network functions:
+
+ - Encapsulation only
+ - De-encapsulation only
+ - Both Encapsulation and De-encapsulation
+
+ For each native packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct overlay net (tunnel) assignment
+ - Encapsulate the packet
+ - Switch the packet to the correct port
+
+ For each encapsulated packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct native network assignment
+ - De-encapsulate the packet, if required
+ - Switch the packet to the correct port
+
+ The selected frame sizes are those previously defined under `Default
+ Test Parameters <#DefaultParams>`__.
+
+ Thus, each test comprises an overlay technology, a network function,
+ and a packet size *with* overlay network overhead included
+ (but see also the discussion at
+ https://etherpad.opnfv.org/p/vSwitchTestsDrafts ).
+
+ The test can also be used to determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result for Throughput.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss (where the value of X is typically equal to zero).
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+ The following are the metrics collected for this test:
+
+ - The maximum Throughput in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ and VNFs (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+