test_spec: LTD: RFC.WorstN-BestN test

 - Test ID: LTD.Throughput.RFC2544.WorstN-BestN

JIRA: VSPERF-123

Change-Id: Ief04d8c415a77f4a9b77ba0d2a52653376b37ff1
Signed-off-by: Miroslav Miklus <mmiklus@cisco.com>
Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
Reviewed-by: Billy O Mahony <billy.o.mahony@intel.com>
Reviewed-by: Maryam Tahhan <maryam.tahhan@intel.com>
Reviewed-by: Al Morton <acmorton@att.com>

diff --git a/docs/requirements/vswitchperf_ltd.rst b/docs/requirements/vswitchperf_ltd.rst
index 4a08bdd..7739612 100644 (file)
--- a/docs/requirements/vswitchperf_ltd.rst
+++ b/docs/requirements/vswitchperf_ltd.rst
@@ -1759,6 +1759,97 @@ Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
        four test ports are required. One of the ports is connected to the test
        device, so it can send broadcast frames and listen for miss-routed frames.
 
+.. 3.2.3.1.13
+
+Test ID: LTD.Throughput.RFC2544.WorstN-BestN
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+    **Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+    **Prerequisite Test**: N/A
+
+    **Priority**:
+
+    **Description**:
+
+    This test determines the DUT's maximum forwarding rate with X% traffic
+    loss for a constant load (fixed length frames at a fixed interval time).
+    The default loss percentages to be tested are: X = 0%, X = 10^-7%
+
+    Modified RFC 2544 throughput benchmarking methodology aims to quantify
+    the throughput measurement variations observed during standard RFC 2544
+    benchmarking measurements of virtual switches and VNFs. The RFC2544
+    binary search algorithm is modified to use more samples per test trial
+    to drive the binary search and yield statistically more meaningful
+    results. This keeps the heart of the RFC2544 methodology, still relying
+    on the binary search of throughput at specified loss tolerance, while
+    providing more useful information about the range of results seen in
+    testing. Instead of using a single traffic trial per iteration step,
+    each traffic trial is repeated N times and the success/failure of the
+    iteration step is based on these N traffic trials. Two types of revised
+    tests are defined - *Worst-of-N* and *Best-of-N*.
+
+    **Worst-of-N**
+
+    *Worst-of-N* indicates the lowest expected maximum throughput for (
+    packet size, loss tolerance) when repeating the test.
+
+    1.  Repeat the same test run N times at a set packet rate, record each
+        result.
+    2.  Take the WORST result (highest packet loss) out of N result samples,
+        called the Worst-of-N sample.
+    3.  If Worst-of-N sample has loss less than the set loss tolerance, then
+        the step is successful - increase the test traffic rate.
+    4.  If Worst-of-N sample has loss greater than the set loss tolerance
+        then the step failed - decrease the test traffic rate.
+    5.  Go to step 1.
+
+    **Best-of-N**
+
+    *Best-of-N* indicates the highest expected maximum throughput for (
+    packet size, loss tolerance) when repeating the test.
+
+    1.  Repeat the same traffic run N times at a set packet rate, record
+        each result.
+    2.  Take the BEST result (least packet loss) out of N result samples,
+        called the Best-of-N sample.
+    3.  If Best-of-N sample has loss less than the set loss tolerance, then
+        the step is successful - increase the test traffic rate.
+    4.  If Best-of-N sample has loss greater than the set loss tolerance,
+        then the step failed - decrease the test traffic rate.
+    5.  Go to step 1.
+
+    Performing both Worst-of-N and Best-of-N benchmark tests yields lower
+    and upper bounds of expected maximum throughput under the operating
+    conditions, giving a very good indication to the user of the
+    deterministic performance range for the tested setup.
+
+    **Expected Result**: At the end of each trial series, the presence or
+    absence of loss determines the modification of offered load for the
+    next trial series, converging on a maximum rate, or
+    `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput
+    with X% loss.
+    The Throughput load is re-used in related
+    `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+    tests.
+
+    **Metrics Collected**:
+
+    The following are the metrics collected for this test:
+
+    -  The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
+       the DUT for each frame size with X% packet loss.
+    -  The average latency of the traffic flow when passing through the DUT
+       (if testing for latency, note that this average is different from the
+       test specified in Section 26.3 of
+       `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+    -  Following may also be collected as part of this test, to determine
+       the vSwitch's performance footprint on the system:
+      -  CPU core utilization.
+      -  CPU cache utilization.
+      -  Memory footprint.
+      -  System bus (QPI, PCI, ...) utilization.
+      -  CPU cycles consumed per packet.
+
 .. 3.2.3.2
 
 Packet Latency tests