+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation, AT&T and others.
.. 3.1
out in order to objectively measure the current characteristics of a
virtual switch in the Network Function Virtualization Infrastructure
(NFVI) as well as the test pass criteria. The detailed test cases will
-be defined in `Section 2 <#DetailsOfTheLevelTestDesign>`__, preceded by
-the `Document identifier <#DocId>`__ and the `Scope <#Scope>`__.
+be defined in details-of-LTD_, preceded by the doc-id_ and the scope_.
This document is currently in draft form.
.. 3.1.1
+
+.. _doc-id:
+
Document identifier
=========================
The document id will be used to uniquely
identify versions of the LTD. The format for the document id will be:
-OPNFV\_vswitchperf\_LTD\_ver\_NUM\_MONTH\_YEAR\_STATUS, where by the
+OPNFV\_vswitchperf\_LTD\_REL\_STATUS, where by the
status is one of: draft, reviewed, corrected or final. The document id
for this version of the LTD is:
-OPNFV\_vswitchperf\_LTD\_ver\_1.6\_Jan\_15\_DRAFT.
+OPNFV\_vswitchperf\_LTD\_Brahmaputra\_REVIEWED.
.. 3.1.2
+.. _scope:
+
Scope
==========
.. 3.2
+.. _details-of-LTD:
+
===================================
Details of the Level Test Design
===================================
This section describes the features to be tested (
-:ref:_FeaturesToBeTested), the test approach (:ref:_Approach);
+FeaturesToBeTested_), the test approach (Approach_);
it also identifies the sets of test cases or scenarios (
-:ref:_TestIdentification) along with the pass/fail criteria and
+TestIdentification_) along with the pass/fail criteria and
the test deliverables.
.. 3.2.1
Types of tests are:
1. Throughput test defines the maximum number of frames per second
- that can be transmitted without any error.
+ that can be transmitted without any error, or 0% loss ratio.
+ In some Throughput tests (and those tests with long duration),
+ evaluation of an additional frame loss ratio is suggested. The
+ current ratio (10^-7 %) is based on understanding the typical
+ user-to-user packet loss ratio needed for good application
+ performance and recognizing that a single transfer through a
+ vswitch must contribute a tiny fraction of user-to-user loss.
+ Further, the ratio 10^-7 % also recognizes practical limitations
+ when measuring loss ratio.
2. Latency test measures the time required for a frame to travel from
the originating device through the network to the destination device.
to the DUT's RFC 2544 Throughput as determined by
LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta
of 0% is equivalent to an offered traffic rate equal to the RFC 2544
- Throughput; A delta of +50% indicates an offered rate half-way
- between the Throughput and line-rate, whereas a delta of
- -50% indicates an offered rate of half the maximum rate. Therefore the
- range of the delta figure is natuarlly bounded at -100% (zero offered
- traffic) and +100% (traffic offered at line rate).
+ Maximum Throughput; A delta of +50% indicates an offered rate half-way
+ between the Maximum RFC2544 Throughput and line-rate, whereas a delta of
+ -50% indicates an offered rate of half the RFC 2544 Maximum Throughput.
+ Therefore the range of the delta figure is natuarlly bounded at -100%
+ (zero offered traffic) and +100% (traffic offered at line rate).
The following deltas to the maximum forwarding rate should be applied:
`RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
- Following may also be collected as part of this test, to determine
the vSwitch's performance footprint on the system:
+
- CPU core utilization.
- CPU cache utilization.
- Memory footprint.
- System bus (QPI, PCI, ...) utilization.
- CPU cycles consumed per packet.
+.. 3.2.3.1.14
+
+Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+
+ NOTE: Throughout this test, four interchangeable overlay technologies are covered by the
+ same test description. They are: VXLAN, GRE, NVGRE and GENEVE.
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+ This test evaluates standard switch performance benchmarks for the scenario where an
+ Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered
+ (replacing <tech> in the test name) include:
+
+ - VXLAN
+ - GRE
+ - NVGRE
+ - GENEVE
+
+ Performance will be assessed for each of the following overlay network functions:
+
+ - Encapsulation only
+ - De-encapsulation only
+ - Both Encapsulation and De-encapsulation
+
+ For each native packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct overlay net (tunnel) assignment
+ - Encapsulate the packet
+ - Switch the packet to the correct port
+
+ For each encapsulated packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct native network assignment
+ - De-encapsulate the packet, if required
+ - Switch the packet to the correct port
+
+ The selected frame sizes are those previously defined under `Default
+ Test Parameters <#DefaultParams>`__.
+
+ Thus, each test comprises an overlay technology, a network function,
+ and a packet size *with* overlay network overhead included
+ (but see also the discussion at
+ https://etherpad.opnfv.org/p/vSwitchTestsDrafts ).
+
+ The test can also be used to determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result for Throughput.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss (where the value of X is typically equal to zero).
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+ The following are the metrics collected for this test:
+
+ - The maximum Throughput in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ and VNFs (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+
.. 3.2.3.2
Packet Latency tests
.. 3.2.3.3.1
-Test ID: LTD.Scalability.RFC2544.0PacketLoss
+Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- **Title**: RFC 2544 0% loss Scalability throughput test
+ **Title**: RFC 2544 0% loss Flow Scalability throughput test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
delta Throughput between the single-flow RFC2544 test and this test with
- The DUT's 0% packet loss throughput in the presence of cache sharing and
memory bandwidth between processes.
+.. 3.2.3.3.3
+
+Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and
+ Latency Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's throughput rate with X% traffic loss for
+ a constant load (fixed length frames at a fixed interval time) when the
+ number of VNFs on the DUT increases. The default loss percentages
+ to be tested are: - X = 0% - X = 10^-7% . The minimum number of
+ VNFs to be tested are 3.
+
+ Flow classification should be conducted with L2, L3 and L4 matching
+ to understand the matching and scaling capability of the vSwitch. The
+ matching fields which were used as part of the test should be reported
+ as part of the benchmark report.
+
+ The vSwitch is responsible for forwarding frames between the VNFs
+
+ The SUT (vSwitch and VNF daisy chain) operation should be validated
+ before running the test. This may be completed by running a burst or
+ continuous stream of traffic through the SUT to ensure proper operation
+ before a test.
+
+ **Note**: The traffic rate used to validate SUT operation should be low
+ enough not to stress the SUT.
+
+ **Note**: Other values can be tested if required by the user.
+
+ **Note**: The same VNF should be used in the "daisy chain" formation.
+ Each addition of a VNF should be conducted in a new test setup (The DUT
+ is brought down, then the DUT is brought up again). An atlernative approach
+ would be to continue to add VNFs without bringing down the DUT. The
+ approach used needs to be documented as part of the test report.
+
+ The selected frame sizes are those previously defined under `Default
+ Test Parameters <#DefaultParams>`__. The test can also be used to
+ determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result for Throughput.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss.
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ If the test VNFs are rather light-weight in terms of processing, the test
+ provides a view of multiple passes through the vswitch on logical
+ interfaces. In other words, the test produces an optimistic count of
+ daisy-chained VNFs, but the cumulative effect of traffic on the vSwitch is
+ "real" (assuming that the vSwitch has some dedicated resources, and the
+ effects on shared resources is understood).
+
+
+ **Metrics Collected**:
+ The following are the metrics collected for this test:
+
+ - The maximum Throughput in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ and VNFs (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+.. 3.2.3.3.4
+
+Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: VNF Scalability RFC 2544 Throughput and Latency Profile
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test reveals how throughput and latency degrades as the number
+ of VNFs increases and offered rate varies in the region of the DUT's
+ maximum forwarding rate as determined by
+ LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss).
+ For example it can be used to determine if the degradation of throughput
+ and latency as the number of VNFs and offered rate increases is slow
+ and graceful, or sudden and severe. The minimum number of VNFs to
+ be tested is 3.
+
+ The selected frame sizes are those previously defined under `Default
+ Test Parameters <#DefaultParams>`__.
+
+ The offered traffic rate is described as a percentage delta with respect
+ to the DUT's RFC 2544 Throughput as determined by
+ LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta
+ of 0% is equivalent to an offered traffic rate equal to the RFC 2544
+ Throughput; A delta of +50% indicates an offered rate half-way
+ between the Throughput and line-rate, whereas a delta of
+ -50% indicates an offered rate of half the maximum rate. Therefore the
+ range of the delta figure is natuarlly bounded at -100% (zero offered
+ traffic) and +100% (traffic offered at line rate).
+
+ The following deltas to the maximum forwarding rate should be applied:
+
+ - -50%, -10%, 0%, +10% & +50%
+
+ **Note**: Other values can be tested if required by the user.
+
+ **Note**: The same VNF should be used in the "daisy chain" formation.
+ Each addition of a VNF should be conducted in a new test setup (The DUT
+ is brought down, then the DUT is brought up again). An atlernative approach
+ would be to continue to add VNFs without bringing down the DUT. The
+ approach used needs to be documented as part of the test report.
+
+ Flow classification should be conducted with L2, L3 and L4 matching
+ to understand the matching and scaling capability of the vSwitch. The
+ matching fields which were used as part of the test should be reported
+ as part of the benchmark report.
+
+ The SUT (vSwitch and VNF daisy chain) operation should be validated
+ before running the test. This may be completed by running a burst or
+ continuous stream of traffic through the SUT to ensure proper operation
+ before a test.
+
+ **Note**: the traffic rate used to validate SUT operation should be low
+ enough not to stress the SUT
+
+ **Expected Result**: For each packet size a profile should be produced
+ of how throughput and latency vary with offered rate.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT
+ for each delta to the maximum forwarding rate and for each frame
+ size.
+ - The average latency for each delta to the maximum forwarding rate and
+ for each frame size.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+ - Any failures experienced (for example if the vSwitch crashes, stops
+ processing packets, restarts or becomes unresponsive to commands)
+ when the offered load is above Maximum Throughput MUST be recorded
+ and reported with the results.
+
.. 3.2.3.4
Activation tests
------------------------
+----------------
The general aim of these tests is to understand the capacity of the
and speed with which the vswitch can accommodate new flows.
.. 3.2.3.6.1
-Test ID: LTD.CPU.RFC2544.0PacketLoss
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- **Title**: RFC 2544 0% Loss Compute Test
+Test ID: LTD.Stress.RFC2544.0PacketLoss
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: RFC 2544 0% Loss CPU OR Memory Stress Test
**Prerequisite Test**:
**Description**:
The aim of this test is to understand the overall performance of the
- system when a CPU intensive application is run on the same DUT as the
- Virtual Switch. For each frame size, an
+ system when a CPU or Memory intensive application is run on the same DUT as
+ the Virtual Switch. For each frame size, an
LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be
- performed. Throughout the entire test a CPU intensive application should
- be run on all cores on the system not in use by the Virtual Switch. For
- NUMA system only cores on the same NUMA node are loaded.
+ performed. Throughout the entire test a CPU or Memory intensive application
+ should be run on all cores on the system not in use by the Virtual Switch.
+ For NUMA system only cores on the same NUMA node are loaded.
It is recommended that stress-ng be used for loading the non-Virtual
Switch cores but any stress tool MAY be used.
The following are the metrics collected for this test:
- - CPU utilization of the cores running the Virtual Switch.
+ - Memory and CPU utilization of the cores running the Virtual Switch.
- The number of identity of the cores allocated to the Virtual Switch.
- The configuration of the stress tool (for example the command line
parameters used to start it.)
+ **Note:** Stress in the test ID can be replaced with the name of the
+ component being stressed, when reporting the results:
+ LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss
+
.. 3.2.3.7
Summary List of Tests
- Test ID: LTD.Throughput.RFC2889.ForwardPressure
- Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
- Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
+ - Test ID: LTD.Throughput.RFC2544.WorstN-BestN
+ - Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
2. Packet Latency tests
3. Scalability tests
- - Test ID: LTD.Scalability.RFC2544.0PacketLoss
+ - Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
- Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+ - LTD.Scalability.VNF.RFC2544.PacketLossProfile
+ - LTD.Scalability.VNF.RFC2544.PacketLossRatio
4. Acivation tests
6. CPU and memory consumption
- - Test ID: LTD.CPU.RFC2544.0PacketLoss
+ - Test ID: LTD.Stress.RFC2544.0PacketLoss