+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Intel Corporation, AT&T and others.
.. 3.1
out in order to objectively measure the current characteristics of a
virtual switch in the Network Function Virtualization Infrastructure
(NFVI) as well as the test pass criteria. The detailed test cases will
-be defined in `Section 2 <#DetailsOfTheLevelTestDesign>`__, preceded by
-the `Document identifier <#DocId>`__ and the `Scope <#Scope>`__.
+be defined in details-of-LTD_, preceded by the doc-id_ and the scope_.
This document is currently in draft form.
.. 3.1.1
+
+.. _doc-id:
+
Document identifier
=========================
The document id will be used to uniquely
identify versions of the LTD. The format for the document id will be:
-OPNFV\_vswitchperf\_LTD\_ver\_NUM\_MONTH\_YEAR\_STATUS, where by the
+OPNFV\_vswitchperf\_LTD\_REL\_STATUS, where by the
status is one of: draft, reviewed, corrected or final. The document id
for this version of the LTD is:
-OPNFV\_vswitchperf\_LTD\_ver\_1.6\_Jan\_15\_DRAFT.
+OPNFV\_vswitchperf\_LTD\_Brahmaputra\_REVIEWED.
.. 3.1.2
+.. _scope:
+
Scope
==========
.. 3.2
+.. _details-of-LTD:
+
===================================
Details of the Level Test Design
===================================
This section describes the features to be tested (
-:ref:_FeaturesToBeTested), the test approach (:ref:_Approach);
+FeaturesToBeTested_), the test approach (Approach_);
it also identifies the sets of test cases or scenarios (
-:ref:_TestIdentification) along with the pass/fail criteria and
+TestIdentification_) along with the pass/fail criteria and
the test deliverables.
.. 3.2.1
they include all required fields, conform to length requirements, pass
integrity checks, etc.
- **Availability and capacity** of the DUT i.e. when the DUT is fully “up”
- and connected:
-
- - Includes power consumption of the CPU (in various power states) and
- system.
- - Includes CPU utilization.
+ and connected, following measurements should be captured for
+ DUT without any network packet load:
+
+ - Includes average power consumption of the CPUs (in various power states) and
+ system over specified period of time. Time period should not be less
+ than 60 seconds.
+ - Includes average per core CPU utilization over specified period of time.
+ Time period should not be less than 60 seconds.
- Includes the number of NIC interfaces supported.
- Includes headroom of VM workload processing cores (i.e. available
for applications).
- **CPU and Memory Consumption Tests** to understand the virtual
switch’s footprint on the system, this includes:
- * CPU utilization
- * Cache utilization
- * Memory footprint
+ * CPU core utilization.
+ * CPU cache utilization.
+ * Memory footprint.
+ * System bus (QPI, PCI, ..) utilization.
+ * Memory lanes utilization.
+ * CPU cycles consumed per packet.
* Time To Establish Flows Tests.
- **Noisy Neighbour Tests**, to understand the effects of resource
Deployment Scenarios
--------------------------
-The following represents possible deployments which can help to
-determine the performance of both the virtual switch and the datapath
-into the VNF:
+The following represents possible deployment test scenarios which can
+help to determine the performance of both the virtual switch and the
+datapaths to physical ports (to NICs) and to logical ports (to VNFs):
.. 3.2.2.2.1
- BIOS configuration: BIOS should be configured for performance where
an explicit option exists, sleep states should be disabled, any
virtualization optimization technologies should be enabled, and
- hyperthreading should also be enabled.
+ hyperthreading should also be enabled, turbo boost and overclocking
+ should be disabled.
**System Validation:**
In this document network “throughput” (measured in millions of frames
per second) is based on RFC 2544, unless otherwise noted. Frame size
refers to Ethernet frames ranging from smallest frames of 64 bytes to
-largest frames of 4K bytes.
+largest frames of 9K bytes.
Types of tests are:
1. Throughput test defines the maximum number of frames per second
- that can be transmitted without any error.
+ that can be transmitted without any error, or 0% loss ratio.
+ In some Throughput tests (and those tests with long duration),
+ evaluation of an additional frame loss ratio is suggested. The
+ current ratio (10^-7 %) is based on understanding the typical
+ user-to-user packet loss ratio needed for good application
+ performance and recognizing that a single transfer through a
+ vswitch must contribute a tiny fraction of user-to-user loss.
+ Further, the ratio 10^-7 % also recognizes practical limitations
+ when measuring loss ratio.
2. Latency test measures the time required for a frame to travel from
the originating device through the network to the destination device.
to the DUT's RFC 2544 Throughput as determined by
LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta
of 0% is equivalent to an offered traffic rate equal to the RFC 2544
- Throughput; A delta of +50% indicates an offered rate half-way
- between the Throughput and line-rate, whereas a delta of
- -50% indicates an offered rate of half the maximum rate. Therefore the
- range of the delta figure is natuarlly bounded at -100% (zero offered
- traffic) and +100% (traffic offered at line rate).
+ Maximum Throughput; A delta of +50% indicates an offered rate half-way
+ between the Maximum RFC2544 Throughput and line-rate, whereas a delta of
+ -50% indicates an offered rate of half the RFC 2544 Maximum Throughput.
+ Therefore the range of the delta figure is natuarlly bounded at -100%
+ (zero offered traffic) and +100% (traffic offered at line rate).
The following deltas to the maximum forwarding rate should be applied:
four test ports are required. One of the ports is connected to the test
device, so it can send broadcast frames and listen for miss-routed frames.
+.. 3.2.3.1.13
+
+Test ID: LTD.Throughput.RFC2544.WorstN-BestN
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's maximum forwarding rate with X% traffic
+ loss for a constant load (fixed length frames at a fixed interval time).
+ The default loss percentages to be tested are: X = 0%, X = 10^-7%
+
+ Modified RFC 2544 throughput benchmarking methodology aims to quantify
+ the throughput measurement variations observed during standard RFC 2544
+ benchmarking measurements of virtual switches and VNFs. The RFC2544
+ binary search algorithm is modified to use more samples per test trial
+ to drive the binary search and yield statistically more meaningful
+ results. This keeps the heart of the RFC2544 methodology, still relying
+ on the binary search of throughput at specified loss tolerance, while
+ providing more useful information about the range of results seen in
+ testing. Instead of using a single traffic trial per iteration step,
+ each traffic trial is repeated N times and the success/failure of the
+ iteration step is based on these N traffic trials. Two types of revised
+ tests are defined - *Worst-of-N* and *Best-of-N*.
+
+ **Worst-of-N**
+
+ *Worst-of-N* indicates the lowest expected maximum throughput for (
+ packet size, loss tolerance) when repeating the test.
+
+ 1. Repeat the same test run N times at a set packet rate, record each
+ result.
+ 2. Take the WORST result (highest packet loss) out of N result samples,
+ called the Worst-of-N sample.
+ 3. If Worst-of-N sample has loss less than the set loss tolerance, then
+ the step is successful - increase the test traffic rate.
+ 4. If Worst-of-N sample has loss greater than the set loss tolerance
+ then the step failed - decrease the test traffic rate.
+ 5. Go to step 1.
+
+ **Best-of-N**
+
+ *Best-of-N* indicates the highest expected maximum throughput for (
+ packet size, loss tolerance) when repeating the test.
+
+ 1. Repeat the same traffic run N times at a set packet rate, record
+ each result.
+ 2. Take the BEST result (least packet loss) out of N result samples,
+ called the Best-of-N sample.
+ 3. If Best-of-N sample has loss less than the set loss tolerance, then
+ the step is successful - increase the test traffic rate.
+ 4. If Best-of-N sample has loss greater than the set loss tolerance,
+ then the step failed - decrease the test traffic rate.
+ 5. Go to step 1.
+
+ Performing both Worst-of-N and Best-of-N benchmark tests yields lower
+ and upper bounds of expected maximum throughput under the operating
+ conditions, giving a very good indication to the user of the
+ deterministic performance range for the tested setup.
+
+ **Expected Result**: At the end of each trial series, the presence or
+ absence of loss determines the modification of offered load for the
+ next trial series, converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput
+ with X% loss.
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The maximum forwarding rate in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - Following may also be collected as part of this test, to determine
+ the vSwitch's performance footprint on the system:
+
+ - CPU core utilization.
+ - CPU cache utilization.
+ - Memory footprint.
+ - System bus (QPI, PCI, ...) utilization.
+ - CPU cycles consumed per packet.
+
+.. 3.2.3.1.14
+
+Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test
+
+
+ NOTE: Throughout this test, four interchangeable overlay technologies are covered by the
+ same test description. They are: VXLAN, GRE, NVGRE and GENEVE.
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+ This test evaluates standard switch performance benchmarks for the scenario where an
+ Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered
+ (replacing <tech> in the test name) include:
+
+ - VXLAN
+ - GRE
+ - NVGRE
+ - GENEVE
+
+ Performance will be assessed for each of the following overlay network functions:
+
+ - Encapsulation only
+ - De-encapsulation only
+ - Both Encapsulation and De-encapsulation
+
+ For each native packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct overlay net (tunnel) assignment
+ - Encapsulate the packet
+ - Switch the packet to the correct port
+
+ For each encapsulated packet, the DUT must perform the following operations:
+
+ - Examine the packet and classify its correct native network assignment
+ - De-encapsulate the packet, if required
+ - Switch the packet to the correct port
+
+ The selected frame sizes are those previously defined under `Default
+ Test Parameters <#DefaultParams>`__.
+
+ Thus, each test comprises an overlay technology, a network function,
+ and a packet size *with* overlay network overhead included
+ (but see also the discussion at
+ https://etherpad.opnfv.org/p/vSwitchTestsDrafts ).
+
+ The test can also be used to determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result for Throughput.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss (where the value of X is typically equal to zero).
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ **Metrics Collected**:
+ The following are the metrics collected for this test:
+
+ - The maximum Throughput in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ and VNFs (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+
.. 3.2.3.2
Packet Latency tests
.. 3.2.3.3.1
-Test ID: LTD.Scalability.RFC2544.0PacketLoss
+Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- **Title**: RFC 2544 0% loss Scalability throughput test
+ **Title**: RFC 2544 0% loss Flow Scalability throughput test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
delta Throughput between the single-flow RFC2544 test and this test with
- The DUT's 0% packet loss throughput in the presence of cache sharing and
memory bandwidth between processes.
+.. 3.2.3.3.3
+
+Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and
+ Latency Test
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test determines the DUT's throughput rate with X% traffic loss for
+ a constant load (fixed length frames at a fixed interval time) when the
+ number of VNFs on the DUT increases. The default loss percentages
+ to be tested are: - X = 0% - X = 10^-7% . The minimum number of
+ VNFs to be tested are 3.
+
+ Flow classification should be conducted with L2, L3 and L4 matching
+ to understand the matching and scaling capability of the vSwitch. The
+ matching fields which were used as part of the test should be reported
+ as part of the benchmark report.
+
+ The vSwitch is responsible for forwarding frames between the VNFs
+
+ The SUT (vSwitch and VNF daisy chain) operation should be validated
+ before running the test. This may be completed by running a burst or
+ continuous stream of traffic through the SUT to ensure proper operation
+ before a test.
+
+ **Note**: The traffic rate used to validate SUT operation should be low
+ enough not to stress the SUT.
+
+ **Note**: Other values can be tested if required by the user.
+
+ **Note**: The same VNF should be used in the "daisy chain" formation.
+ Each addition of a VNF should be conducted in a new test setup (The DUT
+ is brought down, then the DUT is brought up again). An atlernative approach
+ would be to continue to add VNFs without bringing down the DUT. The
+ approach used needs to be documented as part of the test report.
+
+ The selected frame sizes are those previously defined under `Default
+ Test Parameters <#DefaultParams>`__. The test can also be used to
+ determine the average latency of the traffic.
+
+ Under the `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__
+ test methodology, the test duration will
+ include a number of trials; each trial should run for a minimum period
+ of 60 seconds. A binary search methodology must be applied for each
+ trial to obtain the final result for Throughput.
+
+ **Expected Result**: At the end of each trial, the presence or absence
+ of loss determines the modification of offered load for the next trial,
+ converging on a maximum rate, or
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss.
+ The Throughput load is re-used in related
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
+ tests.
+
+ If the test VNFs are rather light-weight in terms of processing, the test
+ provides a view of multiple passes through the vswitch on logical
+ interfaces. In other words, the test produces an optimistic count of
+ daisy-chained VNFs, but the cumulative effect of traffic on the vSwitch is
+ "real" (assuming that the vSwitch has some dedicated resources, and the
+ effects on shared resources is understood).
+
+
+ **Metrics Collected**:
+ The following are the metrics collected for this test:
+
+ - The maximum Throughput in Frames Per Second (FPS) and Mbps of
+ the DUT for each frame size with X% packet loss.
+ - The average latency of the traffic flow when passing through the DUT
+ and VNFs (if testing for latency, note that this average is different from the
+ test specified in Section 26.3 of
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__).
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+
+.. 3.2.3.3.4
+
+Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: VNF Scalability RFC 2544 Throughput and Latency Profile
+
+ **Prerequisite Test**: N/A
+
+ **Priority**:
+
+ **Description**:
+
+ This test reveals how throughput and latency degrades as the number
+ of VNFs increases and offered rate varies in the region of the DUT's
+ maximum forwarding rate as determined by
+ LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss).
+ For example it can be used to determine if the degradation of throughput
+ and latency as the number of VNFs and offered rate increases is slow
+ and graceful, or sudden and severe. The minimum number of VNFs to
+ be tested is 3.
+
+ The selected frame sizes are those previously defined under `Default
+ Test Parameters <#DefaultParams>`__.
+
+ The offered traffic rate is described as a percentage delta with respect
+ to the DUT's RFC 2544 Throughput as determined by
+ LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta
+ of 0% is equivalent to an offered traffic rate equal to the RFC 2544
+ Throughput; A delta of +50% indicates an offered rate half-way
+ between the Throughput and line-rate, whereas a delta of
+ -50% indicates an offered rate of half the maximum rate. Therefore the
+ range of the delta figure is natuarlly bounded at -100% (zero offered
+ traffic) and +100% (traffic offered at line rate).
+
+ The following deltas to the maximum forwarding rate should be applied:
+
+ - -50%, -10%, 0%, +10% & +50%
+
+ **Note**: Other values can be tested if required by the user.
+
+ **Note**: The same VNF should be used in the "daisy chain" formation.
+ Each addition of a VNF should be conducted in a new test setup (The DUT
+ is brought down, then the DUT is brought up again). An atlernative approach
+ would be to continue to add VNFs without bringing down the DUT. The
+ approach used needs to be documented as part of the test report.
+
+ Flow classification should be conducted with L2, L3 and L4 matching
+ to understand the matching and scaling capability of the vSwitch. The
+ matching fields which were used as part of the test should be reported
+ as part of the benchmark report.
+
+ The SUT (vSwitch and VNF daisy chain) operation should be validated
+ before running the test. This may be completed by running a burst or
+ continuous stream of traffic through the SUT to ensure proper operation
+ before a test.
+
+ **Note**: the traffic rate used to validate SUT operation should be low
+ enough not to stress the SUT
+
+ **Expected Result**: For each packet size a profile should be produced
+ of how throughput and latency vary with offered rate.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT
+ for each delta to the maximum forwarding rate and for each frame
+ size.
+ - The average latency for each delta to the maximum forwarding rate and
+ for each frame size.
+ - CPU and memory utilization may also be collected as part of this
+ test, to determine the vSwitch's performance footprint on the system.
+ - Any failures experienced (for example if the vSwitch crashes, stops
+ processing packets, restarts or becomes unresponsive to commands)
+ when the offered load is above Maximum Throughput MUST be recorded
+ and reported with the results.
+
.. 3.2.3.4
Activation tests
------------------------
+----------------
The general aim of these tests is to understand the capacity of the
and speed with which the vswitch can accommodate new flows.
.. 3.2.3.6.1
-Test ID: LTD.CPU.RFC2544.0PacketLoss
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- **Title**: RFC 2544 0% Loss Compute Test
+Test ID: LTD.Stress.RFC2544.0PacketLoss
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Title**: RFC 2544 0% Loss CPU OR Memory Stress Test
**Prerequisite Test**:
**Description**:
The aim of this test is to understand the overall performance of the
- system when a CPU intensive application is run on the same DUT as the
- Virtual Switch. For each frame size, an
+ system when a CPU or Memory intensive application is run on the same DUT as
+ the Virtual Switch. For each frame size, an
LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss) test should be
- performed. Throughout the entire test a CPU intensive application should
- be run on all cores on the system not in use by the Virtual Switch. For
- NUMA system only cores on the same NUMA node are loaded.
+ performed. Throughout the entire test a CPU or Memory intensive application
+ should be run on all cores on the system not in use by the Virtual Switch.
+ For NUMA system only cores on the same NUMA node are loaded.
It is recommended that stress-ng be used for loading the non-Virtual
Switch cores but any stress tool MAY be used.
The following are the metrics collected for this test:
- - CPU utilization of the cores running the Virtual Switch.
+ - Memory and CPU utilization of the cores running the Virtual Switch.
- The number of identity of the cores allocated to the Virtual Switch.
- The configuration of the stress tool (for example the command line
parameters used to start it.)
+ **Note:** Stress in the test ID can be replaced with the name of the
+ component being stressed, when reporting the results:
+ LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss
+
.. 3.2.3.7
Summary List of Tests
- Test ID: LTD.Throughput.RFC2889.ForwardPressure
- Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
- Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
+ - Test ID: LTD.Throughput.RFC2544.WorstN-BestN
+ - Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
2. Packet Latency tests
3. Scalability tests
- - Test ID: LTD.Scalability.RFC2544.0PacketLoss
+ - Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
- Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+ - LTD.Scalability.VNF.RFC2544.PacketLossProfile
+ - LTD.Scalability.VNF.RFC2544.PacketLossRatio
4. Acivation tests
6. CPU and memory consumption
- - Test ID: LTD.CPU.RFC2544.0PacketLoss
+ - Test ID: LTD.Stress.RFC2544.0PacketLoss
Code Review / vswitchperf.git / blobdiff