X-Git-Url: https://gerrit.opnfv.org/gerrit/gitweb?a=blobdiff_plain;ds=sidebyside;f=test_spec%2Fvswitchperf_ltd.md;h=07d49dc94c5ad1d717463196bcc5f08b4c7ffc43;hb=eec369cd944ee4951fe53f542114498a6b7d23dc;hp=4d4400ecf89bdc900f2be4ef62146fe6ec977ffe;hpb=7850dab27d57d732d3a6d41923540707c6984b2d;p=vswitchperf.git diff --git a/test_spec/vswitchperf_ltd.md b/test_spec/vswitchperf_ltd.md index 4d4400ec..07d49dc9 100755 --- a/test_spec/vswitchperf_ltd.md +++ b/test_spec/vswitchperf_ltd.md @@ -11,6 +11,7 @@ - [2. Details of the Level Test Design](#DetailsOfTheLevelTestDesign) - [2.1. Features to be tested](#FeaturesToBeTested) - [2.2. Approach](#Approach) + - [2.2.1 Details of the Test Report](#TestReport) - [2.3. Test identification](#TestIdentification) - [2.3.1 Throughput tests](#ThroughputTests) - [2.3.2 Packet Delay Tests](#PacketDelayTests) @@ -70,36 +71,36 @@ This section describes the features to be tested ([cf. 2.1](#FeaturesToBeTested) ###2.1. Features to be tested Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) includes measuring the following performance metrics: - - Throughput as defined by [RFC1242]: The maximum rate at which none of the offered frames are dropped by the DUT. The maximum frame rate and bit rate that can be transmitted by the DUT without any error should be recorded. Note there is an equivalent bit rate and a specific layer at which the payloads contribute to the bits. Errors and improperly formed frames or packets are dropped. - - Packet delay introduced by the DUT and its cumulative effect on E2E networks. Frame delay can be measured equivalently. - - Packet delay variation: measured from the perspective of the VNF/application. Packet delay variation is sometimes called "jitter". However, we will avoid the term "jitter" as the term holds different meaning to different groups of people. In this document we will simply use the term packet delay variation. The preferred form for this metric is the PDV form of delay variation defined in [RFC5481]. - - Packet loss (within a configured waiting time at the receiver): All packets sent to the DUT should be accounted for. - - Burst behaviour: measures the ability of the DUT to buffer packets. - - Packet re-ordering: measures the ability of the device under test to maintain sending order throughout transfer to the destination. - - Packet correctness Packets or Frames must be well-formed, in that they include all required fields, conform to length requirements, pass integrity checks, etc. - - Availability and capacity of the DUT i.e. when the DUT is fully âupâ and connected: + - **Throughput** as defined by [RFC1242]: The maximum rate at which **none** of the offered frames are dropped by the DUT. The maximum frame rate and bit rate that can be transmitted by the DUT without any error should be recorded. Note there is an equivalent bit rate and a specific layer at which the payloads contribute to the bits. Errors and improperly formed frames or packets are dropped. + - **Packet delay** introduced by the DUT and its cumulative effect on E2E networks. Frame delay can be measured equivalently. + - **Packet delay variation**: measured from the perspective of the VNF/application. Packet delay variation is sometimes called "jitter". However, we will avoid the term "jitter" as the term holds different meaning to different groups of people. In this document we will simply use the term packet delay variation. The preferred form for this metric is the PDV form of delay variation defined in [RFC5481]. + - **Packet loss** (within a configured waiting time at the receiver): All packets sent to the DUT should be accounted for. + - **Burst behaviour**: measures the ability of the DUT to buffer packets. + - **Packet re-ordering**: measures the ability of the device under test to maintain sending order throughout transfer to the destination. + - **Packet correctness**: packets or Frames must be well-formed, in that they include all required fields, conform to length requirements, pass integrity checks, etc. + - **Availability and capacity** of the DUT i.e. when the DUT is fully âupâ and connected: - Includes power consumption of the CPU (in various power states) and system. - Includes CPU utilization. - - Includes # NIC interfaces supported. + - Includes the number of NIC interfaces supported. - Includes headroom of VM workload processing cores (i.e. available for applications). ###2.2. Approach In order to determine the packet transfer characteristics of a virtual switch, the tests will be broken down into the following categories: - - Throughput Tests to measure the maximum forwarding rate (in frames per second or fps) and bit rate (in Mbps) for a constant load (as defined by [RFC1242]) without traffic loss. - - Packet and Frame Delay Tests to measure average, min and max packet and frame delay for constant loads. - - Stream Performance Tests (TCP, UDP) to measure bulk data transfer performance, i.e. how fast systems can send and receive data through the switch. - - Request/Response Performance Tests (TCP, UDP) the measure the transaction rate through the switch. - - Packet delay tests to understand latency distribution for different packet sizes and over an extended test run to uncover outliers. - - Scalability Tests to understand how the virtual switch performs as the number of flows, active ports, complexity of the forwarding logic's configuration... it has to deal with increases. - - Control Path and Datapath Coupling Tests, to understand how closely coupled the datapath and the control path are as well as the effect of this coupling on the performance of the DUT. - - CPU and Memory Consumption Tests to understand the virtual switchâs footprint on the system, this includes: + - **Throughput Tests** to measure the maximum forwarding rate (in frames per second or fps) and bit rate (in Mbps) for a constant load (as defined by [RFC1242]) without traffic loss. + - **Packet and Frame Delay Tests** to measure average, min and max packet and frame delay for constant loads. + - **Stream Performance Tests** (TCP, UDP) to measure bulk data transfer performance, i.e. how fast systems can send and receive data through the switch. + - **Request/Response Performance** Tests (TCP, UDP) the measure the transaction rate through the switch. + - **Packet Delay Tests** to understand latency distribution for different packet sizes and over an extended test run to uncover outliers. + - **Scalability Tests** to understand how the virtual switch performs as the number of flows, active ports, complexity of the forwarding logic's configuration... it has to deal with increases. + - **Control Path and Datapath Coupling** Tests, to understand how closely coupled the datapath and the control path are as well as the effect of this coupling on the performance of the DUT. + - **CPU and Memory Consumption Tests** to understand the virtual switchâs footprint on the system, this includes: - CPU utilization - Cache utilization - Memory footprint - Time To Establish Flows Tests. - - Noisy Neighbour Tests, to understand the effects of resource sharing on the performance of a virtual switch. + - **Noisy Neighbour Tests**, to understand the effects of resource sharing on the performance of a virtual switch. **Note:** some of the tests above can be conducted simultaneously where the combined results would be insightful, for example Packet/Frame Delay and Scalability. @@ -166,38 +167,40 @@ The following represents possible deployments which can help to determine the pe - Physical port â virtual switch â VNF â virtual switch â VNF â virtual switch â physical port.
- __
- +---------------------------------------------------+ +---------------------------------------------------+ |
- | Guest 1 | | Guest 2 | |
- | +-------------------------------------------+ | | +-------------------------------------------+ | |
- | | Application | | | | Application | | |
- | +-------------------------------------------+ | | +-------------------------------------------+ | |
- | ^ : | | ^ : | |
- | | | | | | | | | Guest
- | : v | | : v | |
- | +---------------+ +---------------+ | | +---------------+ +---------------+ | |
- | | logical port 0| | logical port 1| | | | logical port 0| | logical port 1| | |
- +---+---------------+-----------+---------------+---+ +---+---------------+-----------+---------------+---+__|
- ^ : ^ :
- | | | |
- : v : v __
- +---+---------------+----------+---------------+------------+---------------+-----------+---------------+---+ |
- | | port 0 | | port 1 | | port 2 | | port 3 | | |
- | +---------------+ +---------------+ +---------------+ +---------------+ | |
- | ^ : ^ : | |
- | | | | | | | Host
- | : +--------------------+ v | |
- | +--------------+ +--------------+ | |
- | | phy port | vswitch | phy port | | |
- +---+--------------+--------------------------------------------------------------------+--------------+----+__|
- ^ :
- | |
- : v
- +-----------------------------------------------------------------------------------------------------------+
- | |
- | traffic generator |
- | |
- +-----------------------------------------------------------------------------------------------------------+
+
+ __
+ +----------------------+ +----------------------+ |
+ | Guest 1 | | Guest 2 | |
+ | +---------------+ | | +---------------+ | |
+ | | Application | | | | Application | | |
+ | +---------------+ | | +---------------+ | |
+ | ^ | | | ^ | | |
+ | | v | | | v | | Guests
+ | +---------------+ | | +---------------+ | |
+ | | logical ports | | | | logical ports | | |
+ | | 0 1 | | | | 0 1 | | |
+ +---+---------------+--+ +---+---------------+--+__|
+ ^ : ^ :
+ | | | |
+ : v : v _
+ +---+---------------+---------+---------------+--+ |
+ | | 0 1 | | 3 4 | | |
+ | | logical ports | | logical ports | | |
+ | +---------------+ +---------------+ | |
+ | ^ | ^ | | | Host
+ | | L-----------------+ v | |
+ | +--------------+ +--------------+ | |
+ | | phy ports | vSwitch | phy ports | | |
+ +---+--------------+----------+--------------+---+_|
+ ^ : ^ :
+ | | | |
+ : v : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
- Physical port â virtual switch â VNF.
@@ -303,11 +306,45 @@ The following represents possible deployments which can help to determine the pe
+-----------------------------------------------------------------------------------------------------------+__|
+ - HOST 1(Physical port â virtual switch â VNF â virtual switch â Physical port) â HOST 2(Physical port â virtual switch â VNF â virtual switch â Physical port)
+
+
+
+ +--------------------------------------------+ +------------------------------------------+
+ | +---------------------------------+ | | +--------------------------------+ |
+ | | Application | | | | Application | |
+ | +----------------------------+----+ | | +-------------------------+------+ |
+ | ^ | | | ^ | |
+ | | v | | | v |
+ | +-------+----------+ +------------------+ | | +---------+--------+ +----------------+ |
+ | | Logical port 0 | | Logical port 1 | | | | Logical port 0 | |Logical port 1 | |
+ +-+------------------+--+------------------+-+ +-+------------------+--+------+---------+-+
+ ^ | ^ |
+ | | | |
+ | v | v
+ +-+-------+----------+--+------------------+-+ +-+---------+--------+--+----------------+-+
+ | | Logical port 0 | | Logical port 1 | | | | Logical port 0 | | Logical port 1 | |
+ | +------------------+ +----------+-------+ | | +------------------+ +------+---------+ |
+ | ^ | | | ^ | |
+ | | | | | | | |
+ | | vswitch v | | | vswitch v |
+ | +--------+---------+ +------------------+ | | +---------+--------+ +----------------+ |
+ | | phy port | | phy port | | | | phy port | | phy port | |
+ +-+--------+---------+--+----------+-------+-+ +-+---------+--------+--+------+---------+-+
+ ^ +---------------------------------------+ |
+ | v
+ +----------+----------------------------------------------------------------------------------+-----------+
+ | |
+ | traffic generator |
+ | |
+ +---------------------------------------------------------------------------------------------------------+
+
+
**Note:** For tests where the traffic generator and/or measurement receiver are implemented on VM and connected to the virtual switch through vNIC, the issues of shared resources and interactions between the measurement devices and the device under test must be considered.
####General Methodology:
- To establish the baseline performance of the virtual switch, tests would initially be run with a simple workload in the VNF (the recommended simple workload VNF would be [DPDK]'s testpmd application forwarding packets in a VM). Subsequently, the tests would also be executed with a real Telco workload running in the VNF, which would exercise the virtual switch in the context of higher level Telco NFV use cases, and prove that its underlying characteristics and behaviour can be measured and validated. Suitable real Telco workload VNFs are yet to be identified.
+ To establish the baseline performance of the virtual switch, tests would initially be run with a simple workload in the VNF (the recommended simple workload VNF would be [DPDK]'s testpmd application forwarding packets in a VM or vloop_vnf a simple kernel module that forwards traffic between two network interfaces inside the virtualized environment while bypassing the networking stack). Subsequently, the tests would also be executed with a real Telco workload running in the VNF, which would exercise the virtual switch in the context of higher level Telco NFV use cases, and prove that its underlying characteristics and behaviour can be measured and validated. Suitable real Telco workload VNFs are yet to be identified.
#####Default Test Parameters:
@@ -331,6 +368,9 @@ The following represents possible deployments which can help to determine the pe
**Note**: For throughput tests unless stated otherwise, test configurations should ensure that traffic traverses the installed flows through the switch, i.e. flows are installed and have an appropriate time out that doesn't expire before packet transmission starts.
+#####Flow Classification:
+Virtual switches group packets into flows by processing and matching particular header fields in the packet or frame, or by matching packets based on the input ports into the vSwitch. Thus a flow is considered to be a sequence of packets that have a shared set of header field values or have arrived on the same port. Performance results can vary based on the parameters the vSwitch uses to match for a flow. The recommended flow classification parameters for any vSwitch performance tests are: the input port, the source IP address, the destination IP address and the Ethernet protocol type field. It is essential to increase the flow time-out time on a vSwitch before conducting any performance tests that do not measure the flow set-up time. Normally the first packet of a particular flow will install the flow in the vSwitch which adds an additional latency, subsequent packets of the same flow are not subject to this latency if the flow is already installed on the vSwitch.
+
#####Test Priority
Tests will be assigned a priority in order to determine which tests should be implemented immediately and which tests implementations can be deferred.
@@ -440,6 +480,91 @@ The following represents possible deployments which can help to determine the pe
= 14,880,952.38 frame per second (fps)
+####System isolation and validation
+A key consideration when conducting any sort of benchmark is trying to ensure the consistency and repeatability of test results between runs. When benchmarking the performance of a virtual switch there are many factors that can affect the consistency of results. This section describes these factors and the measures that can be taken to limit their effects. In addition, this section will outline some system tests to validate the platform and the VNF before conducting any vSwitch benchmarking tests.
+
+#####System Isolation
+When conducting a benchmarking test on any SUT, it is essential to limit (and if reasonable, eliminate) any noise that may interfere with the accuracy of the metrics collected by the test. This noise may be introduced by other hardware or software (OS, other applications), and can result in significantly varying performance metrics being collected between consecutive runs of the same test. In the case of characterizing the performance of a virtual switch, there are a number of configuration parameters that can help increase the repeatability and stability of test results, including:
+
+ - OS/GRUB configuration:
+ - maxcpus = n where n >= 0; limits the kernel to using 'n' processors. Only use exactly what you need.
+ - isolcpus: Isolate CPUs from the general scheduler. Isolate all CPUs bar one which will be used by the OS.
+ - use taskset to affinitize the forwarding application and the VNFs onto isolated cores. VNFs and the vSwitch should be allocated their own cores, i.e. must not share the same cores. vCPUs for the VNF should be affinitized to individual cores also.
+ - Limit the amount of background applications that are running and set OS to boot to runlevel 3. Make sure to kill any unnecessary system processes/daemons.
+ - Only enable hardware that you need to use for your test â to ensure there are no other interrupts on the system.
+ - Configure NIC interrupts to only use the cores that are not allocated to any other process (VNF/vSwitch).
+ - NUMA configuration: Any unused sockets in a multi-socket system should be disabled.
+ - CPU pinning: The vSwitch and the VNF should each be affinitized to separate logical cores using a combination of maxcpus, isolcpus and taskset.
+ - BIOS configuration: BIOS should be configured for performance where an explicit option exists, sleep states should be disabled, any virtualization optimization technologies should be enabled, and hyperthreading should also be enabled.
+
+#####System Validation
+System validation is broken down into two sub-categories: Platform validation and VNF validation. The validation test itself involves verifying the forwarding capability and stability for the sub-system under test. The rationale behind system validation is two fold. Firstly to give a tester confidence in the stability of the platform or VNF that is being tested; and secondly to provide base performance comparison points to understand the overhead introduced by the virtual switch.
+
+######Benchmark platform forwarding capability
+This is an OPTIONAL test used to verify the platform and measure the base performance (maximum forwarding rate in fps and latency) that can be achieved by the platform without a vSwitch or a VNF.
+
+The following diagram outlines the set-up for benchmarking Platform forwarding capability:
+
+ __
+ +--------------------------------------------------+ |
+ | +------------------------------------------+ | |
+ | | | | |
+ | | l2fw or DPDK L2FWD app | | Host
+ | | | | |
+ | +------------------------------------------+ | |
+ | | NIC | | |
+ +---+------------------------------------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+
+######Benchmark VNF forwarding capability
+This test is used to verify the VNF and measure the base performance (maximum forwarding rate in fps and latency) that can be achieved by the VNF without a vSwitch. The performance metrics collected by this test will serve as a key comparison point for NIC passthrough technologies and vSwitches. VNF in this context refers to the hypervisor and the VM.
+
+The following diagram outlines the set-up for benchmarking VNF forwarding capability:
+
+ __
+ +--------------------------------------------------+ |
+ | +------------------------------------------+ | |
+ | | | | |
+ | | VNF | | |
+ | | | | |
+ | +------------------------------------------+ | |
+ | | Passthrough/SR-IOV | | Host
+ | +------------------------------------------+ | |
+ | | NIC | | |
+ +---+------------------------------------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+
+######Methodology to benchmark Platform/VNF forwarding capability
+The recommended methodology for the platform/VNF validation and benchmark is:
+ - Run [RFC2889] Maximum Forwarding Rate test, this test will produce maximum forwarding rate and latency results that will serve as the expected values. These expected values can be used in subsequent steps or compared with in subsequent validation tests.
+ - Transmit bidirectional traffic at line rate/max forwarding rate (whichever is higher) for at least 72 hours, measure throughput (fps) and latency.
+ - Note: Traffic should be bidirectional.
+ - Establish a baseline forwarding rate for what the platform can achieve.
+ - Additional validation: After the test has completed for 72 hours run bidirectional traffic at the maximum forwarding rate once more to see if the system is still functional and measure throughput (fps) and latency. Compare the measure the new obtained values with the expected values.
+
+**NOTE 1**: How the Platform is configured for its forwarding capability test (BIOS settings, GRUB configuration, runlevel...) is how the platform should be configured for every test after this
+
+**NOTE 2**: How the VNF is configured for its forwarding capability test (# of vCPUs, vNICs, Memory, affinitizationâ¦) is how it should be configured for every test that uses a VNF after this.
+
+####RFCs for testing switch performance
+The starting point for defining the suite of tests for benchmarking the performance of a virtual switch is to take existing RFCs and standards that were designed to test their physical counterparts and adapting them for testing virtual switches. The rationale behind this is to establish a fair comparison between the performance of virtual and physical switches. This section outlines the RFCs that are used by this specification.
+
#####RFC 1242 Benchmarking Terminology for Network Interconnection Devices
RFC 1242 defines the terminology that is used in describing performance benchmarking tests and their results. Definitions and discussions covered include: Back-to-back, bridge, bridge/router, constant load, data link frame size, frame loss rate, inter frame gap, latency, and many more.
@@ -476,6 +601,56 @@ The following represents possible deployments which can help to determine the pe
#####RFC 6201 Device Reset Characterization
RFC 6201 extends the methodology for characterizing the speed of recovery of the DUT from device or software reset described in RFC 2544.
+
+ ####2.2.1 Details of the Test Report
+ There are a number of parameters related to the system, DUT and tests that can affect the repeatability of a test results and should be recorded. In order to minimise the variation in the results of a test, it is recommended that the test report includes the following information:
+
+ - Hardware details including:
+ - Platform details.
+ - Processor details.
+ - Memory information (see below)
+ - Number of enabled cores.
+ - Number of cores used for the test.
+ - Number of physical NICs, as well as their details (manufacturer, versions, type and the PCI slot they are plugged into).
+ - NIC interrupt configuration.
+ - BIOS version, release date and any configurations that were modified.
+ - Software details including:
+ - OS version (for host and VNF)
+ - Kernel version (for host and VNF)
+ - GRUB boot parameters (for host and VNF).
+ - Hypervisor details (Type and version).
+ - Selected vSwitch, version number or commit id used.
+ - vSwitch launch command line if it has been parameterised.
+ - Memory allocation to the vSwitch â which NUMA node it is using, and how many memory channels.
+ - DPDK or any other SW dependency version number or commit id used.
+ - Memory allocation to a VM - if it's from Hugpages/elsewhere.
+ - VM storage type: snapshot/independent persistent/independent non-persistent.
+ - Number of VMs.
+ - Number of Virtual NICs (vNICs), versions, type and driver.
+ - Number of virtual CPUs and their core affinity on the host.
+ - Number vNIC interrupt configuration.
+ - Thread affinitization for the applications (including the vSwitch itself) on the host.
+ - Details of Resource isolation, such as CPUs designated for Host/Kernel (isolcpu) and CPUs designated for specific processes (taskset).
+ - Memory Details
+ - Total memory
+ - Type of memory
+ - Used memory
+ - Active memory
+ - Inactive memory
+ - Free memory
+ - Buffer memory
+ - Swap cache
+ - Total swap
+ - Used swap
+ - Free swap
+ - Test duration.
+ - Number of flows.
+ - Traffic Information:
+ - Traffic type - UDP, TCP, IMIX / Other.
+ - Packet Sizes.
+ - Deployment Scenario.
+
+ Note: Tests that require additional parameters to be recorded will explicitly specify this.
###2.3. Test identification
@@ -514,6 +689,7 @@ The following represents possible deployments which can help to determine the pe
- The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss.
- The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 of [RFC2544]).
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.