<a name="FeaturesToBeTested"></a>
###2.1. Features to be tested
Characterizing virtual switches (i.e. Device Under Test (DUT) in this document) includes measuring the following performance metrics:
- - Throughput as defined by [RFC1242]: The maximum rate at which none of the offered frames are dropped by the DUT. The maximum frame rate and bit rate that can be transmitted by the DUT without any error should be recorded. Note there is an equivalent bit rate and a specific layer at which the payloads contribute to the bits. Errors and improperly formed frames or packets are dropped.
- - Packet delay introduced by the DUT and its cumulative effect on E2E networks. Frame delay can be measured equivalently.
- - Packet delay variation: measured from the perspective of the VNF/application. Packet delay variation is sometimes called "jitter". However, we will avoid the term "jitter" as the term holds different meaning to different groups of people. In this document we will simply use the term packet delay variation. The preferred form for this metric is the PDV form of delay variation defined in [RFC5481].
- - Packet loss (within a configured waiting time at the receiver): All packets sent to the DUT should be accounted for.
- - Burst behaviour: measures the ability of the DUT to buffer packets.
- - Packet re-ordering: measures the ability of the device under test to maintain sending order throughout transfer to the destination.
- - Packet correctness Packets or Frames must be well-formed, in that they include all required fields, conform to length requirements, pass integrity checks, etc.
- - Availability and capacity of the DUT i.e. when the DUT is fully “up” and connected:
+ - **Throughput** as defined by [RFC1242]: The maximum rate at which **none** of the offered frames are dropped by the DUT. The maximum frame rate and bit rate that can be transmitted by the DUT without any error should be recorded. Note there is an equivalent bit rate and a specific layer at which the payloads contribute to the bits. Errors and improperly formed frames or packets are dropped.
+ - **Packet delay** introduced by the DUT and its cumulative effect on E2E networks. Frame delay can be measured equivalently.
+ - **Packet delay variation**: measured from the perspective of the VNF/application. Packet delay variation is sometimes called "jitter". However, we will avoid the term "jitter" as the term holds different meaning to different groups of people. In this document we will simply use the term packet delay variation. The preferred form for this metric is the PDV form of delay variation defined in [RFC5481].
+ - **Packet loss** (within a configured waiting time at the receiver): All packets sent to the DUT should be accounted for.
+ - **Burst behaviour**: measures the ability of the DUT to buffer packets.
+ - **Packet re-ordering**: measures the ability of the device under test to maintain sending order throughout transfer to the destination.
+ - **Packet correctness**: packets or Frames must be well-formed, in that they include all required fields, conform to length requirements, pass integrity checks, etc.
+ - **Availability and capacity** of the DUT i.e. when the DUT is fully “up” and connected:
- Includes power consumption of the CPU (in various power states) and system.
- Includes CPU utilization.
- - Includes # NIC interfaces supported.
+ - Includes the number of NIC interfaces supported.
- Includes headroom of VM workload processing cores (i.e. available for applications).
<a name="Approach"></a>
###2.2. Approach
In order to determine the packet transfer characteristics of a virtual switch, the tests will be broken down into the following categories:
- - Throughput Tests to measure the maximum forwarding rate (in frames per second or fps) and bit rate (in Mbps) for a constant load (as defined by [RFC1242]) without traffic loss.
- - Packet and Frame Delay Tests to measure average, min and max packet and frame delay for constant loads.
- - Stream Performance Tests (TCP, UDP) to measure bulk data transfer performance, i.e. how fast systems can send and receive data through the switch.
- - Request/Response Performance Tests (TCP, UDP) the measure the transaction rate through the switch.
- - Packet delay tests to understand latency distribution for different packet sizes and over an extended test run to uncover outliers.
- - Scalability Tests to understand how the virtual switch performs as the number of flows, active ports, complexity of the forwarding logic's configuration... it has to deal with increases.
- - Control Path and Datapath Coupling Tests, to understand how closely coupled the datapath and the control path are as well as the effect of this coupling on the performance of the DUT.
- - CPU and Memory Consumption Tests to understand the virtual switch’s footprint on the system, this includes:
+ - **Throughput Tests** to measure the maximum forwarding rate (in frames per second or fps) and bit rate (in Mbps) for a constant load (as defined by [RFC1242]) without traffic loss.
+ - **Packet and Frame Delay Tests** to measure average, min and max packet and frame delay for constant loads.
+ - **Stream Performance Tests** (TCP, UDP) to measure bulk data transfer performance, i.e. how fast systems can send and receive data through the switch.
+ - **Request/Response Performance** Tests (TCP, UDP) the measure the transaction rate through the switch.
+ - **Packet Delay Tests** to understand latency distribution for different packet sizes and over an extended test run to uncover outliers.
+ - **Scalability Tests** to understand how the virtual switch performs as the number of flows, active ports, complexity of the forwarding logic's configuration... it has to deal with increases.
+ - **Control Path and Datapath Coupling** Tests, to understand how closely coupled the datapath and the control path are as well as the effect of this coupling on the performance of the DUT.
+ - **CPU and Memory Consumption Tests** to understand the virtual switch’s footprint on the system, this includes:
- CPU utilization
- Cache utilization
- Memory footprint
- Time To Establish Flows Tests.
- - Noisy Neighbour Tests, to understand the effects of resource sharing on the performance of a virtual switch.
+ - **Noisy Neighbour Tests**, to understand the effects of resource sharing on the performance of a virtual switch.
**Note:** some of the tests above can be conducted simultaneously where the combined results would be insightful, for example Packet/Frame Delay and Scalability.
####General Methodology:
- To establish the baseline performance of the virtual switch, tests would initially be run with a simple workload in the VNF (the recommended simple workload VNF would be [DPDK]'s testpmd application forwarding packets in a VM). Subsequently, the tests would also be executed with a real Telco workload running in the VNF, which would exercise the virtual switch in the context of higher level Telco NFV use cases, and prove that its underlying characteristics and behaviour can be measured and validated. Suitable real Telco workload VNFs are yet to be identified.
+ To establish the baseline performance of the virtual switch, tests would initially be run with a simple workload in the VNF (the recommended simple workload VNF would be [DPDK]'s testpmd application forwarding packets in a VM or vloop_vnf a simple kernel module that forwards traffic between two network interfaces inside the virtualized environment while bypassing the networking stack). Subsequently, the tests would also be executed with a real Telco workload running in the VNF, which would exercise the virtual switch in the context of higher level Telco NFV use cases, and prove that its underlying characteristics and behaviour can be measured and validated. Suitable real Telco workload VNFs are yet to be identified.
<a name="DefaultParams"></a>
#####Default Test Parameters:
- The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss.
- The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 of [RFC2544]).
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
<br/>
- #####Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
- The maximum forwarding rate in Frames Per Second (FPS) and Mbps of the DUT for each frame size with X% packet loss and packet modification operations being performed by the DUT.
- The average latency of the traffic flow when passing through the DUT (if testing for latency, note that this average is different from the test specified in Section 26.3 of [RFC2544]).
- The [RFC5481] PDV form of delay variation on the traffic flow, using the 99th percentile.
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
<br/>
- #####Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
The following are the metrics collected for this test:
- The back-to-back value, which is the the number of frames in the longest burst that the DUT will handle without the loss of any frames.
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
**Deployment scenario**:
- Throughput stability of the DUT.
- Any outliers in the Throughput stability.
- Any unexpected variation in Throughput stability.
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
<br/>
- Throughput stability of the DUT.
- Any outliers in the Throughput stability.
- Any unexpected variation in Throughput stability.
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
<br/>
The following are the metrics collected for this test:
- The Max Forwarding Rate for the DUT for each packet size.
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
<br/>
- #####Test ID: LTD.Throughput.RFC2889.ForwardPressure
The following are the metrics collected for this test:
- - Forwarding rate of the DUT.
+ - Forwarding rate of the DUT in FPS or Mbps.
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
**Deployment scenario**:
- Physical → virtual switch → physical.
<br/>
+ - #####Test ID: LTD.Throughput.RFC2889.AddressCachingCapacity
+ **Title**: RFC2889 Address Caching Capacity Test
+
+ **Prerequisite Test**: N\A
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine the address caching capacity of the DUT for a constant load (fixed length frames at a fixed interval time). The selected frame sizes are those previously defined under [Default Test Parameters](#DefaultParams).
+
+ In order to run this test the aging time, that is the maximum time the DUT will keep a learned address in its flow table, and a set of initial addresses, whose value should be >= 1 and <= the max number supported by the implementation must be known. Please note that if the aging time is configurable it must be longer than the time necessary to produce frames from the external source at the specified rate. If the aging time is fixed the frame rate must be brought down to a value that the external source can produce in a time that is less than the aging time.
+
+ Learning Frames should be sent from an external source to the DUT to install a number of flows. The Learning Frames must have a fixed destination address and must vary the source address of the frames. The DUT should install flows in its flow table based on the varying source addresses.
+ Frames should then be transmitted from an external source at a suitable frame rate to see if the DUT has properly learned all of the addresses. If there is no frame loss and no flooding, the number of addresses sent to the DUT should be increased and the test is repeated until the max number of cached addresses supported by the DUT determined.
+
+ **Expected Result**:
+
+ **Metrics collected**:
+
+ The following are the metrics collected for this test:
+
+ - Number of cached addresses supported by the DUT.
+ - CPU and memory utilization may also be collected as part of this test, to determine the vSwitch's performance footprint on the system.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+<br/>
+ - #####Test ID: LTD.Throughput.RFC2889.AddressLearningRate
+ **Title**: RFC2889 Address Learning Rate Test
+
+ **Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine the rate of address learning of the DUT for a constant load (fixed length frames at a fixed interval time). The selected frame sizes are those previously defined under [Default Test Parameters](#DefaultParams), traffic should be sent with each IPv4/IPv6 address incremented by one. The rate at which the DUT learns a new address should be measured. The maximum caching capacity from LTD.Memory.RFC2889.AddressCachingCapacity should be taken into consideration as the maximum number of addresses for which the learning rate can be obtained.
+
+ **Expected Result**:
+ It may be worthwhile to report the behaviour when operating beyond address capacity - some DUTS may be more friendly to new addresses than others.
+
+ **Metrics collected**:
+
+ The following are the metrics collected for this test:
+
+ - The address learning rate of the DUT.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+<br/>
+ - #####Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
+ **Title**: RFC2889 Error Frames Filtering Test
+
+ **Prerequisite Test**: N\A
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine whether the DUT will propagate any erroneous frames it receives or whether it is capable of filtering out the erroneous frames. Traffic should be sent with erroneous frames included within the flow at random intervals. Illegal frames that must be tested include:
+ - Undersize Frames.
+ - Oversize Frames.
+ - CRC error frames.
+ - Fragment Frames.
+
+ The traffic flow exiting the DUT should be recorded and checked to determine if the erroneous frames where passed through the DUT.
+
+ **Expected Result**:
+ Broken frames are not passed!
+
+ **Metrics collected**
+
+ No Metrics are collected in this test, instead it determines:
+
+ - Whether the DUT will propagate erroneous frames.
+ - Or whether the DUT will correctly filter out any erroneous frames from traffic flow with out removing correct frames.
+
+ **Deployment scenario**:
+
+ - Physical → virtual switch → physical.
+
+<br/>
+ - #####Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
+ **Title**: RFC2889 Broadcast Frame Forwarding Test
+
+ **Prerequisite Test**: N\A
+
+ **Priority**:
+
+ **Description**:
+
+ The aim of this test is to determine the maximum forwarding rate of the DUT when forwarding broadcast traffic. For each frame previously defined under [Default Test Parameters](#DefaultParams), the traffic should be set up as broadcast traffic. The traffic throughput of the DUT should be measured.
+
+ **Expected Result**:
+
+ **Metrics collected**
+
+ The following are the metrics collected for this test:
+
+ - The forwarding rate of the DUT when forwarding broadcast traffic.
+
+<br/>
+----
+<a name="LatencyTests"></a>
+####2.3.2 Packet Latency tests
+ These tests will measure the store and forward latency as well as the packet delay variation for various packet types through the virtual switch.
+
+ The following list is not exhaustive but should indicate the type of tests that should be required. It is expected that more will be added.
+
+ - #####Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
+ **Title**: Initial Packet Processing Latency
+
+ **Prerequisite Test**: N\A
+
+ **Priority**:
+
+ **Description**:
+
+ In some virtual switch architectures, the first packets of a flow will take the system longer to process than subsequent packets in the flow. This test determines the latency for these packets. The test will measure the latency of the packets as they are processed by the flow-setup-path of the DUT. This test will send a single packet to the DUT after a fixed interval of time. The time interval will be equivalent to the amount of time it takes for a flow to time out in the virtual switch. Average packet latency will be determined over 1,000,000 packets.
+
+ For this test, only unidirectional traffic is required.
+
+ **Expected Result**:
+ The average latency for the initial packet of all flows should be greater than the latency of subsequent traffic.
+
+ **Metrics Collected**:
+
+ The following are the metrics collected for this test:
+
+ - Average latency of the initial packets of all flows that are processed by the DUT.
+
+ **Deployment scenario**:
+
+ - Physical → Virtual Switch → Physical.
+<br/>
+----
[RFC1242]:(http://www.ietf.org/rfc/rfc1242.txt)
[RFC2544]:(http://www.ietf.org/rfc/rfc2544.txt)
[RFC2885]:(http://www.ietf.org/rfc/rfc2885.txt)
Code Review / vswitchperf.git / blobdiff