X-Git-Url: https://gerrit.opnfv.org/gerrit/gitweb?a=blobdiff_plain;f=docs%2Frequirements%2Fvswitchperf_ltd.rst;h=6b882290b18ad4a296e458c7262cb9118af97cd9;hb=26a7262cee1a2961ed5aa76bd969dddeff8ea472;hp=e64de67dcff2bbc062c62e5c9a82c0dee05fbbe9;hpb=a8f7b9adcd4aea96fd44b201c483b7fab08ce658;p=vswitchperf.git diff --git a/docs/requirements/vswitchperf_ltd.rst b/docs/requirements/vswitchperf_ltd.rst index e64de67d..6b882290 100644 --- a/docs/requirements/vswitchperf_ltd.rst +++ b/docs/requirements/vswitchperf_ltd.rst @@ -959,7 +959,15 @@ largest frames of 9K bytes. Types of tests are: 1. Throughput test defines the maximum number of frames per second - that can be transmitted without any error. + that can be transmitted without any error, or 0% loss ratio. + In some Throughput tests (and those tests with long duration), + evaluation of an additional frame loss ratio is suggested. The + current ratio (10^-7 %) is based on understanding the typical + user-to-user packet loss ratio needed for good application + performance and recognizing that a single transfer through a + vswitch must contribute a tiny fraction of user-to-user loss. + Further, the ratio 10^-7 % also recognizes practical limitations + when measuring loss ratio. 2. Latency test measures the time required for a frame to travel from the originating device through the network to the destination device. @@ -1277,11 +1285,11 @@ Test ID: LTD.Throughput.RFC2544.Profile to the DUT's RFC 2544 Throughput as determined by LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta of 0% is equivalent to an offered traffic rate equal to the RFC 2544 - Throughput; A delta of +50% indicates an offered rate half-way - between the Throughput and line-rate, whereas a delta of - -50% indicates an offered rate of half the maximum rate. Therefore the - range of the delta figure is natuarlly bounded at -100% (zero offered - traffic) and +100% (traffic offered at line rate). + Maximum Throughput; A delta of +50% indicates an offered rate half-way + between the Maximum RFC2544 Throughput and line-rate, whereas a delta of + -50% indicates an offered rate of half the RFC 2544 Maximum Throughput. + Therefore the range of the delta figure is natuarlly bounded at -100% + (zero offered traffic) and +100% (traffic offered at line rate). The following deltas to the maximum forwarding rate should be applied: @@ -1853,12 +1861,93 @@ Test ID: LTD.Throughput.RFC2544.WorstN-BestN `RFC2544 `__). - Following may also be collected as part of this test, to determine the vSwitch's performance footprint on the system: + - CPU core utilization. - CPU cache utilization. - Memory footprint. - System bus (QPI, PCI, ...) utilization. - CPU cycles consumed per packet. +.. 3.2.3.1.14 + +Test ID: LTD.Throughput.Overlay.Network..RFC2544.PacketLossRatio +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test + + + NOTE: Throughout this test, four interchangeable overlay technologies are covered by the + same test description. They are: VXLAN, GRE, NVGRE and GENEVE. + + **Prerequisite Test**: N/A + + **Priority**: + + **Description**: + This test evaluates standard switch performance benchmarks for the scenario where an + Overlay Network is deployed for all paths through the vSwitch. Overlay Technologies covered + (replacing in the test name) include: + + - VXLAN + - GRE + - NVGRE + - GENEVE + + Performance will be assessed for each of the following overlay network functions: + + - Encapsulation only + - De-encapsulation only + - Both Encapsulation and De-encapsulation + + For each native packet, the DUT must perform the following operations: + + - Examine the packet and classify its correct overlay net (tunnel) assignment + - Encapsulate the packet + - Switch the packet to the correct port + + For each encapsulated packet, the DUT must perform the following operations: + + - Examine the packet and classify its correct native network assignment + - De-encapsulate the packet, if required + - Switch the packet to the correct port + + The selected frame sizes are those previously defined under `Default + Test Parameters <#DefaultParams>`__. + + Thus, each test comprises an overlay technology, a network function, + and a packet size *with* overlay network overhead included + (but see also the discussion at + https://etherpad.opnfv.org/p/vSwitchTestsDrafts ). + + The test can also be used to determine the average latency of the traffic. + + Under the `RFC2544 `__ + test methodology, the test duration will + include a number of trials; each trial should run for a minimum period + of 60 seconds. A binary search methodology must be applied for each + trial to obtain the final result for Throughput. + + **Expected Result**: At the end of each trial, the presence or absence + of loss determines the modification of offered load for the next trial, + converging on a maximum rate, or + `RFC2544 `__ Throughput with X% + loss (where the value of X is typically equal to zero). + The Throughput load is re-used in related + `RFC2544 `__ tests and other + tests. + + **Metrics Collected**: + The following are the metrics collected for this test: + + - The maximum Throughput in Frames Per Second (FPS) and Mbps of + the DUT for each frame size with X% packet loss. + - The average latency of the traffic flow when passing through the DUT + and VNFs (if testing for latency, note that this average is different from the + test specified in Section 26.3 of + `RFC2544 `__). + - CPU and memory utilization may also be collected as part of this + test, to determine the vSwitch's performance footprint on the system. + + .. 3.2.3.2 Packet Latency tests @@ -1961,9 +2050,9 @@ It is expected that more will be added. .. 3.2.3.3.1 -Test ID: LTD.Scalability.RFC2544.0PacketLoss +Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - **Title**: RFC 2544 0% loss Scalability throughput test + **Title**: RFC 2544 0% loss Flow Scalability throughput test **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the delta Throughput between the single-flow RFC2544 test and this test with @@ -2063,10 +2152,169 @@ Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability - The DUT's 0% packet loss throughput in the presence of cache sharing and memory bandwidth between processes. +.. 3.2.3.3.3 + +Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and + Latency Test + + **Prerequisite Test**: N/A + + **Priority**: + + **Description**: + + This test determines the DUT's throughput rate with X% traffic loss for + a constant load (fixed length frames at a fixed interval time) when the + number of VNFs on the DUT increases. The default loss percentages + to be tested are: - X = 0% - X = 10^-7% . The minimum number of + VNFs to be tested are 3. + + Flow classification should be conducted with L2, L3 and L4 matching + to understand the matching and scaling capability of the vSwitch. The + matching fields which were used as part of the test should be reported + as part of the benchmark report. + + The vSwitch is responsible for forwarding frames between the VNFs + + The SUT (vSwitch and VNF daisy chain) operation should be validated + before running the test. This may be completed by running a burst or + continuous stream of traffic through the SUT to ensure proper operation + before a test. + + **Note**: The traffic rate used to validate SUT operation should be low + enough not to stress the SUT. + + **Note**: Other values can be tested if required by the user. + + **Note**: The same VNF should be used in the "daisy chain" formation. + Each addition of a VNF should be conducted in a new test setup (The DUT + is brought down, then the DUT is brought up again). An atlernative approach + would be to continue to add VNFs without bringing down the DUT. The + approach used needs to be documented as part of the test report. + + The selected frame sizes are those previously defined under `Default + Test Parameters <#DefaultParams>`__. The test can also be used to + determine the average latency of the traffic. + + Under the `RFC2544 `__ + test methodology, the test duration will + include a number of trials; each trial should run for a minimum period + of 60 seconds. A binary search methodology must be applied for each + trial to obtain the final result for Throughput. + + **Expected Result**: At the end of each trial, the presence or absence + of loss determines the modification of offered load for the next trial, + converging on a maximum rate, or + `RFC2544 `__ Throughput with X% + loss. + The Throughput load is re-used in related + `RFC2544 `__ tests and other + tests. + + If the test VNFs are rather light-weight in terms of processing, the test + provides a view of multiple passes through the vswitch on logical + interfaces. In other words, the test produces an optimistic count of + daisy-chained VNFs, but the cumulative effect of traffic on the vSwitch is + "real" (assuming that the vSwitch has some dedicated resources, and the + effects on shared resources is understood). + + + **Metrics Collected**: + The following are the metrics collected for this test: + + - The maximum Throughput in Frames Per Second (FPS) and Mbps of + the DUT for each frame size with X% packet loss. + - The average latency of the traffic flow when passing through the DUT + and VNFs (if testing for latency, note that this average is different from the + test specified in Section 26.3 of + `RFC2544 `__). + - CPU and memory utilization may also be collected as part of this + test, to determine the vSwitch's performance footprint on the system. + +.. 3.2.3.3.4 + +Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + **Title**: VNF Scalability RFC 2544 Throughput and Latency Profile + + **Prerequisite Test**: N/A + + **Priority**: + + **Description**: + + This test reveals how throughput and latency degrades as the number + of VNFs increases and offered rate varies in the region of the DUT's + maximum forwarding rate as determined by + LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss). + For example it can be used to determine if the degradation of throughput + and latency as the number of VNFs and offered rate increases is slow + and graceful, or sudden and severe. The minimum number of VNFs to + be tested is 3. + + The selected frame sizes are those previously defined under `Default + Test Parameters <#DefaultParams>`__. + + The offered traffic rate is described as a percentage delta with respect + to the DUT's RFC 2544 Throughput as determined by + LTD.Throughput.RFC2544.PacketLoss Ratio (0% Packet Loss case). A delta + of 0% is equivalent to an offered traffic rate equal to the RFC 2544 + Throughput; A delta of +50% indicates an offered rate half-way + between the Throughput and line-rate, whereas a delta of + -50% indicates an offered rate of half the maximum rate. Therefore the + range of the delta figure is natuarlly bounded at -100% (zero offered + traffic) and +100% (traffic offered at line rate). + + The following deltas to the maximum forwarding rate should be applied: + + - -50%, -10%, 0%, +10% & +50% + + **Note**: Other values can be tested if required by the user. + + **Note**: The same VNF should be used in the "daisy chain" formation. + Each addition of a VNF should be conducted in a new test setup (The DUT + is brought down, then the DUT is brought up again). An atlernative approach + would be to continue to add VNFs without bringing down the DUT. The + approach used needs to be documented as part of the test report. + + Flow classification should be conducted with L2, L3 and L4 matching + to understand the matching and scaling capability of the vSwitch. The + matching fields which were used as part of the test should be reported + as part of the benchmark report. + + The SUT (vSwitch and VNF daisy chain) operation should be validated + before running the test. This may be completed by running a burst or + continuous stream of traffic through the SUT to ensure proper operation + before a test. + + **Note**: the traffic rate used to validate SUT operation should be low + enough not to stress the SUT + + **Expected Result**: For each packet size a profile should be produced + of how throughput and latency vary with offered rate. + + **Metrics Collected**: + + The following are the metrics collected for this test: + + - The forwarding rate in Frames Per Second (FPS) and Mbps of the DUT + for each delta to the maximum forwarding rate and for each frame + size. + - The average latency for each delta to the maximum forwarding rate and + for each frame size. + - CPU and memory utilization may also be collected as part of this + test, to determine the vSwitch's performance footprint on the system. + - Any failures experienced (for example if the vSwitch crashes, stops + processing packets, restarts or becomes unresponsive to commands) + when the offered load is above Maximum Throughput MUST be recorded + and reported with the results. + .. 3.2.3.4 Activation tests ------------------------ +---------------- The general aim of these tests is to understand the capacity of the and speed with which the vswitch can accommodate new flows. @@ -2228,7 +2476,7 @@ should be required. It is expected that more will be added. .. 3.2.3.6.1 Test ID: LTD.Stress.RFC2544.0PacketLoss -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Title**: RFC 2544 0% Loss CPU OR Memory Stress Test **Prerequisite Test**: @@ -2259,6 +2507,10 @@ Test ID: LTD.Stress.RFC2544.0PacketLoss - The configuration of the stress tool (for example the command line parameters used to start it.) + **Note:** Stress in the test ID can be replaced with the name of the + component being stressed, when reporting the results: + LTD.CPU.RFC2544.0PacketLoss or LTD.Memory.RFC2544.0PacketLoss + .. 3.2.3.7 Summary List of Tests @@ -2277,6 +2529,8 @@ Summary List of Tests - Test ID: LTD.Throughput.RFC2889.ForwardPressure - Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering - Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding + - Test ID: LTD.Throughput.RFC2544.WorstN-BestN + - Test ID: LTD.Throughput.Overlay.Network..RFC2544.PacketLossRatio 2. Packet Latency tests @@ -2285,8 +2539,10 @@ Summary List of Tests 3. Scalability tests - - Test ID: LTD.Scalability.RFC2544.0PacketLoss + - Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss - Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability + - LTD.Scalability.VNF.RFC2544.PacketLossProfile + - LTD.Scalability.VNF.RFC2544.PacketLossRatio 4. Acivation tests @@ -2299,4 +2555,4 @@ Summary List of Tests 6. CPU and memory consumption - - Test ID: LTD.CPU.RFC2544.0PacketLoss + - Test ID: LTD.Stress.RFC2544.0PacketLoss