.. 3.1
+============
Introduction
-===============
+============
The intention of this Level Test Design (LTD) document is to specify the set of
tests to carry out in order to objectively measure the current characteristics
of a virtual switch in the Network Function Virtualization Infrastructure
(NFVI) as well as the test pass criteria. The detailed test cases will be
-defined in details-of-LTD_, preceded by the doc-id_ and the scope_.
+defined in details-of-LTD_, preceded by the doc-id-of-LTD_ and the scope-of-LTD_.
This document is currently in draft form.
.. 3.1.1
-.. _doc-id:
+.. _doc-id-of-LTD:
Document identifier
-=========================
+===================
The document id will be used to uniquely
identify versions of the LTD. The format for the document id will be:
.. 3.1.2
-.. _scope:
+.. _scope-of-LTD:
Scope
-==========
+=====
The main purpose of this project is to specify a suite of
performance tests in order to objectively measure the current packet
.. 3.1.3
References
-===============
+==========
* `RFC 1242 Benchmarking Terminology for Network Interconnection
Devices <http://www.ietf.org/rfc/rfc1242.txt>`__
.. _details-of-LTD:
-===================================
+================================
Details of the Level Test Design
-===================================
+================================
-This section describes the features to be tested (FeaturesToBeTested_), and
-identifies the sets of test cases or scenarios (TestIdentification_).
+This section describes the features to be tested (FeaturesToBeTested-of-LTD_), and
+identifies the sets of test cases or scenarios (TestIdentification-of-LTD_).
.. 3.2.1
-.. _FeaturesToBeTested:
+.. _FeaturesToBeTested-of-LTD:
Features to be tested
-==========================
+=====================
Characterizing virtual switches (i.e. Device Under Test (DUT) in this document)
includes measuring the following performance metrics:
.. 3.2.2
-.. _TestIdentification:
+.. _TestIdentification-of-LTD:
Test identification
-=========================
+===================
-.. 3.2.3.1
+.. 3.2.2.1
Throughput tests
-----------------------
+----------------
+
The following tests aim to determine the maximum forwarding rate that
can be achieved with a virtual switch. The list is not exhaustive but
should indicate the type of tests that should be required. It is
.. 3.2.2.1.1
Test ID: LTD.Throughput.RFC2544.PacketLossRatio
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test
**Prerequisite Test**: N/A
.. 3.2.2.1.2
Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 X% packet loss Throughput and Latency Test with
packet modification
Test ID: LTD.Throughput.RFC2544.Profile
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 Throughput and Latency Profile
**Prerequisite Test**: N/A
.. 3.2.2.1.4
Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 System Recovery Time Test
**Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
.. 3.2.2.1.5
Test ID: LTD.Throughput.RFC2544.BackToBackFrames
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2544 Back To Back Frames Test
**Prerequisite Test**: N
.. 3.2.2.1.6
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test
**Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
.. 3.2.2.1.7
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification
**Prerequisite Test**:
.. 3.2.2.1.8
Test ID: LTD.Throughput.RFC6201.ResetTime
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 6201 Reset Time Test
**Prerequisite Test**: N/A
* Transmitting test frame despite the physical link status.
whereas the Timestamp method SHOULD be used if the test tool supports:
+
* Timestamping each frame.
* Monitoring received frame's timestamp.
* Transmitting frames only if the physical link status is up.
.. 3.2.2.1.9
Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Forwarding Rate Test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
.. 3.2.2.1.10
Test ID: LTD.Throughput.RFC2889.ForwardPressure
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Forward Pressure Test
**Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate
.. 3.2.2.1.11
Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Error Frames Filtering Test
**Prerequisite Test**: N/A
.. 3.2.2.1.12
Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Broadcast Frame Forwarding Test
**Prerequisite Test**: N
.. 3.2.2.1.13
Test ID: LTD.Throughput.RFC2544.WorstN-BestN
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Modified RFC 2544 X% packet loss ratio Throughput and Latency Test
**Prerequisite Test**: N/A
Test ID: LTD.Throughput.Overlay.Network.<tech>.RFC2544.PacketLossRatio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: <tech> Overlay Network RFC 2544 X% packet loss ratio Throughput and Latency Test
Test ID: LTD.Throughput.RFC2544.MatchAction.PacketLossRatio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 X% packet loss ratio match action Throughput and Latency Test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
a constant load (fixed length frames at a fixed interval time).
Each test case requires:
+
* selection of a specific match action(s),
* specifying a percentage of total traffic that is elligible
for the match action,
or packet that matches a set of flow classification parameters
(typically frame/packet header fields). A match action may or may
not modify a packet/frame. Match actions include [1]:
+
* output : outputs a packet to a particular port.
* normal: Subjects the packet to traditional L2/L3 processing
(MAC learning).
on which flooding is disabled.
* all: Outputs the packet on all switch physical ports other
than the port on which it was received.
- * local: Outputs the packet on the ``local port,'' which
+ * local: Outputs the packet on the ``local port``, which
corresponds to the network device that has the same name as
the bridge.
* in_port: Outputs the packet on the port from which it was
received.
* Controller: Sends the packet and its metadata to the
- OpenFlow controller as a ``packet in'' message.
+ OpenFlow controller as a ``packet in`` message.
* enqueue: Enqueues the packet on the specified queue
within port.
* drop: discard the packet.
Modifications include [1]:
+
* mod vlan: covered by LTD.Throughput.RFC2544.PacketLossRatioFrameModification
* mod_dl_src: Sets the source Ethernet address.
* mod_dl_dst: Sets the destination Ethernet address.
During this test, the DUT must perform the following operations on
the traffic flow:
+
* Perform packet parsing on the DUT’s ingress port.
* Perform any relevant address look-ups on the DUT’s ingress
ports.
**Metrics Collected:**
The following are the metrics collected for this test:
+
* The RFC 2544 Throughput in Frames Per Second (FPS) and Mbps
of the DUT for each frame size with X% packet loss.
* The average latency of the traffic flow when passing through
.. 3.2.2.2
Packet Latency tests
----------------------------
+--------------------
+
These tests will measure the store and forward latency as well as the packet
delay variation for various packet types through the virtual switch. The
following list is not exhaustive but should indicate the type of tests
.. 3.2.2.2.1
Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Initial Packet Processing Latency
**Prerequisite Test**: N/A
.. 3.2.2.2.2
Test ID: LTD.PacketDelayVariation.RFC3393.Soak
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Packet Delay Variation Soak Test
**Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)
.. 3.2.2.3
Scalability tests
-------------------------
+-----------------
+
The general aim of these tests is to understand the impact of large flow
table size and flow lookups on throughput. The following list is not
exhaustive but should indicate the type of tests that should be required.
.. 3.2.2.3.1
Test ID: LTD.Scalability.Flows.RFC2544.0PacketLoss
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 0% loss Flow Scalability throughput test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
.. 3.2.2.3.2
Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 0% loss Memory Bandwidth Scalability test
**Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
Test ID: LTD.Scalability.VNF.RFC2544.PacketLossRatio
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: VNF Scalability RFC 2544 X% packet loss ratio Throughput and
Latency Test
.. 3.2.2.3.4
Test ID: LTD.Scalability.VNF.RFC2544.PacketLossProfile
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: VNF Scalability RFC 2544 Throughput and Latency Profile
**Prerequisite Test**: N/A
Activation tests
----------------
+
The general aim of these tests is to understand the capacity of the
and speed with which the vswitch can accommodate new flows.
Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Address Caching Capacity Test
**Prerequisite Test**: N/A
.. 3.2.2.4.2
Test ID: LTD.Activation.RFC2889.AddressLearningRate
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC2889 Address Learning Rate Test
**Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity
.. 3.2.2.5
Coupling between control path and datapath Tests
--------------------------------------------------------
+------------------------------------------------
+
The following tests aim to determine how tightly coupled the datapath
and the control path are within a virtual switch. The following list
is not exhaustive but should indicate the type of tests that should be
.. 3.2.2.5.1
Test ID: LTD.CPDPCouplingFlowAddition
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: Control Path and Datapath Coupling
**Prerequisite Test**:
.. 3.2.2.6
CPU and memory consumption
----------------------------------
+--------------------------
+
The following tests will profile a virtual switch's CPU and memory
utilization under various loads and circumstances. The following
list is not exhaustive but should indicate the type of tests that
Test ID: LTD.Stress.RFC2544.0PacketLoss
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
**Title**: RFC 2544 0% Loss CPU OR Memory Stress Test
**Prerequisite Test**:
.. 3.2.2.7
Summary List of Tests
-----------------------------
+---------------------
+
1. Throughput tests
- Test ID: LTD.Throughput.RFC2544.PacketLossRatio
- LTD.Scalability.VNF.RFC2544.PacketLossProfile
- LTD.Scalability.VNF.RFC2544.PacketLossRatio
-4. Acivation tests
+4. Activation tests
- Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
- Test ID: LTD.Activation.RFC2889.AddressLearningRate
Code Review / vswitchperf.git / commitdiff