+======================
+VSPERF Design Document
+======================
+
Intended Audience
=================
-This document is intended to aid those who want to modify the vsperf code. Or to extend it - for example to add support for new traffic generators, deployment scenarios and so on.
+This document is intended to aid those who want to modify the vsperf code. Or
+to extend it - for example to add support for new traffic generators,
+deployment scenarios and so on.
Usage
=====
$ ./vsperf --tests 'tput'
-As above but override default configuration with settings in 'my_settings.py'. This is useful as modifying configuration directly in the configuration files in ``conf/NN_*.py`` shows up as changes under git source control:
+As above but override default configuration with settings in 'my_settings.py'.
+This is useful as modifying configuration directly in the configuration files
+in ``conf/NN_*.py`` shows up as changes under git source control:
.. code-block:: console
$ ./vsperf --conf-file my_settings.py --tests 'tput'
-Override specific test parameters. Useful for shortening the duration of tests for development purposes:
+Override specific test parameters. Useful for shortening the duration of tests
+for development purposes:
.. code-block:: console
This is a typical flow of control for a test.
-.. image:: ../images/vsperf.png
+.. image:: vsperf.png
Configuration
=============
-The conf package contains the configuration files (``*.conf``) for all system components, it also provides a ``settings`` object that exposes all of these settings.
+The conf package contains the configuration files (``*.conf``) for all system
+components, it also provides a ``settings`` object that exposes all of these
+settings.
-Settings are not passed from component to component. Rather they are available globally to all components once they import the conf package.
+Settings are not passed from component to component. Rather they are available
+globally to all components once they import the conf package.
.. code-block:: python
...
log_file = settings.getValue('LOG_FILE_DEFAULT')
-Settings files (``*.conf``) are valid python code so can be set to complex types such as lists and dictionaries as well as scalar types:
+Settings files (``*.conf``) are valid python code so can be set to complex
+types such as lists and dictionaries as well as scalar types:
.. code-block:: python
Configuration Procedure and Precedence
--------------------------------------
-Configuration files follow a strict naming convention that allows them to be processed in a specific order. All the .conf files are named ``NN_name.conf``, where NN is a decimal digit. The files are processed in order from 00_name.conf to 99_name.conf so that if the name setting is given in both a lower and higher numbered conf file then the higher numbered file is the effective setting as it is processed after the setting in the lower numbered file.
+Configuration files follow a strict naming convention that allows them to be
+processed in a specific order. All the .conf files are named ``NN_name.conf``,
+where NN is a decimal number. The files are processed in order from 00_name.conf
+to 99_name.conf so that if the name setting is given in both a lower and higher
+numbered conf file then the higher numbered file is the effective setting as it
+is processed after the setting in the lower numbered file.
-The values in the file specified by ``--conf-file`` takes precedence over all the other configuration files and does not have to follow the naming convention.
+The values in the file specified by ``--conf-file`` takes precedence over all
+the other configuration files and does not have to follow the naming
+convention.
Other Configuration
VM, vSwitch, Traffic Generator Independence
===========================================
-VSPERF supports different vSwithes, Traffic Generators and VNFs by using standard object-oriented polymorphism:
+VSPERF supports different vSwithes, Traffic Generators and VNFs by using
+standard object-oriented polymorphism:
* Support for vSwitches is implemented by a class inheriting from IVSwitch.
- * Support for Traffic Generators is implemented by a class inheriting from ITrafficGenerator.
+ * Support for Traffic Generators is implemented by a class inheriting from
+ ITrafficGenerator.
* Support for VNF is implemented by a class inheriting from IVNF.
-By dealing only with the abstract interfaces the core framework can support many implementations of different vSwitches, Traffic Generators and VNFs.
+By dealing only with the abstract interfaces the core framework can support
+many implementations of different vSwitches, Traffic Generators and VNFs.
IVSwitch
--------
Controllers
-----------
-Controllers are used in conjunction with abstract interfaces as way of decoupling the control of vSwtiches, VNFs and TrafficGenerators from other components.
+Controllers are used in conjunction with abstract interfaces as way of
+decoupling the control of vSwtiches, VNFs and TrafficGenerators from other
+components.
-The controlled classes provide basic primitive operations. The Controllers sequence and co-ordinate these primitive operation in to useful actions. For instance the vswitch_controller_PVP can be used to bring any vSwitch (that implements the primitives defined in IVSwitch) into the configuration required by the Phy-to-Phy Deployment Scenario.
+The controlled classes provide basic primitive operations. The Controllers
+sequence and co-ordinate these primitive operation in to useful actions. For
+instance the vswitch_controller_PVP can be used to bring any vSwitch (that
+implements the primitives defined in IVSwitch) into the configuration required
+by the Phy-to-Phy Deployment Scenario.
-In order to support a new vSwitch only a new implementation of IVSwitch needs be created for the new vSwitch to be capable of fulfilling all the Deployment Scenarios provided for by existing or future vSwitch Controllers.
+In order to support a new vSwitch only a new implementation of IVSwitch needs
+be created for the new vSwitch to be capable of fulfilling all the Deployment
+Scenarios provided for by existing or future vSwitch Controllers.
-Similarly if a new Deployment Scenario is required it only needs to be written once as a new vSwitch Controller and it will immediately be capable of controlling all existing and future vSwitches in to that Deployment Scenario.
+Similarly if a new Deployment Scenario is required it only needs to be written
+once as a new vSwitch Controller and it will immediately be capable of
+controlling all existing and future vSwitches in to that Deployment Scenario.
-Similarly the Traffic Controllers can be used to co-ordinate basic operations provided by implementers of ITrafficGenerator to provide useful tests. Though traffic generators generally already implement full test cases i.e. they both generate suitable traffic and analyse returned traffic in order to implement a test which has typically been predefined in an RFC document. However the Traffic Controller class allows for the possibility of further enhancement - such as iterating over tests for various packet sizes or creating new tests.
+Similarly the Traffic Controllers can be used to co-ordinate basic operations
+provided by implementers of ITrafficGenerator to provide useful tests. Though
+traffic generators generally already implement full test cases i.e. they both
+generate suitable traffic and analyse returned traffic in order to implement a
+test which has typically been predefined in an RFC document. However the
+Traffic Controller class allows for the possibility of further enhancement -
+such as iterating over tests for various packet sizes or creating new tests.
Traffic Controller's Role
-------------------------
-.. image:: ../images/traffic_controller.png
+.. image:: traffic_controller.png
Loader & Component Factory
--------------------------
-The working of the Loader package (which is responsible for *finding* arbitrary classes based on configuration data) and the Component Factory which is responsible for *choosing* the correct class for a particular situation - e.g. Deployment Scenario can be seen in this diagram.
+The working of the Loader package (which is responsible for *finding* arbitrary
+classes based on configuration data) and the Component Factory which is
+responsible for *choosing* the correct class for a particular situation - e.g.
+Deployment Scenario can be seen in this diagram.
-.. image:: ../images/factory_and_loader.png
+.. image:: factory_and_loader.png
Routing Tables
==============
-Vsperf uses a standard set of routing tables in order to allow tests to easily mix and match Deployment Scenarios (PVP, P2P topology), Tuple Matching and Frame Modification requirements.
-
-::
-
- +--------------+
- | |
- | Table 0 | table#0 - Match table. Flows designed to force 5 & 10 tuple matches go here.
- | |
- +--------------+
- |
- |
- v
- +--------------+ table#1 - Routing table. Flows to route packets between ports goes here.
- | | The chosen port is communicated to subsequent tables by setting the
- | Table 1 | metadata value to the egress port number. Generally this table
- | | is set-up by by the vSwitchController.
- +--------------+
- |
- |
- v
- +--------------+ table#2 - Frame modification table. Frame modification flow rules are
- | | isolated in this table so that they can be turned on or off
- | Table 2 | without affecting the routing or tuple-matching flow rules.
- | | This allows the frame modification and tuple matching required by the
- +--------------+ tests in the VSWITCH PERFORMANCE FOR TELCO NFV test specification
- | to be independent of the Deployment Scenario set up by the vSwitchController.
- |
- v
- +--------------+
- | |
- | Table 3 | table#3 - Egress table. Egress packets on the ports setup in Table 1.
- | |
- +--------------+
+Vsperf uses a standard set of routing tables in order to allow tests to easily
+mix and match Deployment Scenarios (PVP, P2P topology), Tuple Matching and
+Frame Modification requirements.
+
+.. code-block:: console
+
+ +--------------+
+ | |
+ | Table 0 | table#0 - Match table. Flows designed to force 5 & 10
+ | | tuple matches go here.
+ | |
+ +--------------+
+ |
+ |
+ v
+ +--------------+ table#1 - Routing table. Flows to route packets between
+ | | ports goes here.
+ | Table 1 | The chosen port is communicated to subsequent tables by
+ | | setting the metadata value to the egress port number.
+ | | Generally this table is set-up by by the
+ +--------------+ vSwitchController.
+ |
+ |
+ v
+ +--------------+ table#2 - Frame modification table. Frame modification
+ | | flow rules are isolated in this table so that they can
+ | Table 2 | be turned on or off without affecting the routing or
+ | | tuple-matching flow rules. This allows the frame
+ | | modification and tuple matching required by the tests
+ | | in the VSWITCH PERFORMANCE FOR TELCO NFV test
+ +--------------+ specification to be independent of the Deployment
+ | Scenario set up by the vSwitchController.
+ |
+ v
+ +--------------+
+ | |
+ | Table 3 | table#3 - Egress table. Egress packets on the ports
+ | | setup in Table 1.
+ +--------------+
+
+
-CHARACTERIZE VSWITCH PERFORMANCE FOR TELCO NFV USE CASES LEVEL TEST DESIGN
-==========================================================================
-.. contents:: Table of Contents
+.. 3.1
-1. Introduction
+===============
+Introduction
===============
The objective of the OPNFV project titled
This document is currently in draft form.
-1.1. Document identifier
-------------------------
+.. 3.1.1
+
+Document identifier
+=========================
The document id will be used to uniquely
identify versions of the LTD. The format for the document id will be:
for this version of the LTD is:
OPNFV\_vswitchperf\_LTD\_ver\_1.6\_Jan\_15\_DRAFT.
-1.2. Scope
-----------
+.. 3.1.2
+
+Scope
+==========
The main purpose of this project is to specify a suite of
performance tests in order to objectively measure the current packet
Test Framework - if a vSwitch becomes a standard component of an OPNFV
release.
-1.3. References
----------------
+.. 3.1.3
+
+References
+===============
* `RFC 1242 Benchmarking Terminology for Network Interconnection
Devices <http://www.ietf.org/rfc/rfc1242.txt>`__
* `RFC 6201 Device Reset
Characterization <http://tools.ietf.org/html/rfc6201>`__
-2. Details of the Level Test Design
+.. 3.2
+
+===================================
+Details of the Level Test Design
===================================
-This section describes the features to be tested (`cf. 2.1
-<#FeaturesToBeTested>`__), the test approach (`cf. 2.2 <#Approach>`__);
-it also identifies the sets of test cases or scenarios (`cf. 2.3
-<#TestIdentification>`__) along with the pass/fail criteria (`cf. 2.4
-<#PassFail>`__) and the test deliverables (`cf. 2.5 <#TestDeliverables>`__).
+This section describes the features to be tested (
+:ref:_FeaturesToBeTested), the test approach (:ref:_Approach);
+it also identifies the sets of test cases or scenarios (
+:ref:_TestIdentification) along with the pass/fail criteria and
+the test deliverables.
-2.1. Features to be tested
---------------------------
+.. 3.2.1
+
+.. _FeaturesToBeTested:
+
+Features to be tested
+==========================
Characterizing virtual switches (i.e. Device Under Test (DUT) in this document)
includes measuring the following performance metrics:
- Includes headroom of VM workload processing cores (i.e. available
for applications).
+.. 3.2.2
+
+.. _Approach:
-2.2. Approach
+Approach
==============
In order to determine the packet transfer characteristics of a virtual
switch, the tests will be broken down into the following categories:
-2.2.1 Test Categories
+.. 3.2.2.1
+
+Test Categories
----------------------
- **Throughput Tests** to measure the maximum forwarding rate (in
frames per second or fps) and bit rate (in Mbps) for a constant load
the combined results would be insightful, for example Packet/Frame Delay
and Scalability.
-2.2.2 Deployment Scenarios
+.. 3.2.2.2
+
+Deployment Scenarios
--------------------------
The following represents possible deployments which can help to
determine the performance of both the virtual switch and the datapath
into the VNF:
+.. 3.2.2.2.1
+
Physical port → vSwitch → physical port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
+.. code-block:: console
_
+--------------------------------------------------+ |
| |
+--------------------------------------------------+
+.. 3.2.2.2.2
Physical port → vSwitch → VNF → vSwitch → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. code-block:: console
_
+---------------------------------------------------+ |
| |
+--------------------------------------------------+
+.. 3.2.2.2.3
Physical port → vSwitch → VNF → vSwitch → VNF → vSwitch → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
+.. code-block:: console
_
+----------------------+ +----------------------+ |
| |
+--------------------------------------------------+
+.. 3.2.2.2.4
+
Physical port → VNF → vSwitch → VNF → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
+.. code-block:: console
_
+----------------------+ +----------------------+ |
| |
+--------------------------------------------------+
+.. 3.2.2.2.5
+
Physical port → vSwitch → VNF
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
+.. code-block:: console
_
+---------------------------------------------------+ |
| |
+--------------------------------------------------+
+.. 3.2.2.2.6
+
VNF → vSwitch → physical port
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
+.. code-block:: console
_
+---------------------------------------------------+ |
| |
+--------------------------------------------------+
+.. 3.2.2.2.7
+
VNF → vSwitch → VNF → vSwitch
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- .. code-block:: console
+.. code-block:: console
_
+-------------------------+ +-------------------------+ |
| vswitch | |
+-------------------------------------------------------+ _|
-HOST 1(Physical port → virtual switch → VNF → virtual switch →
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Physical port) → HOST 2(Physical port → virtual switch → VNF →
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-virtual switch → Physical port)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. 3.2.2.2.8
- .. code-block:: console
+HOST 1(Physical port → virtual switch → VNF → virtual switch → Physical port)
+→ HOST 2(Physical port → virtual switch → VNF → virtual switch → Physical port)
+
+HOST 1 (PVP) → HOST 2 (PVP)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
_
+----------------------+ +----------------------+ |
connected to the test device. This port is used to listen for flooded
frames.
-2.2.3 General Methodology:
+.. 3.2.2.3
+
+General Methodology:
--------------------------
To establish the baseline performance of the virtual switch, tests would
initially be run with a simple workload in the VNF (the recommended
underlying characteristics and behaviour can be measured and validated.
Suitable real Telco workload VNFs are yet to be identified.
-2.2.3.1 Default Test Parameters
+.. 3.2.2.3.1
+
+Default Test Parameters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The following list identifies the default parameters for suite of
through the virtual switch, i.e. flows are installed and have an appropriate
time out that doesn't expire before packet transmission starts.
-2.2.3.2 Flow Classification
+.. 3.2.2.3.2
+
+Flow Classification
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Virtual switches classify packets into flows by processing and matching
additional latency, subsequent packets of the same flow are not subject
to this latency if the flow is already installed on the vSwitch.
-2.2.3.3 Test Priority
+.. 3.2.2.3.3
+
+Test Priority
~~~~~~~~~~~~~~~~~~~~~
Tests will be assigned a priority in order to determine which tests
May be implemented after the release. - Low: May or may not be
implemented at all.
-2.2.3.4 SUT Setup
-~~~~~~~~~~~~~~~~~
+.. 3.2.2.3.4
+
+SUT Setup
+~~~~~~~~~~~~~~~~~~
The SUT should be configured to its "default" state. The
SUT's configuration or set-up must not change between tests in any way
other than what is required to do the test. All supported protocols must
be configured and enabled for each test set up.
-2.2.3.4.1 Port Configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. 3.2.2.3.5
+
+Port Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~
The DUT should be configured with n ports where
n is a multiple of 2. Half of the ports on the DUT should be used as
stream directed at port 1 should also output a packet to port 2 followed
by a packet to port 3.
-2.2.3.4.2 Frame Formats
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. 3.2.2.3.6
+
+Frame Formats
+~~~~~~~~~~~~~~~~~~~~~
+
+**Frame formats Layer 2 (data link layer) protocols**
-Frame formats Layer 2 (data link layer) protocols
-++++++++++++++++++++++++++++++++++++++++++++++++++
- Ethernet II
- .. code-block:: console
+.. code-block:: console
- +---------------------+--------------------+-----------+
- | Ethernet Header | Payload | Check Sum |
- +---------------------+--------------------+-----------+
- |_____________________|____________________|___________|
- 14 Bytes 46 - 1500 Bytes 4 Bytes
+ +---------------------------+-----------+
+ | Ethernet Header | Payload | Check Sum |
+ +-----------------+---------+-----------+
+ |_________________|_________|___________|
+ 14 Bytes 46 - 1500 4 Bytes
+ Bytes
-Layer 3 (network layer) protocols
-++++++++++++++++++++++++++++++++++
+
+**Layer 3 (network layer) protocols**
- IPv4
- .. code-block:: console
+.. code-block:: console
- +---------------------+--------------------+--------------------+-----------+
- | Ethernet Header | IP Header | Payload | Check Sum |
- +---------------------+--------------------+--------------------+-----------+
- |_____________________|____________________|____________________|___________|
- 14 Bytes 20 bytes 26 - 1480 Bytes 4 Bytes
+ +-----------------+-----------+---------+-----------+
+ | Ethernet Header | IP Header | Payload | Checksum |
+ +-----------------+-----------+---------+-----------+
+ |_________________|___________|_________|___________|
+ 14 Bytes 20 bytes 26 - 1480 4 Bytes
+ Bytes
- IPv6
- .. code-block:: console
+.. code-block:: console
+
+ +-----------------+-----------+---------+-----------+
+ | Ethernet Header | IP Header | Payload | Checksum |
+ +-----------------+-----------+---------+-----------+
+ |_________________|___________|_________|___________|
+ 14 Bytes 40 bytes 26 - 1460 4 Bytes
+ Bytes
- +---------------------+--------------------+--------------------+-----------+
- | Ethernet Header | IP Header | Payload | Check Sum |
- +---------------------+--------------------+--------------------+-----------+
- |_____________________|____________________|____________________|___________|
- 14 Bytes 40 bytes 26 - 1460 Bytes 4 Bytes
+**Layer 4 (transport layer) protocols**
-Layer 4 (transport layer) protocols
-++++++++++++++++++++++++++++++++++++
- TCP
- UDP
- SCTP
- .. code-block:: console
+.. code-block:: console
+
+ +-----------------+-----------+-----------------+---------+-----------+
+ | Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum |
+ +-----------------+-----------+-----------------+---------+-----------+
+ |_________________|___________|_________________|_________|___________|
+ 14 Bytes 40 bytes 20 Bytes 6 - 1460 4 Bytes
+ Bytes
+
- +---------------------+--------------------+-----------------+--------------------+-----------+
- | Ethernet Header | IP Header | Layer 4 Header | Payload | Check Sum |
- +---------------------+--------------------+-----------------+--------------------+-----------+
- |_____________________|____________________|_________________|____________________|___________|
- 14 Bytes 40 bytes 20 Bytes 6 - 1460 Bytes 4 Bytes
+**Layer 5 (application layer) protocols**
-Layer 5 (application layer) protocols
-+++++++++++++++++++++++++++++++++++++
- RTP
- GTP
- .. code-block:: console
+.. code-block:: console
- +---------------------+--------------------+-----------------+--------------------+-----------+
- | Ethernet Header | IP Header | Layer 4 Header | Payload | Check Sum |
- +---------------------+--------------------+-----------------+--------------------+-----------+
- |_____________________|____________________|_________________|____________________|___________|
- 14 Bytes 20 bytes 20 Bytes Min 6 Bytes 4 Bytes
+ +-----------------+-----------+-----------------+---------+-----------+
+ | Ethernet Header | IP Header | Layer 4 Header | Payload | Checksum |
+ +-----------------+-----------+-----------------+---------+-----------+
+ |_________________|___________|_________________|_________|___________|
+ 14 Bytes 20 bytes 20 Bytes >= 6 Bytes 4 Bytes
+.. 3.2.2.3.7
-2.2.3.4.3 Packet Throughput
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Packet Throughput
+~~~~~~~~~~~~~~~~~~~~~~~~~
There is a difference between an Ethernet frame,
an IP packet, and a UDP datagram. In the seven-layer OSI model of
computer networking, packet refers to a data unit at layer 3 (network
= 10,000,000,000 / 672
= 14,880,952.38 frame per second (fps)
-2.2.3.4.4 System isolation and validation
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+.. 3.2.2.3.8
+
+System isolation and validation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A key consideration when conducting any sort of benchmark is trying to
ensure the consistency and repeatability of test results between runs.
to validate the platform and the VNF before conducting any vSwitch
benchmarking tests.
-System Isolation
-++++++++++++++++
+**System Isolation:**
+
When conducting a benchmarking test on any SUT, it is essential to limit
(and if reasonable, eliminate) any noise that may interfere with the
accuracy of the metrics collected by the test. This noise may be
virtualization optimization technologies should be enabled, and
hyperthreading should also be enabled.
-System Validation
-+++++++++++++++++
+**System Validation:**
+
System validation is broken down into two sub-categories: Platform
validation and VNF validation. The validation test itself involves
verifying the forwarding capability and stability for the sub-system
+--------------------------------------------------+
-Methodology to benchmark Platform/VNF forwarding capability
-++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+**Methodology to benchmark Platform/VNF forwarding capability**
+
The recommended methodology for the platform/VNF validation and
benchmark is: - Run `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__
(# of vCPUs, vNICs, Memory, affinitization…) is how it should be
configured for every test that uses a VNF after this.
-2.2.4 RFCs for testing virtual switch performance
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. 3.2.2.4
+
+RFCs for testing virtual switch performance
+--------------------------------------------------
The starting point for defining the suite of tests for benchmarking the
performance of a virtual switch is to take existing RFCs and standards
switches. This section outlines the RFCs that are used by this
specification.
+.. 3.2.2.4.1
+
RFC 1242 Benchmarking Terminology for Network Interconnection
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Devices RFC 1242 defines the terminology that is used in describing
performance benchmarking tests and their results. Definitions and
discussions covered include: Back-to-back, bridge, bridge/router,
constant load, data link frame size, frame loss rate, inter frame gap,
latency, and many more.
+.. 3.2.2.4.2
+
RFC 2544 Benchmarking Methodology for Network Interconnect Devices
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 2544 outlines a benchmarking methodology for network Interconnect
Devices. The methodology results in performance metrics such as latency,
frame loss percentage, and maximum data throughput.
condition.
6. Reset to characterize speed of recovery from device or software
- reset. This type of test has been updated by `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ as such,
+ reset. This type of test has been updated by `RFC6201
+ <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ as such,
the methodology defined by this specification will be that of RFC 6201.
Although not included in the defined RFC 2544 standard, another crucial
definition set out by this specification comes from
`RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__.
+.. 3.2.2.4.3
+
RFC 2285 Benchmarking Terminology for LAN Switching Devices
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 2285 defines the terminology that is used to describe the
terminology for benchmarking a LAN switching device. It extends RFC
1242 and defines: DUTs, SUTs, Traffic orientation and distribution,
bursts, loads, forwarding rates, etc.
+.. 3.2.2.4.4
+
RFC 2889 Benchmarking Methodology for LAN Switching
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 2889 outlines a benchmarking methodology for LAN switching, it
extends RFC 2544. The outlined methodology gathers performance
metrics for forwarding, congestion control, latency, address handling
and finally filtering.
+.. 3.2.2.4.5
+
RFC 3918 Methodology for IP Multicast Benchmarking
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 3918 outlines a methodology for IP Multicast benchmarking.
+.. 3.2.2.4.6
+
RFC 4737 Packet Reordering Metrics
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 4737 describes metrics for identifying and counting re-ordered
packets within a stream, and metrics to measure the extent each
packet has been re-ordered.
+.. 3.2.2.4.7
+
RFC 5481 Packet Delay Variation Applicability Statement
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 5481 defined two common, but different forms of delay variation
metrics, and compares the metrics over a range of networking
circumstances and tasks. The most suitable form for vSwitch
benchmarking is the "PDV" form.
+.. 3.2.2.4.8
+
RFC 6201 Device Reset Characterization
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RFC 6201 extends the methodology for characterizing the speed of
recovery of the DUT from device or software reset described in RFC
2544.
-2.2.5 Details of the Test Report
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. 3.2.2.5
+
+Details of the Test Report
+---------------------------------
There are a number of parameters related to the system, DUT and tests
that can affect the repeatability of a test results and should be
**Note**: Tests that require additional parameters to be recorded will
explicitly specify this.
-2.3. Test identification
-------------------------
-2.3.1 Throughput tests
-~~~~~~~~~~~~~~~~~~~~~~
+.. _TestIdentification:
+
+.. 3.2.3
+
+Test identification
+=========================
+
+.. 3.2.3.1
+
+Throughput tests
+----------------------
The following tests aim to determine the maximum forwarding rate that
can be achieved with a virtual switch. The list is not exhaustive but
should indicate the type of tests that should be required. It is
expected that more will be added.
+.. 3.2.3.1.1
+
Test ID: LTD.Throughput.RFC2544.PacketLossRatio
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2544 X% packet loss ratio Throughput and Latency Test
**Prerequisite Test**: N/A
**Expected Result**: At the end of each trial, the presence or absence
of loss determines the modification of offered load for the next trial,
converging on a maximum rate, or
- `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X% loss.
+ `RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ Throughput with X%
+ loss.
The Throughput load is re-used in related
`RFC2544 <https://www.rfc-editor.org/rfc/rfc2544.txt>`__ tests and other
tests.
- CPU and memory utilization may also be collected as part of this
test, to determine the vSwitch's performance footprint on the system.
+.. 3.2.3.1.2
+
Test ID: LTD.Throughput.RFC2544.PacketLossRatioFrameModification
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2544 X% packet loss Throughput and Latency Test with
packet modification
- CPU and memory utilization may also be collected as part of this
test, to determine the vSwitch's performance footprint on the system.
+.. 3.2.3.1.3
+
Test ID: LTD.Throughput.RFC2544.Profile
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2544 Throughput and Latency Profile
**Prerequisite Test**: N/A
when the offered load is above Maximum Throughput MUST be recorded
and reported with the results.
+.. 3.2.3.1.4
+
Test ID: LTD.Throughput.RFC2544.SystemRecoveryTime
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2544 System Recovery Time Test
**Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
- Physical → virtual switch → physical.
+.. 3.2.3.1.5
+
Test ID: LTD.Throughput.RFC2544.BackToBackFrames
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC2544 Back To Back Frames Test
**Prerequisite Test**: N
- Physical → virtual switch → physical.
+.. 3.2.3.1.6
+
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoak
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2889 X% packet loss Max Forwarding Rate Soak Test
**Prerequisite Test** LTD.Throughput.RFC2544.PacketLossRatio
PDV form of delay variation on the traffic flow,
using the 99th percentile.
+.. 3.2.3.1.7
+
Test ID: LTD.Throughput.RFC2889.MaxForwardingRateSoakFrameModification
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2889 Max Forwarding Rate Soak Test with Frame Modification
- **Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss)
+ **Prerequisite Test**:
+ LTD.Throughput.RFC2544.PacketLossRatioFrameModification (0% Packet Loss)
**Priority**:
- CPU and memory utilization may also be collected as part of this
test, to determine the vSwitch's performance footprint on the system.
- - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__ PDV form of delay variation on the traffic flow,
- using the 99th percentile.
+ - The `RFC5481 <https://www.rfc-editor.org/rfc/rfc5481.txt>`__
+ PDV form of delay variation on the traffic flow, using the 99th
+ percentile.
+
+.. 3.2.3.1.8
Test ID: LTD.Throughput.RFC6201.ResetTime
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 6201 Reset Time Test
**Prerequisite Test**: N/A
flows are passing through the DUT, the DUT should be reset and the Reset
time measured. The Reset time is the total time that a device is
determined to be out of operation and includes the time to perform the
- reset and the time to recover from it (cf. `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__).
+ reset and the time to recover from it (cf. `RFC6201
+ <https://www.rfc-editor.org/rfc/rfc6201.txt>`__).
+
+ `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ defines two methods
+ to measure the Reset time:
- `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ defines two methods to measure the Reset time:
- Frame-Loss Method: which requires the monitoring of the number of
lost frames and calculates the Reset time based on the number of
frames lost and the offered rate according to the following
is received after the reset. The Reset time is the difference between
these two timestamps.
- According to `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ the choice of method depends on the test
- tool's capability; the Frame-Loss method SHOULD be used if the test tool
- supports: - Counting the number of lost frames per stream. -
- Transmitting test frame despite the physical link status.
+ According to `RFC6201 <https://www.rfc-editor.org/rfc/rfc6201.txt>`__ the
+ choice of method depends on the test tool's capability; the Frame-Loss
+ method SHOULD be used if the test tool supports:
+
+ * Counting the number of lost frames per stream.
+ * Transmitting test frame despite the physical link status.
- whereas the Timestamp method SHOULD be used if the test tool supports: -
- Timestamping each frame. - Monitoring received frame's timestamp. -
- Transmitting frames only if the physical link status is up.
+ whereas the Timestamp method SHOULD be used if the test tool supports:
+ * Timestamping each frame.
+ * Monitoring received frame's timestamp.
+ * Transmitting frames only if the physical link status is up.
**Expected Result**:
**Metrics collected**
- The following are the metrics collected for this test: - Average Reset
- Time over the number of trials performed.
+ The following are the metrics collected for this test:
+
+ * Average Reset Time over the number of trials performed.
- Results of this test should include the following information: - The
- reset method used. - Throughput in Fps and Mbps. - Average Frame Loss
- over the number of trials performed. - Average Reset Time in
- milliseconds over the number of trials performed. - Number of trials
- performed. - Protocol: IPv4, IPv6, MPLS, etc. - Frame Size in Octets -
- Port Media: Ethernet, Gigabit Ethernet (GbE), etc. - Port Speed: 10
- Gbps, 40 Gbps etc. - Interface Encapsulation: Ethernet, Ethernet VLAN,
- etc.
+ Results of this test should include the following information:
+
+ * The reset method used.
+ * Throughput in Fps and Mbps.
+ * Average Frame Loss over the number of trials performed.
+ * Average Reset Time in milliseconds over the number of trials performed.
+ * Number of trials performed.
+ * Protocol: IPv4, IPv6, MPLS, etc.
+ * Frame Size in Octets
+ * Port Media: Ethernet, Gigabit Ethernet (GbE), etc.
+ * Port Speed: 10 Gbps, 40 Gbps etc.
+ * Interface Encapsulation: Ethernet, Ethernet VLAN, etc.
**Deployment scenario**:
- - Physical → virtual switch → physical.
+ * Physical → virtual switch → physical.
+
+.. 3.2.3.1.9
Test ID: LTD.Throughput.RFC2889.MaxForwardingRate
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC2889 Forwarding Rate Test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio
ends.
**Expected Result**: According to
- `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ The Max Forwarding Rate
- is the highest forwarding rate of a DUT taken from an iterative set of
- forwarding rate measurements. The iterative set of forwarding rate
- measurements are made by setting the intended load transmitted from an
- external source and measuring the offered load (i.e what the DUT is
- capable of forwarding). If the Throughput == the Maximum Offered Load,
- it follows that Max Forwarding Rate is equal to the Maximum Offered
- Load.
+ `RFC2889 <https://www.rfc-editor.org/rfc/rfc2289.txt>`__ The Max Forwarding
+ Rate is the highest forwarding rate of a DUT taken from an iterative set of
+ forwarding rate measurements. The iterative set of forwarding rate measurements
+ are made by setting the intended load transmitted from an external source and
+ measuring the offered load (i.e what the DUT is capable of forwarding). If the
+ Throughput == the Maximum Offered Load, it follows that Max Forwarding Rate is
+ equal to the Maximum Offered Load.
**Metrics Collected**:
benchmarks, and scenarios with both 2 and 4 ports should be tested.
In any case, the number of ports used must be reported.
+.. 3.2.3.1.10
Test ID: LTD.Throughput.RFC2889.ForwardPressure
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC2889 Forward Pressure Test
**Prerequisite Test**: LTD.Throughput.RFC2889.MaxForwardingRate
- Physical → virtual switch → physical.
+.. 3.2.3.1.11
Test ID: LTD.Throughput.RFC2889.ErrorFramesFiltering
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC2889 Error Frames Filtering Test
**Prerequisite Test**: N/A
- Physical → virtual switch → physical.
+.. 3.2.3.1.12
+
Test ID: LTD.Throughput.RFC2889.BroadcastFrameForwarding
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC2889 Broadcast Frame Forwarding Test
**Prerequisite Test**: N
**Deployment scenario**:
- Physical → virtual switch 3x physical. In the Broadcast rate testing,
- four test ports are required. One of the ports is connected to the test
- device, so it can send broadcast frames and listen for miss-routed frames.
+ four test ports are required. One of the ports is connected to the test
+ device, so it can send broadcast frames and listen for miss-routed frames.
-2.3.2 Packet Latency tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. 3.2.3.2
+
+Packet Latency tests
+---------------------------
These tests will measure the store and forward latency as well as the packet
delay variation for various packet types through the virtual switch. The
following list is not exhaustive but should indicate the type of tests
that should be required. It is expected that more will be added.
+.. 3.2.3.2.1
+
Test ID: LTD.PacketLatency.InitialPacketProcessingLatency
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: Initial Packet Processing Latency
**Prerequisite Test**: N/A
- Physical → Virtual Switch → Physical.
+.. 3.2.3.2.2
+
Test ID: LTD.PacketDelayVariation.RFC3393.Soak
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: Packet Delay Variation Soak Test
**Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio (0% Packet Loss)
- CPU and memory utilization may also be collected as part of this
test, to determine the vSwitch's performance footprint on the system.
-2.3.3 Scalability tests
-~~~~~~~~~~~~~~~~~~~~~~~~
+.. 3.2.3.3
+
+Scalability tests
+------------------------
The general aim of these tests is to understand the impact of large flow
table size and flow lookups on throughput. The following list is not
exhaustive but should indicate the type of tests that should be required.
It is expected that more will be added.
+.. 3.2.3.3.1
+
Test ID: LTD.Scalability.RFC2544.0PacketLoss
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2544 0% loss Scalability throughput test
**Prerequisite Test**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
specified number of flows and the specified frame size, with zero
packet loss.
+.. 3.2.3.3.2
+
Test ID: LTD.MemoryBandwidth.RFC2544.0PacketLoss.Scalability
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2544 0% loss Memory Bandwidth Scalability test
**Prerequisite Tests**: LTD.Throughput.RFC2544.PacketLossRatio, IF the
The following are the metrics collected for this test:
- - The DUT's 0% packet loss throughput in the presence of cache sharing and memory bandwidth between processes.
+ - The DUT's 0% packet loss throughput in the presence of cache sharing and
+ memory bandwidth between processes.
+
+.. 3.2.3.4
-2.3.4 Activation tests
-~~~~~~~~~~~~~~~~~~~~~~~~
+Activation tests
+-----------------------
The general aim of these tests is to understand the capacity of the
-and speed with which the vswitch can accomodate new flows.
+and speed with which the vswitch can accommodate new flows.
+
+.. 3.2.3.4.1
Test ID: LTD.Activation.RFC2889.AddressCachingCapacity
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC2889 Address Caching Capacity Test
**Prerequisite Test**: N/A
- Physical → virtual switch → 2 x physical (one receiving, one listening).
+.. 3.2.3.4.2
+
Test ID: LTD.Activation.RFC2889.AddressLearningRate
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC2889 Address Learning Rate Test
**Prerequisite Test**: LTD.Memory.RFC2889.AddressCachingCapacity
- Physical → virtual switch → 2 x physical (one receiving, one listening).
+.. 3.2.3.5
-2.3.5 Coupling between control path and datapath Tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Coupling between control path and datapath Tests
+-------------------------------------------------------
The following tests aim to determine how tightly coupled the datapath
and the control path are within a virtual switch. The following list
is not exhaustive but should indicate the type of tests that should be
required. It is expected that more will be added.
+.. 3.2.3.5.1
+
Test ID: LTD.CPDPCouplingFlowAddition
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: Control Path and Datapath Coupling
**Prerequisite Test**:
- Physical → virtual switch → physical.
-2.3.6 CPU and memory consumption
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. 3.2.3.6
+
+CPU and memory consumption
+---------------------------------
The following tests will profile a virtual switch's CPU and memory
utilization under various loads and circumstances. The following
list is not exhaustive but should indicate the type of tests that
should be required. It is expected that more will be added.
+.. 3.2.3.6.1
+
Test ID: LTD.CPU.RFC2544.0PacketLoss
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
**Title**: RFC 2544 0% Loss Compute Test
**Prerequisite Test**:
- The configuration of the stress tool (for example the command line
parameters used to start it.)
-2.3.7 Summary List of Tests
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. 3.2.3.7
+
+Summary List of Tests
+----------------------------
1. Throughput tests
- Test ID: LTD.Throughput.RFC2544.PacketLossRatio
Code Review / vswitchperf.git / commitdiff
