+
+
+
+
+Network Working Group M. Tahhan
+Internet-Draft B. O'Mahony
+Intended status: Informational Intel
+Expires: April 16, 2016 A. Morton
+ AT&T Labs
+ October 14, 2015
+
+
+ Benchmarking Virtual Switches in OPNFV
+ draft-vsperf-bmwg-vswitch-opnfv-01
+
+Abstract
+
+ This memo describes the progress of the Open Platform for NFV (OPNFV)
+ project on virtual switch performance "VSWITCHPERF". This project
+ intends to build on the current and completed work of the
+ Benchmarking Methodology Working Group in IETF, by referencing
+ existing literature. The Benchmarking Methodology Working Group has
+ traditionally conducted laboratory characterization of dedicated
+ physical implementations of internetworking functions. Therefore,
+ this memo begins to describe the additional considerations when
+ virtual switches are implemented in general-purpose hardware. The
+ expanded tests and benchmarks are also influenced by the OPNFV
+ mission to support virtualization of the "telco" infrastructure.
+
+Requirements Language
+
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+ document are to be interpreted as described in RFC 2119 [RFC2119].
+
+Status of This Memo
+
+ This Internet-Draft is submitted in full conformance with the
+ provisions of BCP 78 and BCP 79.
+
+ Internet-Drafts are working documents of the Internet Engineering
+ Task Force (IETF). Note that other groups may also distribute
+ working documents as Internet-Drafts. The list of current Internet-
+ Drafts is at http://datatracker.ietf.org/drafts/current/.
+
+ Internet-Drafts are draft documents valid for a maximum of six months
+ and may be updated, replaced, or obsoleted by other documents at any
+ time. It is inappropriate to use Internet-Drafts as reference
+ material or to cite them other than as "work in progress."
+
+ This Internet-Draft will expire on April 16, 2016.
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 1]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+Copyright Notice
+
+ Copyright (c) 2015 IETF Trust and the persons identified as the
+ document authors. All rights reserved.
+
+ This document is subject to BCP 78 and the IETF Trust's Legal
+ Provisions Relating to IETF Documents
+ (http://trustee.ietf.org/license-info) in effect on the date of
+ publication of this document. Please review these documents
+ carefully, as they describe your rights and restrictions with respect
+ to this document. Code Components extracted from this document must
+ include Simplified BSD License text as described in Section 4.e of
+ the Trust Legal Provisions and are provided without warranty as
+ described in the Simplified BSD License.
+
+Table of Contents
+
+ 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
+ 2. Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
+ 3. Benchmarking Considerations . . . . . . . . . . . . . . . . . 4
+ 3.1. Comparison with Physical Network Functions . . . . . . . 4
+ 3.2. Continued Emphasis on Black-Box Benchmarks . . . . . . . 4
+ 3.3. New Configuration Parameters . . . . . . . . . . . . . . 4
+ 3.4. Flow classification . . . . . . . . . . . . . . . . . . . 6
+ 3.5. Benchmarks using Baselines with Resource Isolation . . . 7
+ 4. VSWITCHPERF Specification Summary . . . . . . . . . . . . . . 8
+ 5. 3x3 Matrix Coverage . . . . . . . . . . . . . . . . . . . . . 16
+ 5.1. Speed of Activation . . . . . . . . . . . . . . . . . . . 17
+ 5.2. Accuracy of Activation section . . . . . . . . . . . . . 17
+ 5.3. Reliability of Activation . . . . . . . . . . . . . . . . 17
+ 5.4. Scale of Activation . . . . . . . . . . . . . . . . . . . 17
+ 5.5. Speed of Operation . . . . . . . . . . . . . . . . . . . 17
+ 5.6. Accuracy of Operation . . . . . . . . . . . . . . . . . . 17
+ 5.7. Reliability of Operation . . . . . . . . . . . . . . . . 17
+ 5.8. Scalability of Operation . . . . . . . . . . . . . . . . 18
+ 5.9. Summary . . . . . . . . . . . . . . . . . . . . . . . . . 18
+ 6. Security Considerations . . . . . . . . . . . . . . . . . . . 18
+ 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
+ 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 19
+ 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 19
+ 9.1. Normative References . . . . . . . . . . . . . . . . . . 19
+ 9.2. Informative References . . . . . . . . . . . . . . . . . 21
+ Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 2]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+1. Introduction
+
+ Benchmarking Methodology Working Group (BMWG) has traditionally
+ conducted laboratory characterization of dedicated physical
+ implementations of internetworking functions. The Black-box
+ Benchmarks of Throughput, Latency, Forwarding Rates and others have
+ served our industry for many years. Now, Network Function
+ Virtualization (NFV) has the goal to transform how internetwork
+ functions are implemented, and therefore has garnered much attention.
+
+ This memo describes the progress of the Open Platform for NFV (OPNFV)
+ project on virtual switch performance characterization,
+ "VSWITCHPERF". This project intends to build on the current and
+ completed work of the Benchmarking Methodology Working Group in IETF,
+ by referencing existing literature. For example, currently the most
+ often referenced RFC is [RFC2544] (which depends on [RFC1242]) and
+ foundation of the benchmarking work in OPNFV is common and strong.
+
+ See https://wiki.opnfv.org/
+ characterize_vswitch_performance_for_telco_nfv_use_cases for more
+ background, and the OPNFV website for general information:
+ https://www.opnfv.org/
+
+ The authors note that OPNFV distinguishes itself from other open
+ source compute and networking projects through its emphasis on
+ existing "telco" services as opposed to cloud-computing. There are
+ many ways in which telco requirements have different emphasis on
+ performance dimensions when compared to cloud computing: support for
+ and transfer of isochronous media streams is one example.
+
+ Note also that the move to NFV Infrastructure has resulted in many
+ new benchmarking initiatives across the industry, and the authors are
+ currently doing their best to maintain alignment with many other
+ projects, and this Internet Draft is evidence of the efforts.
+
+2. Scope
+
+ The primary purpose and scope of the memo is to inform BMWG of work-
+ in-progress that builds on the body of extensive literature and
+ experience. Additionally, once the initial information conveyed here
+ is received, this memo may be expanded to include more detail and
+ commentary from both BMWG and OPNFV communities, under BMWG's
+ chartered work to characterize the NFV Infrastructure (a virtual
+ switch is an important aspect of that infrastructure).
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 3]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+3. Benchmarking Considerations
+
+ This section highlights some specific considerations (from
+ [I-D.ietf-bmwg-virtual-net])related to Benchmarks for virtual
+ switches. The OPNFV project is sharing its present view on these
+ areas, as they develop their specifications in the Level Test Design
+ (LTD) document.
+
+3.1. Comparison with Physical Network Functions
+
+ To compare the performance of virtual designs and implementations
+ with their physical counterparts, identical benchmarks are needed.
+ BMWG has developed specifications for many network functions this
+ memo re-uses existing benchmarks through references, and expands them
+ during development of new methods. A key configuration aspect is the
+ number of parallel cores required to achieve comparable performance
+ with a given physical device, or whether some limit of scale was
+ reached before the cores could achieve the comparable level.
+
+ It's unlikely that the virtual switch will be the only application
+ running on the SUT, so CPU utilization, Cache utilization, and Memory
+ footprint should also be recorded for the virtual implementations of
+ internetworking functions.
+
+3.2. Continued Emphasis on Black-Box Benchmarks
+
+ External observations remain essential as the basis for Benchmarks.
+ Internal observations with fixed specification and interpretation
+ will be provided in parallel to assist the development of operations
+ procedures when the technology is deployed.
+
+3.3. New Configuration Parameters
+
+ A key consideration when conducting any sort of benchmark is trying
+ to ensure the consistency and repeatability of test results. When
+ benchmarking the performance of a vSwitch there are many factors that
+ can affect the consistency of results, one key factor is matching the
+ various hardware and software details of the SUT. This section lists
+ some of the many new parameters which this project believes are
+ critical to report in order to achieve repeatability.
+
+ Hardware details including:
+
+ o Platform details
+
+ o Processor details
+
+ o Memory information (type and size)
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 4]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ o Number of enabled cores
+
+ o Number of cores used for the test
+
+ o Number of physical NICs, as well as their details (manufacturer,
+ versions, type and the PCI slot they are plugged into)
+
+ o NIC interrupt configuration
+
+ o BIOS version, release date and any configurations that were
+ modified
+
+ o CPU microcode level
+
+ o Memory DIMM configurations (quad rank performance may not be the
+ same as dual rank) in size, freq and slot locations
+
+ o PCI configuration parameters (payload size, early ack option...)
+
+ o Power management at all levels (ACPI sleep states, processor
+ package, OS...)
+
+ Software details including:
+
+ o OS parameters and behavior (text vs graphical no one typing at the
+ console on one system)
+
+ o OS version (for host and VNF)
+
+ o Kernel version (for host and VNF)
+
+ o GRUB boot parameters (for host and VNF)
+
+ o Hypervisor details (Type and version)
+
+ o Selected vSwitch, version number or commit id used
+
+ o vSwitch launch command line if it has been parameterised
+
+ o Memory allocation to the vSwitch
+
+ o which NUMA node it is using, and how many memory channels
+
+ o DPDK or any other SW dependency version number or commit id used
+
+ o Memory allocation to a VM - if it's from Hugpages/elsewhere
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 5]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ o VM storage type: snapshot/independent persistent/independent non-
+ persistent
+
+ o Number of VMs
+
+ o Number of Virtual NICs (vNICs), versions, type and driver
+
+ o Number of virtual CPUs and their core affinity on the host
+
+ o Number vNIC interrupt configuration
+
+ o Thread affinitization for the applications (including the vSwitch
+ itself) on the host
+
+ o Details of Resource isolation, such as CPUs designated for Host/
+ Kernel (isolcpu) and CPUs designated for specific processes
+ (taskset). - Test duration. - Number of flows.
+
+ Test Traffic Information:
+
+ o Traffic type - UDP, TCP, IMIX / Other
+
+ o Packet Sizes
+
+ o Deployment Scenario
+
+3.4. Flow classification
+
+ Virtual switches group packets into flows by processing and matching
+ particular packet or frame header information, or by matching packets
+ based on the input ports. Thus a flow can be thought of a sequence
+ of packets that have the same set of header field values or have
+ arrived on the same port. Performance results can vary based on the
+ parameters the vSwitch uses to match for a flow. The recommended
+ flow classification parameters for any vSwitch performance tests are:
+ the input port, the source IP address, the destination IP address and
+ the Ethernet protocol type field. It is essential to increase the
+ flow timeout time on a vSwitch before conducting any performance
+ tests that do not measure the flow setup time. Normally the first
+ packet of a particular stream will install the flow in the virtual
+ switch which adds an additional latency, subsequent packets of the
+ same flow are not subject to this latency if the flow is already
+ installed on the vSwitch.
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 6]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+3.5. Benchmarks using Baselines with Resource Isolation
+
+ This outline describes measurement of baseline with isolated
+ resources at a high level, which is the intended approach at this
+ time.
+
+ 1. Baselines:
+
+ * Optional: Benchmark platform forwarding capability without a
+ vswitch or VNF for at least 72 hours (serves as a means of
+ platform validation and a means to obtain the base performance
+ for the platform in terms of its maximum forwarding rate and
+ latency).
+
+ Benchmark platform forwarding capability
+
+ __
+ +--------------------------------------------------+ |
+ | +------------------------------------------+ | |
+ | | | | |
+ | | Simple Forwarding App | | Host
+ | | | | |
+ | +------------------------------------------+ | |
+ | | NIC | | |
+ +---+------------------------------------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+ * Benchmark VNF forwarding capability with direct connectivity
+ (vSwitch bypass, e.g., SR/IOV) for at least 72 hours (serves
+ as a means of VNF validation and a means to obtain the base
+ performance for the VNF in terms of its maximum forwarding
+ rate and latency). The metrics gathered from this test will
+ serve as a key comparison point for vSwitch bypass
+ technologies performance and vSwitch performance.
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 7]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ Benchmark VNF forwarding capability
+
+ __
+ +--------------------------------------------------+ |
+ | +------------------------------------------+ | |
+ | | | | |
+ | | VNF | | |
+ | | | | |
+ | +------------------------------------------+ | |
+ | | Passthrough/SR-IOV | | Host
+ | +------------------------------------------+ | |
+ | | NIC | | |
+ +---+------------------------------------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+ * Benchmarking with isolated resources alone, with other
+ resources (both HW&SW) disabled Example, vSw and VM are SUT
+
+ * Benchmarking with isolated resources alone, leaving some
+ resources unused
+
+ * Benchmark with isolated resources and all resources occupied
+
+ 2. Next Steps
+
+ * Limited sharing
+
+ * Production scenarios
+
+ * Stressful scenarios
+
+4. VSWITCHPERF Specification Summary
+
+ The overall specification in preparation is referred to as a Level
+ Test Design (LTD) document, which will contain a suite of performance
+ tests. The base performance tests in the LTD are based on the pre-
+ existing specifications developed by BMWG to test the performance of
+ physical switches. These specifications include:
+
+ o [RFC2544] Benchmarking Methodology for Network Interconnect
+ Devices
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 8]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ o [RFC2889] Benchmarking Methodology for LAN Switching
+
+ o [RFC6201] Device Reset Characterization
+
+ o [RFC5481] Packet Delay Variation Applicability Statement
+
+ Some of the above/newer RFCs are being applied in benchmarking for
+ the first time, and represent a development challenge for test
+ equipment developers. Fortunately, many members of the testing
+ system community have engaged on the VSPERF project, including an
+ open source test system.
+
+ In addition to this, the LTD also re-uses the terminology defined by:
+
+ o [RFC2285] Benchmarking Terminology for LAN Switching Devices
+
+ o [RFC5481] Packet Delay Variation Applicability Statement
+
+ Specifications to be included in future updates of the LTD include:
+
+ o [RFC3918] Methodology for IP Multicast Benchmarking
+
+ o [RFC4737] Packet Reordering Metrics
+
+ As one might expect, the most fundamental internetworking
+ characteristics of Throughput and Latency remain important when the
+ switch is virtualized, and these benchmarks figure prominently in the
+ specification.
+
+ When considering characteristics important to "telco" network
+ functions, we must begin to consider additional performance metrics.
+ In this case, the project specifications have referenced metrics from
+ the IETF IP Performance Metrics (IPPM) literature. This means that
+ the [RFC2544] test of Latency is replaced by measurement of a metric
+ derived from IPPM's [RFC2679], where a set of statistical summaries
+ will be provided (mean, max, min, etc.). Further metrics planned to
+ be benchmarked include packet delay variation as defined by [RFC5481]
+ , reordering, burst behaviour, DUT availability, DUT capacity and
+ packet loss in long term testing at Throughput level, where some low-
+ level of background loss may be present and characterized.
+
+ Tests have been (or will be) designed to collect the metrics below:
+
+ o Throughput Tests to measure the maximum forwarding rate (in frames
+ per second or fps) and bit rate (in Mbps) for a constant load (as
+ defined by RFC1242) without traffic loss.
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 9]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ o Packet and Frame Delay Distribution Tests to measure average, min
+ and max packet and frame delay for constant loads.
+
+ o Packet Delay Tests to understand latency distribution for
+ different packet sizes and over an extended test run to uncover
+ outliers.
+
+ o Scalability Tests to understand how the virtual switch performs as
+ the number of flows, active ports, complexity of the forwarding
+ logic's configuration... it has to deal with increases.
+
+ o Stream Performance Tests (TCP, UDP) to measure bulk data transfer
+ performance, i.e. how fast systems can send and receive data
+ through the switch.
+
+ o Control Path and Datapath Coupling Tests, to understand how
+ closely coupled the datapath and the control path are as well as
+ the effect of this coupling on the performance of the DUT
+ (example: delay of the initial packet of a flow).
+
+ o CPU and Memory Consumption Tests to understand the virtual
+ switch's footprint on the system, usually conducted as auxiliary
+ measurements with benchmarks above. They include: CPU
+ utilization, Cache utilization and Memory footprint.
+
+ Future/planned test specs include:
+
+ o Request/Response Performance Tests (TCP, UDP) which measure the
+ transaction rate through the switch.
+
+ o Noisy Neighbour Tests, to understand the effects of resource
+ sharing on the performance of a virtual switch.
+
+ o Tests derived from examination of ETSI NFV Draft GS IFA003
+ requirements [IFA003] on characterization of acceleration
+ technologies applied to vswitches.
+
+ The flexibility of deployment of a virtual switch within a network
+ means that the BMWG IETF existing literature needs to be used to
+ characterize the performance of a switch in various deployment
+ scenarios. The deployment scenarios under consideration include:
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 10]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ Physical port to virtual switch to physical port
+
+ __
+ +--------------------------------------------------+ |
+ | +--------------------+ | |
+ | | | | |
+ | | v | | Host
+ | +--------------+ +--------------+ | |
+ | | phy port | vSwitch | phy port | | |
+ +---+--------------+------------+--------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 11]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ Physical port to virtual switch to VNF to virtual switch to physical
+ port
+
+ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | ^ : | |
+ | | | | | Guest
+ | : v | |
+ | +---------------+ +---------------+ | |
+ | | logical port 0| | logical port 1| | |
+ +---+---------------+-----------+---------------+---+ __|
+ ^ :
+ | |
+ : v __
+ +---+---------------+----------+---------------+---+ |
+ | | logical port 0| | logical port 1| | |
+ | +---------------+ +---------------+ | |
+ | ^ : | |
+ | | | | | Host
+ | : v | |
+ | +--------------+ +--------------+ | |
+ | | phy port | vSwitch | phy port | | |
+ +---+--------------+------------+--------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 12]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ Physical port to virtual switch to VNF to virtual switch to VNF to
+ virtual switch to physical port
+
+ __
+ +----------------------+ +----------------------+ |
+ | Guest 1 | | Guest 2 | |
+ | +---------------+ | | +---------------+ | |
+ | | Application | | | | Application | | |
+ | +---------------+ | | +---------------+ | |
+ | ^ | | | ^ | | |
+ | | v | | | v | | Guests
+ | +---------------+ | | +---------------+ | |
+ | | logical ports | | | | logical ports | | |
+ | | 0 1 | | | | 0 1 | | |
+ +---+---------------+--+ +---+---------------+--+__|
+ ^ : ^ :
+ | | | |
+ : v : v _
+ +---+---------------+---------+---------------+--+ |
+ | | 0 1 | | 3 4 | | |
+ | | logical ports | | logical ports | | |
+ | +---------------+ +---------------+ | |
+ | ^ | ^ | | | Host
+ | | |-----------------| v | |
+ | +--------------+ +--------------+ | |
+ | | phy ports | vSwitch | phy ports | | |
+ +---+--------------+----------+--------------+---+_|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 13]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ Physical port to virtual switch to VNF
+
+ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | ^ | |
+ | | | | Guest
+ | : | |
+ | +---------------+ | |
+ | | logical port 0| | |
+ +---+---------------+-------------------------------+ __|
+ ^
+ |
+ : __
+ +---+---------------+------------------------------+ |
+ | | logical port 0| | |
+ | +---------------+ | |
+ | ^ | |
+ | | | | Host
+ | : | |
+ | +--------------+ | |
+ | | phy port | vSwitch | |
+ +---+--------------+------------ -------------- ---+ __|
+ ^
+ |
+ :
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 14]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ VNF to virtual switch to physical port
+
+ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | : | |
+ | | | | Guest
+ | v | |
+ | +---------------+ | |
+ | | logical port | | |
+ +-------------------------------+---------------+---+ __|
+ :
+ |
+ v __
+ +------------------------------+---------------+---+ |
+ | | logical port | | |
+ | +---------------+ | |
+ | : | |
+ | | | | Host
+ | v | |
+ | +--------------+ | |
+ | vSwitch | phy port | | |
+ +-------------------------------+--------------+---+ __|
+ :
+ |
+ v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 15]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ VNF to virtual switch to VNF
+
+ __
+ +----------------------+ +----------------------+ |
+ | Guest 1 | | Guest 2 | |
+ | +---------------+ | | +---------------+ | |
+ | | Application | | | | Application | | |
+ | +---------------+ | | +---------------+ | |
+ | | | | ^ | |
+ | v | | | | | Guests
+ | +---------------+ | | +---------------+ | |
+ | | logical ports | | | | logical ports | | |
+ | | 0 | | | | 0 | | |
+ +---+---------------+--+ +---+---------------+--+__|
+ : ^
+ | |
+ v : _
+ +---+---------------+---------+---------------+--+ |
+ | | 1 | | 1 | | |
+ | | logical ports | | logical ports | | |
+ | +---------------+ +---------------+ | |
+ | | ^ | | Host
+ | L-----------------+ | |
+ | | |
+ | vSwitch | |
+ +------------------------------------------------+_|
+
+ A set of Deployment Scenario figures is available on the VSPERF Test
+ Methodology Wiki page [TestTopo].
+
+5. 3x3 Matrix Coverage
+
+ This section organizes the many existing test specifications into the
+ "3x3" matrix (introduced in [I-D.ietf-bmwg-virtual-net]). Because
+ the LTD specification ID names are quite long, this section is
+ organized into lists for each occupied cell of the matrix (not all
+ are occupied, also the matrix has grown to 3x4 to accommodate scale
+ metrics when displaying the coverage of many metrics/benchmarks).
+
+ The tests listed below assess the activation of paths in the data
+ plane, rather than the control plane.
+
+ A complete list of tests with short summaries is available on the
+ VSPERF "LTD Test Spec Overview" Wiki page [LTDoverV].
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 16]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+5.1. Speed of Activation
+
+ o Activation.RFC2889.AddressLearningRate
+
+ o PacketLatency.InitialPacketProcessingLatency
+
+5.2. Accuracy of Activation section
+
+ o CPDP.Coupling.Flow.Addition
+
+5.3. Reliability of Activation
+
+ o Throughput.RFC2544.SystemRecoveryTime
+
+ o Throughput.RFC2544.ResetTime
+
+5.4. Scale of Activation
+
+ o Activation.RFC2889.AddressCachingCapacity
+
+5.5. Speed of Operation
+
+ o Throughput.RFC2544.PacketLossRate
+
+ o CPU.RFC2544.0PacketLoss
+
+ o Throughput.RFC2544.PacketLossRateFrameModification
+
+ o Throughput.RFC2544.BackToBackFrames
+
+ o Throughput.RFC2889.MaxForwardingRate
+
+ o Throughput.RFC2889.ForwardPressure
+
+ o Throughput.RFC2889.BroadcastFrameForwarding
+
+5.6. Accuracy of Operation
+
+ o Throughput.RFC2889.ErrorFramesFiltering
+
+ o Throughput.RFC2544.Profile
+
+5.7. Reliability of Operation
+
+ o Throughput.RFC2889.Soak
+
+ o Throughput.RFC2889.SoakFrameModification
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 17]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ o PacketDelayVariation.RFC3393.Soak
+
+5.8. Scalability of Operation
+
+ o Scalability.RFC2544.0PacketLoss
+
+ o MemoryBandwidth.RFC2544.0PacketLoss.Scalability
+
+5.9. Summary
+
+|------------------------------------------------------------------------|
+| | | | | |
+| | SPEED | ACCURACY | RELIABILITY | SCALE |
+| | | | | |
+|------------------------------------------------------------------------|
+| | | | | |
+| Activation | X | X | X | X |
+| | | | | |
+|------------------------------------------------------------------------|
+| | | | | |
+| Operation | X | X | X | X |
+| | | | | |
+|------------------------------------------------------------------------|
+| | | | | |
+| De-activation | | | | |
+| | | | | |
+|------------------------------------------------------------------------|
+
+6. Security Considerations
+
+ Benchmarking activities as described in this memo are limited to
+ technology characterization of a Device Under Test/System Under Test
+ (DUT/SUT) using controlled stimuli in a laboratory environment, with
+ dedicated address space and the constraints specified in the sections
+ above.
+
+ The benchmarking network topology will be an independent test setup
+ and MUST NOT be connected to devices that may forward the test
+ traffic into a production network, or misroute traffic to the test
+ management network.
+
+ Further, benchmarking is performed on a "black-box" basis, relying
+ solely on measurements observable external to the DUT/SUT.
+
+ Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
+ benchmarking purposes. Any implications for network security arising
+ from the DUT/SUT SHOULD be identical in the lab and in production
+ networks.
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 18]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+7. IANA Considerations
+
+ No IANA Action is requested at this time.
+
+8. Acknowledgements
+
+ The authors acknowledge
+
+9. References
+
+9.1. Normative References
+
+ [NFV.PER001]
+ "Network Function Virtualization: Performance and
+ Portability Best Practices", Group Specification ETSI GS
+ NFV-PER 001 V1.1.1 (2014-06), June 2014.
+
+ [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
+ Requirement Levels", BCP 14, RFC 2119,
+ DOI 10.17487/RFC2119, March 1997,
+ <http://www.rfc-editor.org/info/rfc2119>.
+
+ [RFC2285] Mandeville, R., "Benchmarking Terminology for LAN
+ Switching Devices", RFC 2285, DOI 10.17487/RFC2285,
+ February 1998, <http://www.rfc-editor.org/info/rfc2285>.
+
+ [RFC2330] Paxson, V., Almes, G., Mahdavi, J., and M. Mathis,
+ "Framework for IP Performance Metrics", RFC 2330,
+ DOI 10.17487/RFC2330, May 1998,
+ <http://www.rfc-editor.org/info/rfc2330>.
+
+ [RFC2544] Bradner, S. and J. McQuaid, "Benchmarking Methodology for
+ Network Interconnect Devices", RFC 2544,
+ DOI 10.17487/RFC2544, March 1999,
+ <http://www.rfc-editor.org/info/rfc2544>.
+
+ [RFC2679] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
+ Delay Metric for IPPM", RFC 2679, DOI 10.17487/RFC2679,
+ September 1999, <http://www.rfc-editor.org/info/rfc2679>.
+
+ [RFC2680] Almes, G., Kalidindi, S., and M. Zekauskas, "A One-way
+ Packet Loss Metric for IPPM", RFC 2680,
+ DOI 10.17487/RFC2680, September 1999,
+ <http://www.rfc-editor.org/info/rfc2680>.
+
+ [RFC2681] Almes, G., Kalidindi, S., and M. Zekauskas, "A Round-trip
+ Delay Metric for IPPM", RFC 2681, DOI 10.17487/RFC2681,
+ September 1999, <http://www.rfc-editor.org/info/rfc2681>.
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 19]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ [RFC2889] Mandeville, R. and J. Perser, "Benchmarking Methodology
+ for LAN Switching Devices", RFC 2889,
+ DOI 10.17487/RFC2889, August 2000,
+ <http://www.rfc-editor.org/info/rfc2889>.
+
+ [RFC3393] Demichelis, C. and P. Chimento, "IP Packet Delay Variation
+ Metric for IP Performance Metrics (IPPM)", RFC 3393,
+ DOI 10.17487/RFC3393, November 2002,
+ <http://www.rfc-editor.org/info/rfc3393>.
+
+ [RFC3432] Raisanen, V., Grotefeld, G., and A. Morton, "Network
+ performance measurement with periodic streams", RFC 3432,
+ DOI 10.17487/RFC3432, November 2002,
+ <http://www.rfc-editor.org/info/rfc3432>.
+
+ [RFC3918] Stopp, D. and B. Hickman, "Methodology for IP Multicast
+ Benchmarking", RFC 3918, DOI 10.17487/RFC3918, October
+ 2004, <http://www.rfc-editor.org/info/rfc3918>.
+
+ [RFC4689] Poretsky, S., Perser, J., Erramilli, S., and S. Khurana,
+ "Terminology for Benchmarking Network-layer Traffic
+ Control Mechanisms", RFC 4689, DOI 10.17487/RFC4689,
+ October 2006, <http://www.rfc-editor.org/info/rfc4689>.
+
+ [RFC4737] Morton, A., Ciavattone, L., Ramachandran, G., Shalunov,
+ S., and J. Perser, "Packet Reordering Metrics", RFC 4737,
+ DOI 10.17487/RFC4737, November 2006,
+ <http://www.rfc-editor.org/info/rfc4737>.
+
+ [RFC5357] Hedayat, K., Krzanowski, R., Morton, A., Yum, K., and J.
+ Babiarz, "A Two-Way Active Measurement Protocol (TWAMP)",
+ RFC 5357, DOI 10.17487/RFC5357, October 2008,
+ <http://www.rfc-editor.org/info/rfc5357>.
+
+ [RFC5905] Mills, D., Martin, J., Ed., Burbank, J., and W. Kasch,
+ "Network Time Protocol Version 4: Protocol and Algorithms
+ Specification", RFC 5905, DOI 10.17487/RFC5905, June 2010,
+ <http://www.rfc-editor.org/info/rfc5905>.
+
+ [RFC6201] Asati, R., Pignataro, C., Calabria, F., and C. Olvera,
+ "Device Reset Characterization", RFC 6201,
+ DOI 10.17487/RFC6201, March 2011,
+ <http://www.rfc-editor.org/info/rfc6201>.
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 20]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+9.2. Informative References
+
+ [I-D.ietf-bmwg-virtual-net]
+ Morton, A., "Considerations for Benchmarking Virtual
+ Network Functions and Their Infrastructure", draft-ietf-
+ bmwg-virtual-net-01 (work in progress), September 2015.
+
+ [IFA003] "https://docbox.etsi.org/ISG/NFV/Open/Drafts/
+ IFA003_Acceleration_-_vSwitch_Spec/".
+
+ [LTDoverV]
+ "LTD Test Spec Overview https://wiki.opnfv.org/wiki/
+ vswitchperf_test_spec_review".
+
+ [RFC1242] Bradner, S., "Benchmarking Terminology for Network
+ Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242,
+ July 1991, <http://www.rfc-editor.org/info/rfc1242>.
+
+ [RFC5481] Morton, A. and B. Claise, "Packet Delay Variation
+ Applicability Statement", RFC 5481, DOI 10.17487/RFC5481,
+ March 2009, <http://www.rfc-editor.org/info/rfc5481>.
+
+ [RFC6049] Morton, A. and E. Stephan, "Spatial Composition of
+ Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011,
+ <http://www.rfc-editor.org/info/rfc6049>.
+
+ [RFC6248] Morton, A., "RFC 4148 and the IP Performance Metrics
+ (IPPM) Registry of Metrics Are Obsolete", RFC 6248,
+ DOI 10.17487/RFC6248, April 2011,
+ <http://www.rfc-editor.org/info/rfc6248>.
+
+ [RFC6390] Clark, A. and B. Claise, "Guidelines for Considering New
+ Performance Metric Development", BCP 170, RFC 6390,
+ DOI 10.17487/RFC6390, October 2011,
+ <http://www.rfc-editor.org/info/rfc6390>.
+
+ [TestTopo]
+ "Test Topologies https://wiki.opnfv.org/vsperf/
+ test_methodology".
+
+Authors' Addresses
+
+ Maryam Tahhan
+ Intel
+
+ Email: maryam.tahhan@intel.com
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 21]
+\f
+Internet-Draft Benchmarking vSwitches October 2015
+
+
+ Billy O'Mahony
+ Intel
+
+ Email: billy.o.mahony@intel.com
+
+
+ Al Morton
+ AT&T Labs
+ 200 Laurel Avenue South
+ Middletown,, NJ 07748
+ USA
+
+ Phone: +1 732 420 1571
+ Fax: +1 732 368 1192
+ Email: acmorton@att.com
+ URI: http://home.comcast.net/~acmacm/
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Tahhan, et al. Expires April 16, 2016 [Page 22]
Code Review / vswitchperf.git / commitdiff