--- /dev/null
+<?xml version="1.0" encoding="US-ASCII"?>
+<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
+<?rfc toc="yes"?>
+<?rfc tocompact="yes"?>
+<?rfc tocdepth="3"?>
+<?rfc tocindent="yes"?>
+<?rfc symrefs="yes"?>
+<?rfc sortrefs="yes"?>
+<?rfc comments="yes"?>
+<?rfc inline="yes"?>
+<?rfc compact="yes"?>
+<?rfc subcompact="no"?>
+<rfc category="info" docName="draft-vsperf-bmwg-vswitch-opnfv-00"
+ ipr="trust200902">
+ <front>
+ <title abbrev="Benchmarking vSwitches">Benchmarking Virtual Switches in
+ OPNFV</title>
+
+ <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
+ <organization>Intel</organization>
+
+ <address>
+ <postal>
+ <street/>
+
+ <city/>
+
+ <region/>
+
+ <code/>
+
+ <country/>
+ </postal>
+
+ <phone/>
+
+ <facsimile/>
+
+ <email/>
+
+ <uri/>
+ </address>
+ </author>
+
+ <author fullname="Billy O Mahony" initials="B." surname="O Mahony">
+ <organization>Intel</organization>
+
+ <address>
+ <postal>
+ <street/>
+
+ <city/>
+
+ <region/>
+
+ <code/>
+
+ <country/>
+ </postal>
+
+ <phone/>
+
+ <facsimile/>
+
+ <email/>
+
+ <uri/>
+ </address>
+ </author>
+
+ <author fullname="Al Morton" initials="A." surname="Morton">
+ <organization>AT&T Labs</organization>
+
+ <address>
+ <postal>
+ <street>200 Laurel Avenue South</street>
+
+ <city>Middletown,</city>
+
+ <region>NJ</region>
+
+ <code>07748</code>
+
+ <country>USA</country>
+ </postal>
+
+ <phone>+1 732 420 1571</phone>
+
+ <facsimile>+1 732 368 1192</facsimile>
+
+ <email>acmorton@att.com</email>
+
+ <uri>http://home.comcast.net/~acmacm/</uri>
+ </address>
+ </author>
+
+ <date day="6" month="June" year="2015"/>
+
+ <abstract>
+ <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
+ project on virtual switch performance "VSWITCHPERF". This project
+ intends to build on the current and completed work of the Benchmarking
+ Methodology Working Group in IETF, by referencing existing literature.
+ The Benchmarking Methodology Working Group has traditionally conducted
+ laboratory characterization of dedicated physical implementations of
+ internetworking functions. Therefore, this memo begins to describes the
+ additional considerations when virtual switches are implemented in
+ general-purpose hardware. The expanded tests and benchmarks are also
+ influenced by the OPNFV mission to support virtualization of the "telco"
+ infrastructure.</t>
+ </abstract>
+
+ <note title="Requirements Language">
+ <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+ document are to be interpreted as described in <xref
+ target="RFC2119">RFC 2119</xref>.</t>
+
+ <t/>
+ </note>
+ </front>
+
+ <middle>
+ <section title="Introduction">
+ <t>Benchmarking Methodology Working Group (BMWG) has traditionally
+ conducted laboratory characterization of dedicated physical
+ implementations of internetworking functions. The Black-box Benchmarks
+ of Throughput, Latency, Forwarding Rates and others have served our
+ industry for many years. Now, Network Function Virtualization (NFV) has
+ the goal to transform how internetwork functions are implemented, and
+ therefore has garnered much attention.</t>
+
+ <t>This memo describes the progress of the Open Platform for NFV (OPNFV)
+ project on virtual switch performance characterization, "VSWITCHPERF".
+ This project intends to build on the current and completed work of the
+ Benchmarking Methodology Working Group in IETF, by referencing existing
+ literature. For example, currently the most referenced RFC is <xref
+ target="RFC2544"/> (which depens on <xref target="RFC1242"/>) and
+ foundation of the benchmarking work in OPNFV is common and strong.</t>
+
+ <t>See
+ https://wiki.opnfv.org/characterize_vswitch_performance_for_telco_nfv_use_cases
+ for more background, and the OPNFV website for general information:
+ https://www.opnfv.org/</t>
+
+ <t>The authors note that OPNFV distinguishes itself from other open
+ source compute and networking projects through its emphasis on existing
+ "telco" services as opposed to cloud-computing. There are many ways in
+ which telco requirements have different emhasis on performance
+ dimensions when compared to cloud computing: support for and transfer of
+ isochronous media streams is one example.</t>
+
+ <t>Note also that the move to NFV Infrastructure has resulted in many
+ new benchmarking initiatives across the industry, and the authors are
+ currently doing their best to maintain alignment with many other
+ projects, and this Internet Draft is evidence of the efforts.</t>
+ </section>
+
+ <section title="Scope">
+ <t>The primary purpose and scope of the memo is to inform BMWG of
+ work-in-progress that builds on the body of extensive literature and
+ experience. Additionally, once the initial information conveyed here is
+ received, this memo may be expanded to include more detail and
+ commentary from both BMWG and OPNFV communities, under BMWG's chartered
+ work to characterize the NFV Infrastructure (a virtual switch is an
+ important aspect of that infrastructure).</t>
+ </section>
+
+ <section title="Benchmarking Considerations">
+ <t>This section highlights some specific considerations (from <xref
+ target="ID.ietf-bmwg-virtual-net"/>)related to Benchmarks for virtual
+ switches.</t>
+
+ <section title="Comparison with Physical Network Functions">
+ <t>To compare the performance of virtual designs and implementations
+ with their physical counterparts, identical benchmarks are needed.
+ BMWG has developed specifications for many network functions this memo
+ re-uses existing benchmarks through references, and expands them
+ during development of new methods. A key configuration aspect is the
+ number of parallel cores required to achieve comparable performance
+ with a given physical device, or whether some limit of scale was
+ reached before the cores could achieve the comparable level. </t>
+
+ <t>It's unlikely that the virtual switch will be the only application
+ running on the SUT, so CPU utilization, Cache utilization, and Memory
+ footprint should also be recorded for the virtual implementations of
+ internetworking functions.</t>
+ </section>
+
+ <section title="Continued Emphasis on Black-Box Benchmarks">
+ <t>External observations remain essential as the basis for Benchmarks.
+ Internal observations with fixed specification and interpretation will
+ be provided in parallel to assist the development of operations
+ procedures when the technology is deployed. </t>
+ </section>
+ </section>
+
+ <section title="VSWITCHPERF Specification Summary">
+ <t>The overall specification in preparation is referred to as a Level
+ Test Design (LTD) document, which will contain a suite of performace
+ tests.</t>
+
+ <t>As one might expect, the most fundamental internetworking
+ characteristics of Throughput and Latency remain important when the
+ switch is virtualized, and these benchmarks figure prominently in the
+ specification.</t>
+
+ <t>When considering characteristics important to "telco" network
+ functions, we must begin to consider additional performance metrics. In
+ this case, the project specifications have referenced metrics from the
+ IETF IP Performance Metrics (IPPM) literature. This means that the <xref
+ target="RFC2544"/> test of Latency is replaced by measurement of a
+ metric derived from IPPM's <xref target="RFC2679"/>, where a set of
+ statistical summaries will be provided (mean, max, min, etc.). Further
+ metrics planned to be benchmarked include packet delay variation as
+ defined by <xref target="RFC5481"/> , reordering, burst behaviour, DUT
+ availability, DUT capacity and packet loss in long term testing at
+ Throughput level, where some low-level of background loss may be present
+ and characterized.</t>
+
+ <t>Tests have been (or will be) designed to collect the metrics
+ below:</t>
+
+ <t><list style="symbols">
+ <t>Throughput Tests to measure the maximum forwarding rate (in
+ frames per second or fps) and bit rate (in Mbps) for a constant load
+ (as defined by RFC1242) without traffic loss.</t>
+
+ <t>Packet and Frame Delay Distribution Tests to measure average, min
+ and max packet and frame delay for constant loads.</t>
+
+ <t>Packet Delay Tests to understand latency distribution for
+ different packet sizes and over an extended test run to uncover
+ outliers.</t>
+
+ <t>Scalability Tests to understand how the virtual switch performs
+ as the number of flows, active ports, complexity of the forwarding
+ logic’s configuration… it has to deal with
+ increases.</t>
+
+ <t>Stream Performance Tests (TCP, UDP) to measure bulk data transfer
+ performance, i.e. how fast systems can send and receive data through
+ the switch.</t>
+
+ <t>Request/Response Performance Tests (TCP, UDP) the measure the
+ transaction rate through the switch.</t>
+
+ <t>Control Path and Datapath Coupling Tests, to understand how
+ closely coupled the datapath and the control path are as well as the
+ effect of this coupling on the performance of the DUT (example:
+ delay of the initial packet of a flow).</t>
+
+ <t>Noisy Neighbour Tests, to understand the effects of resource
+ sharing on the performance of a virtual switch.</t>
+
+ <t>CPU and Memory Consumption Tests to understand the virtual
+ switch’s footprint on the system, usually conducted as
+ auxilliary measurements with benchmarks above. They include: CPU
+ utilization, Cache utilization and Memory footprint.</t>
+ </list></t>
+
+ <t>The felixability of deployemnt of a virtual switch within a network
+ means that the BMWG IETF existing literature needs to be used to
+ characterize the performance of a switch in various deployment
+ scenarios. The deployment scenarios under consideration include:</t>
+
+ <t><figure>
+ <preamble>Physical port to virtual switch to physical
+ port</preamble>
+
+ <artwork><![CDATA[ __
+ +--------------------------------------------------+ |
+ | +--------------------+ | |
+ | | | | |
+ | | v | | Host
+ | +--------------+ +--------------+ | |
+ | | phy port | vSwitch | phy port | | |
+ +---+--------------+------------+--------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+]]></artwork>
+ </figure></t>
+
+ <t><figure>
+ <preamble>Physical port to virtual switch to VNF to virtual switch
+ to physical port</preamble>
+
+ <artwork><![CDATA[ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | ^ : | |
+ | | | | | Guest
+ | : v | |
+ | +---------------+ +---------------+ | |
+ | | logical port 0| | logical port 1| | |
+ +---+---------------+-----------+---------------+---+ __|
+ ^ :
+ | |
+ : v __
+ +---+---------------+----------+---------------+---+ |
+ | | logical port 0| | logical port 1| | |
+ | +---------------+ +---------------+ | |
+ | ^ : | |
+ | | | | | Host
+ | : v | |
+ | +--------------+ +--------------+ | |
+ | | phy port | vSwitch | phy port | | |
+ +---+--------------+------------+--------------+---+ __|
+ ^ :
+ | |
+ : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+]]></artwork>
+ </figure><figure>
+ <preamble>Physical port to virtual switch to VNF to virtual switch
+ to VNF to virtual switch to physical port</preamble>
+
+ <artwork><![CDATA[ __
+ +----------------------+ +----------------------+ |
+ | Guest 1 | | Guest 2 | |
+ | +---------------+ | | +---------------+ | |
+ | | Application | | | | Application | | |
+ | +---------------+ | | +---------------+ | |
+ | ^ | | | ^ | | |
+ | | v | | | v | | Guests
+ | +---------------+ | | +---------------+ | |
+ | | logical ports | | | | logical ports | | |
+ | | 0 1 | | | | 0 1 | | |
+ +---+---------------+--+ +---+---------------+--+__|
+ ^ : ^ :
+ | | | |
+ : v : v _
+ +---+---------------+---------+---------------+--+ |
+ | | 0 1 | | 3 4 | | |
+ | | logical ports | | logical ports | | |
+ | +---------------+ +---------------+ | |
+ | ^ | ^ | | | Host
+ | | L-----------------+ v | |
+ | +--------------+ +--------------+ | |
+ | | phy ports | vSwitch | phy ports | | |
+ +---+--------------+----------+--------------+---+_|
+ ^ : ^ :
+ | | | |
+ : v : v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+]]></artwork>
+ </figure><figure>
+ <preamble>Physical port to virtual switch to VNF</preamble>
+
+ <artwork><![CDATA[ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | ^ | |
+ | | | | Guest
+ | : | |
+ | +---------------+ | |
+ | | logical port 0| | |
+ +---+---------------+-------------------------------+ __|
+ ^
+ |
+ : __
+ +---+---------------+------------------------------+ |
+ | | logical port 0| | |
+ | +---------------+ | |
+ | ^ | |
+ | | | | Host
+ | : | |
+ | +--------------+ | |
+ | | phy port | vSwitch | |
+ +---+--------------+------------ -------------- ---+ __|
+ ^
+ |
+ :
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+]]></artwork>
+ </figure><figure>
+ <preamble>VNF to virtual switch to physical port</preamble>
+
+ <artwork><![CDATA[ __
+ +---------------------------------------------------+ |
+ | | |
+ | +-------------------------------------------+ | |
+ | | Application | | |
+ | +-------------------------------------------+ | |
+ | : | |
+ | | | | Guest
+ | v | |
+ | +---------------+ | |
+ | | logical port | | |
+ +-------------------------------+---------------+---+ __|
+ :
+ |
+ v __
+ +------------------------------+---------------+---+ |
+ | | logical port | | |
+ | +---------------+ | |
+ | : | |
+ | | | | Host
+ | v | |
+ | +--------------+ | |
+ | vSwitch | phy port | | |
+ +-------------------------------+--------------+---+ __|
+ :
+ |
+ v
+ +--------------------------------------------------+
+ | |
+ | traffic generator |
+ | |
+ +--------------------------------------------------+]]></artwork>
+ </figure><figure>
+ <preamble> VNF to virtual switch to VNF</preamble>
+
+ <artwork><![CDATA[ __
+ +----------------------+ +----------------------+ |
+ | Guest 1 | | Guest 2 | |
+ | +---------------+ | | +---------------+ | |
+ | | Application | | | | Application | | |
+ | +---------------+ | | +---------------+ | |
+ | | | | ^ | |
+ | v | | | | | Guests
+ | +---------------+ | | +---------------+ | |
+ | | logical ports | | | | logical ports | | |
+ | | 0 | | | | 0 | | |
+ +---+---------------+--+ +---+---------------+--+__|
+ : ^
+ | |
+ v : _
+ +---+---------------+---------+---------------+--+ |
+ | | 1 | | 1 | | |
+ | | logical ports | | logical ports | | |
+ | +---------------+ +---------------+ | |
+ | | ^ | | Host
+ | L-----------------+ | |
+ | | |
+ | vSwitch | |
+ +------------------------------------------------+_|
+
+]]></artwork>
+ </figure></t>
+ </section>
+
+ <section title="Security Considerations">
+ <t>Benchmarking activities as described in this memo are limited to
+ technology characterization of a Device Under Test/System Under Test
+ (DUT/SUT) using controlled stimuli in a laboratory environment, with
+ dedicated address space and the constraints specified in the sections
+ above.</t>
+
+ <t>The benchmarking network topology will be an independent test setup
+ and MUST NOT be connected to devices that may forward the test traffic
+ into a production network, or misroute traffic to the test management
+ network.</t>
+
+ <t>Further, benchmarking is performed on a "black-box" basis, relying
+ solely on measurements observable external to the DUT/SUT.</t>
+
+ <t>Special capabilities SHOULD NOT exist in the DUT/SUT specifically for
+ benchmarking purposes. Any implications for network security arising
+ from the DUT/SUT SHOULD be identical in the lab and in production
+ networks.</t>
+ </section>
+
+ <section anchor="IANA" title="IANA Considerations">
+ <t>No IANA Action is requested at this time.</t>
+ </section>
+
+ <section title="Acknowledgements">
+ <t>The authors acknowledge</t>
+ </section>
+ </middle>
+
+ <back>
+ <references title="Normative References">
+ <?rfc ?>
+
+ <?rfc include="reference.RFC.2119"?>
+
+ <?rfc include="reference.RFC.2330"?>
+
+ <?rfc include='reference.RFC.2544'?>
+
+ <?rfc include="reference.RFC.2679"?>
+
+ <?rfc include='reference.RFC.2680'?>
+
+ <?rfc include='reference.RFC.3393'?>
+
+ <?rfc include='reference.RFC.3432'?>
+
+ <?rfc include='reference.RFC.2681'?>
+
+ <?rfc include='reference.RFC.5905'?>
+
+ <?rfc include='reference.RFC.4689'?>
+
+ <?rfc include='reference.RFC.4737'?>
+
+ <?rfc include='reference.RFC.5357'?>
+
+ <reference anchor="NFV.PER001">
+ <front>
+ <title>Network Function Virtualization: Performance and Portability
+ Best Practices</title>
+
+ <author fullname="ETSI NFV" initials="" surname="">
+ <organization/>
+ </author>
+
+ <date month="June" year="2014"/>
+ </front>
+
+ <seriesInfo name="Group Specification"
+ value="ETSI GS NFV-PER 001 V1.1.1 (2014-06)"/>
+
+ <format type="PDF"/>
+ </reference>
+ </references>
+
+ <references title="Informative References">
+ <?rfc include='reference.RFC.1242'?>
+
+ <?rfc include='reference.RFC.5481'?>
+
+ <?rfc include='reference.RFC.6049'?>
+
+ <?rfc include='reference.RFC.6248'?>
+
+ <?rfc include='reference.RFC.6390'?>
+
+ <?rfc include='reference.ID.ietf-bmwg-virtual-net'?>
+ </references>
+ </back>
+</rfc>
Code Review / vswitchperf.git / commitdiff