.. http://creativecommons.org/licenses/by/4.0
.. (c) OPNFV, 2016-2017 Intel Corporation.
+.. Convention for heading levels in Yardstick documentation:
+
+ ======= Heading 0 (reserved for the title in a document)
+ ------- Heading 1
+ ^^^^^^^ Heading 2
+ +++++++ Heading 3
+ ''''''' Heading 4
+
+ Avoid deeper levels because they do not render well.
+
===================================
Network Services Benchmarking (NSB)
===================================
Overview
--------
-The goal of NSB is to Extend Yardstick to perform real world VNFs and NFVi
-Characterization and benchmarking with repeatable and deterministic methods.
-
-The Network Service Benchmarking (NSB) extends the yardstick framework to do
-VNF characterization and benchmarking in three different execution
-environments - bare metal i.e. native Linux environment, standalone virtual
-environment and managed virtualized environment (e.g. Open stack etc.).
-It also brings in the capability to interact with external traffic generators
-both hardware & software based for triggering and validating the traffic
-according to user defined profiles.
+Network Services Benchmarking (:term:`NSB`) uses the :term:`Yardstick`
+framework for performing :term:`VNF` and :term:`NFVI` characterisation in an
+:term:`NFV` environment.
+
+For VNF characterisation, NSB will onboard a VNF, source and sink traffic to it
+via traffic generators, and collect a variety of key performance indicators
+(:term:`KPI`) during VNF execution. The stream of KPI data is stored in a
+database, and it is visualized in a performance-visualization dashboard.
+
+For NFVI characterisation, a fixed test VNF, called :term:`PROX` is used.
+PROX implements a suite of test cases and visualizes the output data of the
+test suite. The PROX test cases implement various execution kernels found in
+real-world VNFs, and the output of the test cases provides an indication of
+the fitness of the infrastructure for running NFV services, in addition to
+indicating potential performance optimizations for the NFVI.
+
+NSB extends the Yardstick framework to do VNF characterization in three
+different execution environments - bare metal i.e. native Linux environment,
+standalone virtual environment and managed virtualized environment (e.g.
+OpenStack). It also brings in the capability to interact with external traffic
+generators, both hardware and software based, for triggering and validating the
+traffic according to user defined profiles.
NSB extension includes:
* Generic data models of Network Services, based on ETSI spec
`ETSI GS NFV-TST 001`_
-* Standalone :term:`context` for VNF testing like SRIOV, OVS, OVS-DPDK, etc
+* Standalone :term:`context` for VNF testing SRIOV, OVS, OVS-DPDK, etc
* Generic VNF configuration models and metrics implemented with Python
classes
* Traffic generator features and traffic profiles
* L4-L7 state-full traffic profiles
* Tunneling protocol/network overlay support
+* Scenarios that handle NSB test cases execution
+
+ * NSPerf - scenario that handles generic NSB test case execution
+ (setup and init tg/vnf, trigger traffic on tg, collect kpi)
+ * NSPerf-RFC2544 - scenario that allows repeatable triggering of traffic on
+ traffic generators until test case acceptance criteria is met
+ (for example RFC2544 binary search)
+
* Test case samples
* Ping
* VNF KPIs e.g. packet_in, packet_drop, packet_fwd
Architecture
-============
+------------
The Network Service (NS) defines a set of Virtual Network Functions (VNF)
connected together using NFV infrastructure.
* Retrieve the KPI's provided by particular VNF
Components of Network Service
------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. TODO: provide a list of components in this section and describe them in
+ later sub-sections
+
+.. Components are the methodology, TGs, framework extensions, KPI collection,
+ Testcases, SampleVNFs
+.. Framework extentions include: VNF models, NSPerf Scenario, contexts
* *Models for Network Service benchmarking*: The Network Service benchmarking
requires the proper modelling approach. The NSB provides models using Python
files and defining of NSDs and VNFDs.
-The benchmark control application being a part of OPNFV yardstick can call
-that python models to instantiate and configure the VNFs. Depending on
+The benchmark control application being a part of OPNFV Yardstick can call
+that Python models to instantiate and configure the VNFs. Depending on
infrastructure type (bare-metal or fully virtualized) that calls could be
made directly or using MANO system.
* RFC2544 throughput for various loss rate defined (1% is a default)
+KPI Collection
+^^^^^^^^^^^^^^
+
+KPI collection is the process of sampling KPIs at multiple intervals to allow
+for investigation into anomalies during runtime. Some KPI intervals are
+adjustable. KPIs are collected from traffic generators and NFVI for the SUT.
+There is already some reporting in NSB available, but NSB collects all KPIs for
+analytics to process.
+
+Below is an example list of basic KPIs:
+* Throughput
+* Latency
+* Packet delay variation
+* Maximum establishment rate
+* Maximum tear-down rate
+* Maximum simultaneous number of sessions
+
+Of course, there can be many other KPIs that will be relevant for a specific
+NFVI, but in most cases these KPIs are enough to give you a basic picture of
+the SUT. NSB also uses :term:`collectd` in order to collect the KPIs. Currently
+the following collectd plug-ins are enabled for NSB testcases:
+
+* Libvirt
+* Interface stats
+* OvS events
+* vSwitch stats
+* Huge Pages
+* RAM
+* CPU usage
+* Intel® PMU
+* Intel(r) RDT
+
Graphical Overview
-==================
+------------------
NSB Testing with Yardstick framework facilitate performance testing of various
VNFs provided.
Figure 1: Network Service - 2 server configuration
-VNFs supported for chracterization:
------------------------------------
+VNFs supported for chracterization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. CGNAPT - Carrier Grade Network Address and port Translation
2. vFW - Virtual Firewall
3. vACL - Access Control List
-4. Prox - Packet pROcessing eXecution engine:
+4. PROX - Packet pROcessing eXecution engine:
* VNF can act as Drop, Basic Forwarding (no touch),
L2 Forwarding (change MAC), GRE encap/decap, Load balance based on
packet fields, Symmetric load balancing
Code Review / yardstick.git / blobdiff