1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
5 licensed under the terms of the Apache License, Version 2.0.
7 =======================
8 Yardstick Release Notes
9 =======================
14 .. _Yardstick: https://wiki.opnfv.org/display/yardstick
16 .. _Dashboard: http://testresults.opnfv.org/grafana/
18 .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
24 This document compiles the release notes for the Gambia release of OPNFV Yardstick.
29 +-------------------+-----------+---------------------------------+
30 | *Date* | *Version* | *Comment* |
32 +-------------------+-----------+---------------------------------+
33 | November 9, 2018 | 7.0.0 | Yardstick for Gambia release |
35 +-------------------+-----------+---------------------------------+
41 The software delivered in the OPNFV Yardstick_ Project, comprising the
42 *Yardstick framework*, and the *Yardstick test cases* is a realization of
43 the methodology in ETSI-ISG NFV-TST001_.
45 The *Yardstick* framework is *installer*, *infrastructure* and *application*
51 This Gambia release provides *Yardstick* as a framework for NFVI testing
52 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
54 * Documentation generated with Sphinx
58 * Release notes (this document)
61 * Automated Yardstick test suite (daily, weekly)
63 * Jenkins Jobs for OPNFV community labs
65 * Automated Yardstick test results visualization
67 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
70 * Yardstick framework source code
72 * Yardstick test cases yaml files
74 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
76 For Gambia release, the *Yardstick framework* is used for the following
79 * OPNFV platform testing - generic test cases to measure the categories:
85 * OPNFV platform network service benchmarking (NSB)
89 * Test cases for the following OPNFV Projects:
99 The *Yardstick framework* is developed in the OPNFV community, by the
102 .. note:: The test case description template used for the Yardstick test cases
103 is based on the document ETSI-ISG NFV-TST001_; the results report template
104 used for the Yardstick results is based on the IEEE Std 829-2008.
110 +--------------------------------+-----------------------+
111 | **Project** | Yardstick |
113 +--------------------------------+-----------------------+
114 | **Repo/tag** | yardstick/opnfv-7.0.0 |
116 +--------------------------------+-----------------------+
117 | **Yardstick Docker image tag** | opnfv-7.0.0 |
119 +--------------------------------+-----------------------+
120 | **Release designation** | Gambia 7.0 |
122 +--------------------------------+-----------------------+
123 | **Release date** | November 9, 2018 |
125 +--------------------------------+-----------------------+
126 | **Purpose of the delivery** | OPNFV Gambia 7.0.0 |
128 +--------------------------------+-----------------------+
137 - User Guide: :ref:`<yardstick:userguide>`
139 - Developer Guide: :ref:`<yardstick:devguide>`
142 Software Deliverables
143 ---------------------
145 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)
150 +--------------+-------------------------------------------+
151 | **Context** | **Description** |
153 +--------------+-------------------------------------------+
154 | *Heat* | Models orchestration using OpenStack Heat |
156 +--------------+-------------------------------------------+
157 | *Node* | Models Baremetal, Controller, Compute |
159 +--------------+-------------------------------------------+
160 | *Standalone* | Models VM running on Non-Managed NFVi |
162 +--------------+-------------------------------------------+
163 | *Kubernetes* | Models VM running on Non-Managed NFVi |
165 +--------------+-------------------------------------------+
171 .. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC".
173 +----------------+-------------------------------------------------------+
174 | **Runner** | **Description** |
176 +----------------+-------------------------------------------------------+
177 | *Arithmetic* | Steps every run arithmetically according to specified |
180 +----------------+-------------------------------------------------------+
181 | *Duration* | Runs for a specified period of time |
183 +----------------+-------------------------------------------------------+
184 | *Iteration* | Runs for a specified number of iterations |
186 +----------------+-------------------------------------------------------+
187 | *IterationIPC* | Runs a configurable number of times before it |
188 | | returns. Each iteration has a configurable timeout. |
190 +----------------+-------------------------------------------------------+
191 | *Sequence* | Selects input value to a scenario from an input file |
192 | | and runs all entries sequentially |
194 +----------------+-------------------------------------------------------+
195 | *Dynamictp* | A runner that searches for the max throughput with |
198 +----------------+-------------------------------------------------------+
199 | *Search* | A runner that runs a specific time before it returns |
201 +----------------+-------------------------------------------------------+
207 +----------------+-----------------------------------------------------+
208 | **Category** | **Delivered** |
210 +----------------+-----------------------------------------------------+
211 | *Availability* | Attacker: |
213 | | * baremetal, process |
217 | | * check host, openstack, process, service |
219 | | * start/stop service |
223 | | * command, process |
225 +----------------+-----------------------------------------------------+
226 | *Compute* | * cpuload |
229 | | * lmbench_cache |
235 | | * computecapacity |
238 +----------------+-----------------------------------------------------+
239 | *Networking* | * iperf3 |
246 | | * sfc with tacker |
247 | | * networkcapacity |
248 | | * netutilization |
252 +----------------+-----------------------------------------------------+
253 | *Parser* | Tosca2Heat |
255 +----------------+-----------------------------------------------------+
256 | *Storage* | * fio |
258 | | * storagecapacity |
260 +----------------+-----------------------------------------------------+
261 | *StorPerf* | storperf |
263 +----------------+-----------------------------------------------------+
264 | *NSB* | vFW thoughput test case |
266 +----------------+-----------------------------------------------------+
272 .. note:: Yardstick Gambia 7.0.0 adds no new test cases.
274 * Generic NFVI test cases
276 * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
280 * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
286 Module Version Changes
287 ----------------------
289 This is the seventh tracked release of Yardstick. It is based on following
293 - OpenDayLight Oxygen
296 Document Version Changes
297 ------------------------
299 This is the seventh tracked version of the Yardstick framework in OPNFV.
300 It includes the following documentation updates:
302 - Yardstick User Guide:
304 - Remove vTC chapter;
306 - Yardstick Developer Guide
307 - Yardstick Release Notes for Yardstick: this document
313 - Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)....
316 - Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
317 - PROX enhancements and the addition of Standalone test case using SRIOV and
319 - Ixia enhancements for vBNG and PPPoE traffic.
320 - Improvements of unit tests and gating.
321 - Add DPDK pktgen traffic generator.
322 - Kubernetes context enhancements.
323 - Kubernetes sample test cases of fio and lmbench added.
329 For Gambia 7.0.0, Yardstick was tested on the following scenarios:
331 +-------------------------+------+---------+----------+------+
332 | Scenario | Apex | Compass | Fuel-arm | Fuel |
333 +=========================+======+=========+==========+======+
334 | os-nosdn-nofeature-noha | X | | | |
335 +-------------------------+------+---------+----------+------+
336 | os-nosdn-nofeature-ha | X | | | |
337 +-------------------------+------+---------+----------+------+
338 | os-odl-bgpvpn-noha | X | | | |
339 +-------------------------+------+---------+----------+------+
340 | os-nosdn-calipso-noha | X | | | |
341 +-------------------------+------+---------+----------+------+
342 | os-nosdn-kvm-ha | | X | | |
343 +-------------------------+------+---------+----------+------+
344 | os-odl-nofeature-ha | | | X | X |
345 +-------------------------+------+---------+----------+------+
346 | os-odl-sfc-noha | X | | | |
347 +-------------------------+------+---------+----------+------+
348 | os-nosdn-ovs-ha | | | | X |
349 +-------------------------+------+---------+----------+------+
350 | k8-nosdn-nofeature-ha | | X | | |
351 +-------------------------+------+---------+----------+------+
352 | k8-nosdn-stor4nfv-noha | | X | | |
353 +-------------------------+------+---------+----------+------+
354 | k8-nosdn-stor4nfv-ha | | X | | |
355 +-------------------------+------+---------+----------+------+
361 Test results are available in:
363 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
375 +--------------------+--------------------------------------------------------------------------+
376 | **JIRA REFERENCE** | **DESCRIPTION** |
377 +====================+==========================================================================+
378 | YARDSTICK-1137 | Fix CLI argument handling in nsb_setup.sh |
379 +--------------------+--------------------------------------------------------------------------+
380 | YARDSTICK-1220 | Get stats for multiple port simultaneously |
381 +--------------------+--------------------------------------------------------------------------+
382 | YARDSTICK-1260 | Added missing functionality to start VM and access it using SSH keys |
383 | | in Standalone contexts. |
384 +--------------------+--------------------------------------------------------------------------+
385 | YARDSTICK-1298 | Allows for in-line overriding/modification of traffic profile variables |
386 | | from the testcase file. |
387 +--------------------+--------------------------------------------------------------------------+
388 | YARDSTICK-1368 | Updated existing test cases in Yardstick to minimize changes done |
389 | | manually to run standalone tests for Trex. |
390 +--------------------+--------------------------------------------------------------------------+
391 | YARDSTICK-1389 | Add status filed for RFC2544 TC iterations |
392 +--------------------+--------------------------------------------------------------------------+
393 | YARDSTICK-1395 | Update 'configure_uwsgi' role to work in baremetal/container modes. |
394 +--------------------+--------------------------------------------------------------------------+
395 | YARDSTICK-1402 | Change IP assignment for VM to static for standalone context |
396 +--------------------+--------------------------------------------------------------------------+
397 | YARDSTICK-1404 | CPU Utilization for VNF and traffic generator are now graphed on Grafana |
398 +--------------------+--------------------------------------------------------------------------+
399 | YARDSTICK-1411 | Fix Yardstick Docker image ARM build |
400 +--------------------+--------------------------------------------------------------------------+
401 | YARDSTICK-1414 | Update the pinned sampleVNF version to use a commit instead of a branch |
402 +--------------------+--------------------------------------------------------------------------+
403 | YARDSTICK-1418 | NSB PROX NFVi test now stops after reaching expected precision |
404 +--------------------+--------------------------------------------------------------------------+
405 | YARDSTICK-1457 | Fix influxdb "field type conflict" error |
406 +--------------------+--------------------------------------------------------------------------+
407 | YARDSTICK-1458 | Update Grafana to display "real-time" data instead of historical data. |
408 +--------------------+--------------------------------------------------------------------------+
409 | YARDSTICK-1462 | NSB: Add OvS 2.8.1 support in SA context |
410 +--------------------+--------------------------------------------------------------------------+
411 | YARDSTICK-1492 | Make OvS server to listen on TCP |
412 +--------------------+--------------------------------------------------------------------------+
413 | YARDSTICK-1493 | The RX queues number is hard-codded and cannot be changed |
414 +--------------------+--------------------------------------------------------------------------+
417 Gambia 7.0.0 known restrictions/issues
418 ======================================
420 +-----------+-----------------------+------------------+
421 | Installer | Scenario | Issue |
422 +===========+=======================+==================+
423 | apex | os-ovn-nofeature-ha | YARDSTICK-1517 |
424 +-----------+-----------------------+------------------+
429 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
431 - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia
433 - Yardstick repo: https://git.opnfv.org/yardstick
435 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
437 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
439 - Yardstick IRC channel: #opnfv-yardstick