1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
5 licensed under the terms of the Apache License, Version 2.0.
7 =======================
8 Yardstick Release Notes
9 =======================
14 .. _Yardstick: https://wiki.opnfv.org/display/yardstick
16 .. _Dashboard: http://testresults.opnfv.org/grafana/
18 .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
24 This document compiles the release notes for the Gambia release of OPNFV Yardstick.
29 +-------------------+-----------+---------------------------------+
30 | *Date* | *Version* | *Comment* |
32 +-------------------+-----------+---------------------------------+
33 | November 9, 2018 | 7.0.0 | Yardstick for Gambia release |
35 +-------------------+-----------+---------------------------------+
36 | December 14, 2018 | 7.1.0 | Yardstick for Gambia release |
38 +-------------------+-----------+---------------------------------+
44 The software delivered in the OPNFV Yardstick_ Project, comprising the
45 *Yardstick framework*, and the *Yardstick test cases* is a realization of
46 the methodology in ETSI-ISG NFV-TST001_.
48 The *Yardstick* framework is *installer*, *infrastructure* and *application*
54 This Gambia release provides *Yardstick* as a framework for NFVI testing
55 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
57 * Documentation generated with Sphinx
61 * Release notes (this document)
64 * Automated Yardstick test suite (daily, weekly)
66 * Jenkins Jobs for OPNFV community labs
68 * Automated Yardstick test results visualization
70 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
73 * Yardstick framework source code
75 * Yardstick test cases yaml files
77 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
79 For Gambia release, the *Yardstick framework* is used for the following
82 * OPNFV platform testing - generic test cases to measure the categories:
88 * OPNFV platform network service benchmarking (NSB)
92 * Test cases for the following OPNFV Projects:
102 The *Yardstick framework* is developed in the OPNFV community, by the
105 .. note:: The test case description template used for the Yardstick test cases
106 is based on the document ETSI-ISG NFV-TST001_; the results report template
107 used for the Yardstick results is based on the IEEE Std 829-2008.
113 +--------------------------------+-----------------------+
114 | **Project** | Yardstick |
116 +--------------------------------+-----------------------+
117 | **Repo/tag** | yardstick/opnfv-7.1.0 |
119 +--------------------------------+-----------------------+
120 | **Yardstick Docker image tag** | opnfv-7.1.0 |
122 +--------------------------------+-----------------------+
123 | **Release designation** | Gambia 7.1 |
125 +--------------------------------+-----------------------+
126 | **Release date** | December 14, 2018 |
128 +--------------------------------+-----------------------+
129 | **Purpose of the delivery** | OPNFV Gambia 7.1.0 |
131 +--------------------------------+-----------------------+
140 - User Guide: :ref:`<yardstick:userguide>`
142 - Developer Guide: :ref:`<yardstick:devguide>`
145 Software Deliverables
146 ---------------------
148 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)
153 +--------------+-------------------------------------------+
154 | **Context** | **Description** |
156 +--------------+-------------------------------------------+
157 | *Heat* | Models orchestration using OpenStack Heat |
159 +--------------+-------------------------------------------+
160 | *Node* | Models Baremetal, Controller, Compute |
162 +--------------+-------------------------------------------+
163 | *Standalone* | Models VM running on Non-Managed NFVi |
165 +--------------+-------------------------------------------+
166 | *Kubernetes* | Models VM running on Non-Managed NFVi |
168 +--------------+-------------------------------------------+
174 .. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC".
176 +----------------+-------------------------------------------------------+
177 | **Runner** | **Description** |
179 +----------------+-------------------------------------------------------+
180 | *Arithmetic* | Steps every run arithmetically according to specified |
183 +----------------+-------------------------------------------------------+
184 | *Duration* | Runs for a specified period of time |
186 +----------------+-------------------------------------------------------+
187 | *Iteration* | Runs for a specified number of iterations |
189 +----------------+-------------------------------------------------------+
190 | *IterationIPC* | Runs a configurable number of times before it |
191 | | returns. Each iteration has a configurable timeout. |
193 +----------------+-------------------------------------------------------+
194 | *Sequence* | Selects input value to a scenario from an input file |
195 | | and runs all entries sequentially |
197 +----------------+-------------------------------------------------------+
198 | *Dynamictp* | A runner that searches for the max throughput with |
201 +----------------+-------------------------------------------------------+
202 | *Search* | A runner that runs a specific time before it returns |
204 +----------------+-------------------------------------------------------+
210 +----------------+-----------------------------------------------------+
211 | **Category** | **Delivered** |
213 +----------------+-----------------------------------------------------+
214 | *Availability* | Attacker: |
216 | | * baremetal, process |
220 | | * check host, openstack, process, service |
222 | | * start/stop service |
226 | | * command, process |
228 +----------------+-----------------------------------------------------+
229 | *Compute* | * cpuload |
232 | | * lmbench_cache |
238 | | * computecapacity |
241 +----------------+-----------------------------------------------------+
242 | *Networking* | * iperf3 |
249 | | * sfc with tacker |
250 | | * networkcapacity |
251 | | * netutilization |
255 +----------------+-----------------------------------------------------+
256 | *Parser* | Tosca2Heat |
258 +----------------+-----------------------------------------------------+
259 | *Storage* | * fio |
261 | | * storagecapacity |
263 +----------------+-----------------------------------------------------+
264 | *StorPerf* | storperf |
266 +----------------+-----------------------------------------------------+
267 | *NSB* | vFW thoughput test case |
269 +----------------+-----------------------------------------------------+
275 .. note:: Yardstick Gambia 7.1.0 adds no new test cases.
277 * Generic NFVI test cases
279 * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
283 * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
289 Module Version Changes
290 ----------------------
292 This is the seventh tracked release of Yardstick. It is based on following
296 - OpenDayLight Oxygen
299 Document Version Changes
300 ------------------------
302 This is the seventh tracked version of the Yardstick framework in OPNFV.
303 It includes the following documentation updates:
305 - Yardstick User Guide:
307 - Remove vTC chapter;
309 - Yardstick Developer Guide
310 - Yardstick Release Notes for Yardstick: this document
316 - Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)....
319 - Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
320 - PROX enhancements and the addition of Standalone test case using SRIOV and
322 - Ixia enhancements for vBNG and PPPoE traffic.
323 - Improvements of unit tests and gating.
324 - Add DPDK pktgen traffic generator.
325 - Kubernetes context enhancements.
326 - Kubernetes sample test cases of fio and lmbench added.
332 For Gambia 7.1.0, Yardstick was tested on the following scenarios:
334 +-------------------------+------+---------+----------+------+
335 | Scenario | Apex | Compass | Fuel-arm | Fuel |
336 +=========================+======+=========+==========+======+
337 | os-nosdn-nofeature-noha | X | | | |
338 +-------------------------+------+---------+----------+------+
339 | os-nosdn-nofeature-ha | X | | | |
340 +-------------------------+------+---------+----------+------+
341 | os-odl-bgpvpn-noha | X | | | |
342 +-------------------------+------+---------+----------+------+
343 | os-nosdn-calipso-noha | X | | | |
344 +-------------------------+------+---------+----------+------+
345 | os-nosdn-kvm-ha | | X | | |
346 +-------------------------+------+---------+----------+------+
347 | os-odl-nofeature-ha | | | X | X |
348 +-------------------------+------+---------+----------+------+
349 | os-odl-sfc-noha | X | | | |
350 +-------------------------+------+---------+----------+------+
351 | os-nosdn-ovs-ha | | | | X |
352 +-------------------------+------+---------+----------+------+
353 | k8-nosdn-nofeature-ha | | X | | |
354 +-------------------------+------+---------+----------+------+
355 | k8-nosdn-stor4nfv-noha | | X | | |
356 +-------------------------+------+---------+----------+------+
357 | k8-nosdn-stor4nfv-ha | | X | | |
358 +-------------------------+------+---------+----------+------+
364 Test results are available in:
366 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
378 +--------------------+--------------------------------------------------------------------------+
379 | **JIRA REFERENCE** | **DESCRIPTION** |
380 +====================+==========================================================================+
381 | YARDSTICK-1241 | Update NSB PROX devguide. |
382 +--------------------+--------------------------------------------------------------------------+
383 | YARDSTICK-1458 | NSB NFVi PROX Should report realtime port activity not historical data. |
384 +--------------------+--------------------------------------------------------------------------+
385 | YARDSTICK-1471 | Add Testcase Prox Standalone SRIOV. |
386 +--------------------+--------------------------------------------------------------------------+
387 | YARDSTICK-1475 | Adding Testcase for Prox Stanalone OvS-DPDK. |
388 +--------------------+--------------------------------------------------------------------------+
389 | YARDSTICK-1500 | Adding Testcase for Prox L2FWD PktTouch Stanalone OvS-DPDK. |
390 +--------------------+--------------------------------------------------------------------------+
391 | YARDSTICK-1517 | Missing opnfv "os-ovn-nofeature-ha" scenario test suite. |
392 +--------------------+--------------------------------------------------------------------------+
393 | YARDSTICK-l526 | Run testcase 074 result overridden by job status. |
394 +--------------------+--------------------------------------------------------------------------+
395 | YARDSTICK-1547 | Adding scale up test case for l3fwd OvS-DPDK. |
396 +--------------------+--------------------------------------------------------------------------+
397 | YARDSTICK-1560 | Fix pip environment. |
398 +--------------------+--------------------------------------------------------------------------+
399 | YARDSTICK-1561 | L3FWD Gradana Dashboards Out-of-date and incorrect. |
400 +--------------------+--------------------------------------------------------------------------+
402 Gambia 7.1.0 known restrictions/issues
403 ======================================
409 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
411 - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia
413 - Yardstick repo: https://git.opnfv.org/yardstick
415 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
417 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
419 - Yardstick IRC channel: #opnfv-yardstick