5 OPNFV Fraser release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
11 licensed under the terms of the Apache License, Version 2.0.
13 =======================================
14 OPNFV Fraser Release Note for Yardstick
15 =======================================
20 .. _Yardstick: https://wiki.opnfv.org/yardstick
22 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
24 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
30 This document describes the release note of Yardstick project.
35 +-------------------+-----------+---------------------------------+
36 | *Date* | *Version* | *Comment* |
38 +-------------------+-----------+---------------------------------+
39 | April 27, 2018 | 6.0.0 | Yardstick for Fraser release |
41 +-------------------+-----------+---------------------------------+
47 The software delivered in the OPNFV Yardstick_ Project, comprising the
48 *Yardstick framework*, and the *Yardstick test cases* is a realization of
49 the methodology in ETSI-ISG NFV-TST001_.
51 The *Yardstick* framework is *installer*, *infrastructure* and *application*
57 This Fraser release provides *Yardstick* as a framework for NFVI testing
58 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
60 * Documentation generated with Sphinx
64 * Release notes (this document)
67 * Automated Yardstick test suite (daily, weekly)
69 * Jenkins Jobs for OPNFV community labs
71 * Automated Yardstick test results visualization
73 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
76 * Yardstick framework source code
78 * Yardstick test cases yaml files
80 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
82 For Fraser release, the *Yardstick framework* is used for the following
85 * OPNFV platform testing - generic test cases to measure the categories:
91 * OPNFV platform network service benchmarking (NSB)
95 * Test cases for the following OPNFV Projects:
105 The *Yardstick framework* is developed in the OPNFV community, by the
108 .. note:: The test case description template used for the Yardstick test cases
109 is based on the document ETSI-ISG NFV-TST001_; the results report template
110 used for the Yardstick results is based on the IEEE Std 829-2008.
116 +--------------------------------+-----------------------+
117 | **Project** | Yardstick |
119 +--------------------------------+-----------------------+
120 | **Repo/tag** | yardstick/opnfv-6.0.0 |
122 +--------------------------------+-----------------------+
123 | **Yardstick Docker image tag** | opnfv-6.0.0 |
125 +--------------------------------+-----------------------+
126 | **Release designation** | Fraser |
128 +--------------------------------+-----------------------+
129 | **Release date** | April 27, 2018 |
131 +--------------------------------+-----------------------+
132 | **Purpose of the delivery** | OPNFV Fraser 6.0.0 |
134 +--------------------------------+-----------------------+
143 - User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/index.html
145 - Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/developer/devguide/index.html
148 Software Deliverables
149 ---------------------
151 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.0.0)
156 +--------------+-------------------------------------------+
157 | **Context** | **Description** |
159 +--------------+-------------------------------------------+
160 | *Heat* | Models orchestration using OpenStack Heat |
162 +--------------+-------------------------------------------+
163 | *Node* | Models Baremetal, Controller, Compute |
165 +--------------+-------------------------------------------+
166 | *Standalone* | Models VM running on Non-Managed NFVi |
168 +--------------+-------------------------------------------+
169 | *Kubernetes* | Models VM running on Non-Managed NFVi |
171 +--------------+-------------------------------------------+
177 Note: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search".
179 +---------------+-------------------------------------------------------+
180 | **Runner** | **Description** |
182 +---------------+-------------------------------------------------------+
183 | *Arithmetic* | Steps every run arithmetically according to specified |
186 +---------------+-------------------------------------------------------+
187 | *Duration* | Runs for a specified period of time |
189 +---------------+-------------------------------------------------------+
190 | *Iteration* | Runs for a specified number of iterations |
192 +---------------+-------------------------------------------------------+
193 | *Sequence* | Selects input value to a scenario from an input file |
194 | | and runs all entries sequentially |
196 +---------------+-------------------------------------------------------+
197 | **Dynamictp** | A runner that searches for the max throughput with |
200 +---------------+-------------------------------------------------------+
201 | **Search** | A runner that runs a specific time before it returns |
203 +---------------+-------------------------------------------------------+
209 +----------------+-----------------------------------------------------+
210 | **Category** | **Delivered** |
212 +----------------+-----------------------------------------------------+
213 | *Availability* | Attacker: |
215 | | * baremetal, process |
219 | | * check host, openstack, process, service |
221 | | * start/stop service |
225 | | * command, process |
227 +----------------+-----------------------------------------------------+
228 | *Compute* | * cpuload |
231 | | * lmbench_cache |
237 | | * computecapacity |
240 +----------------+-----------------------------------------------------+
241 | *Networking* | * iperf3 |
248 | | * sfc with tacker |
249 | | * networkcapacity |
250 | | * netutilization |
254 +----------------+-----------------------------------------------------+
255 | *Parser* | Tosca2Heat |
257 +----------------+-----------------------------------------------------+
258 | *Storage* | * fio |
260 | | * storagecapacity |
262 +----------------+-----------------------------------------------------+
263 | *StorPerf* | storperf |
265 +----------------+-----------------------------------------------------+
266 | *NSB* | vFW thoughput test case |
268 +----------------+-----------------------------------------------------+
274 * Generic NFVI test cases
276 * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
280 * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration
281 * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance
282 * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api
288 Module Version Changes
289 ----------------------
291 This is the sixth tracked release of Yardstick. It is based on following
295 - OpenDayLight Oxygen
298 Document Version Changes
299 ------------------------
301 This is the sixth tracked version of the Yardstick framework in OPNFV.
302 It includes the following documentation updates:
304 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
305 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
306 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
307 - Yardstick Developer Guide
308 - Yardstick Release Notes for Yardstick: this document
314 - Plugin-based test cases support Heat context
315 - SR-IOV support for the Heat context
316 - Support using existing network in Heat context
317 - Support running test cases with existing VNFs/without destroying VNF in Heat context
318 - Add vFW scale-up template
319 - Improvements of unit tests and gating
320 - GUI improvement about passing parameters
326 For Fraser 6.0.0, Yardstick was tested on the following scenarios:
328 +-------------------------+------+---------+----------+------+------+-------+
329 | Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy |
330 +=========================+======+=========+==========+======+======+=======+
331 | os-nosdn-nofeature-noha | X | X | | | X | |
332 +-------------------------+------+---------+----------+------+------+-------+
333 | os-nosdn-nofeature-ha | X | X | X | X | X | X |
334 +-------------------------+------+---------+----------+------+------+-------+
335 | os-nosdn-bar-noha | X | X | | | | |
336 +-------------------------+------+---------+----------+------+------+-------+
337 | os-nosdn-bar-ha | X | | | | | |
338 +-------------------------+------+---------+----------+------+------+-------+
339 | os-odl-bgpvpn-ha | X | | | | | |
340 +-------------------------+------+---------+----------+------+------+-------+
341 | os-nosdn-calipso-noha | X | | | | | |
342 +-------------------------+------+---------+----------+------+------+-------+
343 | os-nosdn-kvm-ha | | X | | | | |
344 +-------------------------+------+---------+----------+------+------+-------+
345 | os-odl_l3-nofeature-ha | | X | | | | |
346 +-------------------------+------+---------+----------+------+------+-------+
347 | os-odl-sfc-ha | | X | | | | |
348 +-------------------------+------+---------+----------+------+------+-------+
349 | os-odl-nofeature-ha | | | | X | | X |
350 +-------------------------+------+---------+----------+------+------+-------+
351 | os-nosdn-ovs-ha | | | | X | | |
352 +-------------------------+------+---------+----------+------+------+-------+
353 | k8-nosdn-nofeature-ha | | X | | | | |
354 +-------------------------+------+---------+----------+------+------+-------+
355 | k8-nosdn-stor4nfv-noha | | X | | | | |
356 +-------------------------+------+---------+----------+------+------+-------+
362 Test results are available in:
364 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
366 The reporting pages can be found at:
368 +---------------+----------------------------------------------------------------------------------+
369 | apex | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html |
370 +---------------+----------------------------------------------------------------------------------+
371 | compass | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html |
372 +---------------+----------------------------------------------------------------------------------+
373 | fuel\@x86 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html |
374 +---------------+----------------------------------------------------------------------------------+
375 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html |
376 +---------------+----------------------------------------------------------------------------------+
377 | joid | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html |
378 +---------------+----------------------------------------------------------------------------------+
389 +--------------------+--------------------------------------------------------------------------+
390 | **JIRA REFERENCE** | **DESCRIPTION** |
391 +====================+==========================================================================+
392 | YARDSTICK-831 | tc053 kill haproxy wrong |
393 +--------------------+--------------------------------------------------------------------------+
394 | YARDSTICK-842 | load image fails when there's cirros image exist |
395 +--------------------+--------------------------------------------------------------------------+
396 | YARDSTICK-857 | tc006 failed due to volume attached to different location "/dev/vdc" |
397 +--------------------+--------------------------------------------------------------------------+
398 | YARDSTICK-874 | Specify supported architecture for Ubuntu backports repository |
399 +--------------------+--------------------------------------------------------------------------+
400 | YARDSTICK-875 | Check if multiverse repository is available in Ubuntu |
401 +--------------------+--------------------------------------------------------------------------+
402 | YARDSTICK-893 | Fix proxy env handling and ansible multinode support |
403 +--------------------+--------------------------------------------------------------------------+
404 | YARDSTICK-899 | Variable local_iface_name is read before it is set |
405 +--------------------+--------------------------------------------------------------------------+
406 | YARDSTICK-900 | Section in "upload_yardstick_image.yml" invalid |
407 +--------------------+--------------------------------------------------------------------------+
408 | YARDSTICK-911 | Remove 'inconsistent-return-statements' from Pylint checks |
409 +--------------------+--------------------------------------------------------------------------+
410 | YARDSTICK-989 | Yardstick real-time influxdb KPI reporting regressions |
411 +--------------------+--------------------------------------------------------------------------+
412 | YARDSTICK-994 | NSB set-up build script for baremetal broken |
413 +--------------------+--------------------------------------------------------------------------+
414 | YARDSTICK-996 | Error in address input format in "_ip_range_action_partial" |
415 +--------------------+--------------------------------------------------------------------------+
416 | YARDSTICK-1003 | Prox vnf descriptor cleanup for tg and vnf |
417 +--------------------+--------------------------------------------------------------------------+
418 | YARDSTICK-1006 | Ansible destroy script will fail if vm has already been undefined |
419 +--------------------+--------------------------------------------------------------------------+
420 | YARDSTICK-1012 | constants: fix pylint warnings for OSError |
421 +--------------------+--------------------------------------------------------------------------+
422 | YARDSTICK-1014 | Remove unused args in |
423 | | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile |
424 +--------------------+--------------------------------------------------------------------------+
425 | YARDSTICK-1016 | Allow vm to access outside world through default gateway |
426 +--------------------+--------------------------------------------------------------------------+
427 | YARDSTICK-1019 | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script |
428 +--------------------+--------------------------------------------------------------------------+
429 | YARDSTICK-1021 | NSB: All Sample VNF test cases timeout after 1 hour of execution |
430 +--------------------+--------------------------------------------------------------------------+
431 | YARDSTICK-1036 | Prox: Addition of storage of extra counters for Grafana |
432 +--------------------+--------------------------------------------------------------------------+
433 | YARDSTICK-1038 | Missing file which is described in the operation_conf.yaml |
434 +--------------------+--------------------------------------------------------------------------+
435 | YARDSTICK-1047 | Error in string format in HeatTemplateError message |
436 +--------------------+--------------------------------------------------------------------------+
437 | YARDSTICK-1056 | yardstick report command print error when run test case |
438 +--------------------+--------------------------------------------------------------------------+
439 | YARDSTICK-1059 | Reduce the log level if TRex client is no connected |
440 +--------------------+--------------------------------------------------------------------------+
441 | YARDSTICK-1073 | Error when retrieving "options" section in "scenario" |
442 +--------------------+--------------------------------------------------------------------------+
443 | YARDSTICK-1080 | Running Test Case in Latest Yardstick Docker Image shows Error |
444 +--------------------+--------------------------------------------------------------------------+
445 | YARDSTICK-1082 | tc043,tc055, tc063, tc075, pass wrong node name in the ci scenario yaml |
446 +--------------------+--------------------------------------------------------------------------+
447 | YARDSTICK-1102 | Don't hide exception traceback from Task.start() |
448 +--------------------+--------------------------------------------------------------------------+
449 | YARDSTICK-1107 | bad exception traceback print due to atexit_handler |
450 +--------------------+--------------------------------------------------------------------------+
451 | YARDSTICK-1120 | HA test case tc050 should start monitor before attack |
452 +--------------------+--------------------------------------------------------------------------+
454 Fraser 6.0.0 known restrictions/issues
455 ======================================
457 +-----------+-----------+----------------------------------------------+
458 | Installer | Scenario | Issue |
459 +===========+===========+==============================================+
461 +-----------+-----------+----------------------------------------------+
466 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
468 - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser
470 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
472 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
474 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
476 - Yardstick IRC chanel: #opnfv-yardstick