5 OPNFV Fraser release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
11 licensed under the terms of the Apache License, Version 2.0.
13 =======================================
14 OPNFV Fraser Release Note for Yardstick
15 =======================================
20 .. _Yardstick: https://wiki.opnfv.org/yardstick
22 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
24 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
30 This document describes the release note of Yardstick project.
35 +-------------------+-----------+---------------------------------+
36 | *Date* | *Version* | *Comment* |
38 +-------------------+-----------+---------------------------------+
39 | Jul 2, 2018 | 6.2.1 | Yardstick for Fraser release |
41 +-------------------+-----------+---------------------------------+
42 | Jun 29, 2018 | 6.2.0 | Yardstick for Fraser release |
44 +-------------------+-----------+---------------------------------+
45 | May 25, 2018 | 6.1.0 | Yardstick for Fraser release |
47 +-------------------+-----------+---------------------------------+
48 | April 27, 2018 | 6.0.0 | Yardstick for Fraser release |
50 +-------------------+-----------+---------------------------------+
56 The software delivered in the OPNFV Yardstick_ Project, comprising the
57 *Yardstick framework*, and the *Yardstick test cases* is a realization of
58 the methodology in ETSI-ISG NFV-TST001_.
60 The *Yardstick* framework is *installer*, *infrastructure* and *application*
66 This Fraser release provides *Yardstick* as a framework for NFVI testing
67 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
69 * Documentation generated with Sphinx
73 * Release notes (this document)
76 * Automated Yardstick test suite (daily, weekly)
78 * Jenkins Jobs for OPNFV community labs
80 * Automated Yardstick test results visualization
82 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
85 * Yardstick framework source code
87 * Yardstick test cases yaml files
89 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
91 For Fraser release, the *Yardstick framework* is used for the following
94 * OPNFV platform testing - generic test cases to measure the categories:
100 * OPNFV platform network service benchmarking (NSB)
104 * Test cases for the following OPNFV Projects:
114 The *Yardstick framework* is developed in the OPNFV community, by the
117 .. note:: The test case description template used for the Yardstick test cases
118 is based on the document ETSI-ISG NFV-TST001_; the results report template
119 used for the Yardstick results is based on the IEEE Std 829-2008.
125 +--------------------------------+-----------------------+
126 | **Project** | Yardstick |
128 +--------------------------------+-----------------------+
129 | **Repo/tag** | yardstick/opnfv-6.2.0 |
131 +--------------------------------+-----------------------+
132 | **Yardstick Docker image tag** | opnfv-6.2.0 |
134 +--------------------------------+-----------------------+
135 | **Release designation** | Fraser |
137 +--------------------------------+-----------------------+
138 | **Release date** | Jun 29, 2018 |
140 +--------------------------------+-----------------------+
141 | **Purpose of the delivery** | OPNFV Fraser 6.2.0 |
143 +--------------------------------+-----------------------+
152 - User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/index.html
154 - Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/developer/devguide/index.html
157 Software Deliverables
158 ---------------------
160 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.2.0)
165 +--------------+-------------------------------------------+
166 | **Context** | **Description** |
168 +--------------+-------------------------------------------+
169 | *Heat* | Models orchestration using OpenStack Heat |
171 +--------------+-------------------------------------------+
172 | *Node* | Models Baremetal, Controller, Compute |
174 +--------------+-------------------------------------------+
175 | *Standalone* | Models VM running on Non-Managed NFVi |
177 +--------------+-------------------------------------------+
178 | *Kubernetes* | Models VM running on Non-Managed NFVi |
180 +--------------+-------------------------------------------+
186 .. note:: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search".
188 +---------------+-------------------------------------------------------+
189 | **Runner** | **Description** |
191 +---------------+-------------------------------------------------------+
192 | *Arithmetic* | Steps every run arithmetically according to specified |
195 +---------------+-------------------------------------------------------+
196 | *Duration* | Runs for a specified period of time |
198 +---------------+-------------------------------------------------------+
199 | *Iteration* | Runs for a specified number of iterations |
201 +---------------+-------------------------------------------------------+
202 | *Sequence* | Selects input value to a scenario from an input file |
203 | | and runs all entries sequentially |
205 +---------------+-------------------------------------------------------+
206 | **Dynamictp** | A runner that searches for the max throughput with |
209 +---------------+-------------------------------------------------------+
210 | **Search** | A runner that runs a specific time before it returns |
212 +---------------+-------------------------------------------------------+
218 +----------------+-----------------------------------------------------+
219 | **Category** | **Delivered** |
221 +----------------+-----------------------------------------------------+
222 | *Availability* | Attacker: |
224 | | * baremetal, process |
228 | | * check host, openstack, process, service |
230 | | * start/stop service |
234 | | * command, process |
236 +----------------+-----------------------------------------------------+
237 | *Compute* | * cpuload |
240 | | * lmbench_cache |
246 | | * computecapacity |
249 +----------------+-----------------------------------------------------+
250 | *Networking* | * iperf3 |
257 | | * sfc with tacker |
258 | | * networkcapacity |
259 | | * netutilization |
263 +----------------+-----------------------------------------------------+
264 | *Parser* | Tosca2Heat |
266 +----------------+-----------------------------------------------------+
267 | *Storage* | * fio |
269 | | * storagecapacity |
271 +----------------+-----------------------------------------------------+
272 | *StorPerf* | storperf |
274 +----------------+-----------------------------------------------------+
275 | *NSB* | vFW thoughput test case |
277 +----------------+-----------------------------------------------------+
283 .. note:: Yardstick Fraser 6.1.0 added two new test cases, "TC092" and "TC093".
285 * Generic NFVI test cases
287 * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
291 * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration
292 * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance
293 * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api
294 * OPNFV_YARDSTICK_TC092 - SDN Controller resilience in HA configuration
295 * OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
301 Module Version Changes
302 ----------------------
304 This is the sixth tracked release of Yardstick. It is based on following
308 - OpenDayLight Oxygen
311 Document Version Changes
312 ------------------------
314 This is the sixth tracked version of the Yardstick framework in OPNFV.
315 It includes the following documentation updates:
317 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
318 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
319 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
320 - Yardstick Developer Guide
321 - Yardstick Release Notes for Yardstick: this document
327 - Plugin-based test cases support Heat context
328 - SR-IOV support for the Heat context
329 - Support using existing network in Heat context
330 - Support running test cases with existing VNFs/without destroying VNF in Heat context
331 - Add vFW scale-up template
332 - Improvements of unit tests and gating
333 - GUI improvement about passing parameters
339 For Fraser 6.0.0, Yardstick was tested on the following scenarios:
341 +-------------------------+------+---------+----------+------+------+-------+
342 | Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy |
343 +=========================+======+=========+==========+======+======+=======+
344 | os-nosdn-nofeature-noha | X | X | | | X | |
345 +-------------------------+------+---------+----------+------+------+-------+
346 | os-nosdn-nofeature-ha | X | X | X | X | X | X |
347 +-------------------------+------+---------+----------+------+------+-------+
348 | os-nosdn-bar-noha | X | X | | | | |
349 +-------------------------+------+---------+----------+------+------+-------+
350 | os-nosdn-bar-ha | X | | | | | |
351 +-------------------------+------+---------+----------+------+------+-------+
352 | os-odl-bgpvpn-ha | X | | | | | |
353 +-------------------------+------+---------+----------+------+------+-------+
354 | os-nosdn-calipso-noha | X | | | | | |
355 +-------------------------+------+---------+----------+------+------+-------+
356 | os-nosdn-kvm-ha | | X | | | | |
357 +-------------------------+------+---------+----------+------+------+-------+
358 | os-odl_l3-nofeature-ha | | X | | | | |
359 +-------------------------+------+---------+----------+------+------+-------+
360 | os-odl-sfc-ha | | X | | | | |
361 +-------------------------+------+---------+----------+------+------+-------+
362 | os-odl-nofeature-ha | | | | X | | X |
363 +-------------------------+------+---------+----------+------+------+-------+
364 | os-nosdn-ovs-ha | | | | X | | |
365 +-------------------------+------+---------+----------+------+------+-------+
366 | k8-nosdn-nofeature-ha | | X | | | | |
367 +-------------------------+------+---------+----------+------+------+-------+
368 | k8-nosdn-stor4nfv-noha | | X | | | | |
369 +-------------------------+------+---------+----------+------+------+-------+
375 Test results are available in:
377 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
379 The reporting pages can be found at:
381 +---------------+----------------------------------------------------------------------------------+
382 | apex | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html |
383 +---------------+----------------------------------------------------------------------------------+
384 | compass | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html |
385 +---------------+----------------------------------------------------------------------------------+
386 | fuel\@x86 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html |
387 +---------------+----------------------------------------------------------------------------------+
388 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html |
389 +---------------+----------------------------------------------------------------------------------+
390 | joid | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html |
391 +---------------+----------------------------------------------------------------------------------+
402 +--------------------+--------------------------------------------------------------------------+
403 | **JIRA REFERENCE** | **DESCRIPTION** |
404 +====================+==========================================================================+
405 | YARDSTICK-1147 | Fix ansible scripts for running in container |
406 +--------------------+--------------------------------------------------------------------------+
407 | YARDSTICK-1157 | Bug Fix: correct the file path to build docker file |
408 +--------------------+--------------------------------------------------------------------------+
409 | YARDSTICK-1276 | Bugfix: docker build failed |
410 +--------------------+--------------------------------------------------------------------------+
411 | YARDSTICK-1280 | Bugfix: uwsgi config file yardstick.ini output error |
412 +--------------------+--------------------------------------------------------------------------+
416 +--------------------+--------------------------------------------------------------------------+
417 | **JIRA REFERENCE** | **DESCRIPTION** |
418 +====================+==========================================================================+
419 | YARDSTICK-1246 | Update pmd/lcore mask for OVS-DPDK context |
420 +--------------------+--------------------------------------------------------------------------+
421 | YARDSTICK-837 | Move tests: unit/network_services/{lib/,collector/,*.py} |
422 +--------------------+--------------------------------------------------------------------------+
423 | YARDSTICK-1144 | Correctly set PYTHONPATH in Dockerfile |
424 +--------------------+--------------------------------------------------------------------------+
425 | YARDSTICK-1205 | Set "cmd2" library to version 0.8.6 |
426 +--------------------+--------------------------------------------------------------------------+
427 | YARDSTICK-1204 | Bump oslo.messaging version to 5.36.0 |
428 +--------------------+--------------------------------------------------------------------------+
429 | YARDSTICK-1210 | Remove __init__ method overriding in HeatContextTestCase |
430 +--------------------+--------------------------------------------------------------------------+
431 | YARDSTICK-1189 | Error when adding SR-IOV interfaces in SR-IOV context |
432 +--------------------+--------------------------------------------------------------------------+
433 | YARDSTICK-1214 | Remove AnsibleCommon class method mock |
434 +--------------------+--------------------------------------------------------------------------+
435 | YARDSTICK-1159 | Add --hwlb options as a command line argument for SampleVNF |
436 +--------------------+--------------------------------------------------------------------------+
437 | YARDSTICK-1203 | Add scale out TCs with availability zone support |
438 +--------------------+--------------------------------------------------------------------------+
439 | YARDSTICK-1167 | Do not start collectd twice when SampleVNF is running on Baremetal |
440 +--------------------+--------------------------------------------------------------------------+
441 | YARDSTICK-1188 | Add "host_name_separator" variable to Context class |
442 +--------------------+--------------------------------------------------------------------------+
443 | YARDSTICK-1112 | MQ startup process refactor |
444 +--------------------+--------------------------------------------------------------------------+
445 | YARDSTICK-1229 | Cleanup BaseMonitor unit tests |
446 +--------------------+--------------------------------------------------------------------------+
447 | - | Configure ACL via static file |
448 +--------------------+--------------------------------------------------------------------------+
449 | YARDSTICK-1191 | Use TRex release v2.41 to support both x86 and aarch64 |
450 +--------------------+--------------------------------------------------------------------------+
451 | YARDSTICK-1106 | Add IxNetwork API Python Binding package |
452 +--------------------+--------------------------------------------------------------------------+
453 | YARDSTICK-1224 | Cleanup TestYardstickNSCli class |
454 +--------------------+--------------------------------------------------------------------------+
455 | YARDSTICK-1225 | Remove print out of logger exception in TestUtils |
456 +--------------------+--------------------------------------------------------------------------+
457 | YARDSTICK-1194 | Add "duration" parameter to test case definition |
458 +--------------------+--------------------------------------------------------------------------+
459 | YARDSTICK-1209 | Remove instantiated contexts in "test_task" |
460 +--------------------+--------------------------------------------------------------------------+
461 | YARDSTICK-1192 | Standalone XML machine type is not longer valid |
462 +--------------------+--------------------------------------------------------------------------+
463 | YARDSTICK-1197 | Refactor RFC2455 TRex traffic profile injection |
464 +--------------------+--------------------------------------------------------------------------+
465 | - | Fix "os.path" mock problems during tests |
466 +--------------------+--------------------------------------------------------------------------+
467 | YARDSTICK-1218 | Refactor "utils.parse_ini_file" testing |
468 +--------------------+--------------------------------------------------------------------------+
469 | YARDSTICK-1179 | Start nginx and uwsgi servicies only in not container mode |
470 +--------------------+--------------------------------------------------------------------------+
471 | YARDSTICK-1177 | Install dependencies: bare-metal, standalone |
472 +--------------------+--------------------------------------------------------------------------+
473 | YARDSTICK-1126 | Migrate install.sh script to ansible |
474 +--------------------+--------------------------------------------------------------------------+
475 | YARDSTICK-1146 | Fix nsb_setup.sh script |
476 +--------------------+--------------------------------------------------------------------------+
477 | YARDSTICK-1247 | NSB setup inventory name changed |
478 +--------------------+--------------------------------------------------------------------------+
479 | YARDSTICK-1116 | Changed IxNextgen library load in IXIA RFC2544 traffic generator call. |
480 +--------------------+--------------------------------------------------------------------------+
481 | - | Corrected scale-up command line arguments |
482 +--------------------+--------------------------------------------------------------------------+
483 | YARDSTICK-878 | OpenStack client replacement |
484 +--------------------+--------------------------------------------------------------------------+
485 | YARDSTICK-1222 | Bugfix: HA kill process recovery has a conflict |
486 +--------------------+--------------------------------------------------------------------------+
487 | YARDSTICK-1139 | Add "os_cloud_config" as a new context flag parameter |
488 +--------------------+--------------------------------------------------------------------------+
489 | YARDSTICK-1255 | Extended Context class with get_physical_nodes functionality |
490 +--------------------+--------------------------------------------------------------------------+
491 | YARDSTICK-1244 | NSB NFVi BNG test fails to run - stops after one step |
492 +--------------------+--------------------------------------------------------------------------+
493 | YARDSTICK-1219 | Decrease Sampling interval |
494 +--------------------+--------------------------------------------------------------------------+
495 | YARDSTICK-1101 | NSB NFVi PROX BNG losing many packets |
496 +--------------------+--------------------------------------------------------------------------+
497 | YARDSTICK-1217 | Fix NSB NfVi support for 25 and 40Gbps |
498 +--------------------+--------------------------------------------------------------------------+
499 | YARDSTICK-1185 | NSB Topology fix for Prox 4 port test case |
500 +--------------------+--------------------------------------------------------------------------+
501 | YARDSTICK-966 | Convert SLA asserts to raises |
502 +--------------------+--------------------------------------------------------------------------+
506 +--------------------+--------------------------------------------------------------------------+
507 | **JIRA REFERENCE** | **DESCRIPTION** |
508 +====================+==========================================================================+
509 | YARDSTICK-995 | Test case spec for SDN Virtual Switch resilience |
510 +--------------------+--------------------------------------------------------------------------+
511 | YARDSTICK-1097 | Add pod.yaml file for APEX installer |
512 +--------------------+--------------------------------------------------------------------------+
513 | YARDSTICK-1122 | Remove unused code in SampleVNF |
514 +--------------------+--------------------------------------------------------------------------+
515 | YARDSTICK-1125 | Update samples/test_suite.yaml |
516 +--------------------+--------------------------------------------------------------------------+
517 | YARDSTICK-1132 | Document for Euphrates test case results |
518 +--------------------+--------------------------------------------------------------------------+
519 | YARDSTICK-1138 | Support Restart Operation |
520 +--------------------+--------------------------------------------------------------------------+
521 | YARDSTICK-1142 | start_service script fails to start openvswitch service in centos distro |
522 +--------------------+--------------------------------------------------------------------------+
523 | YARDSTICK-1165 | Bugfix: openrc api dump should be safe_dump |
524 +--------------------+--------------------------------------------------------------------------+
528 +--------------------+--------------------------------------------------------------------------+
529 | **JIRA REFERENCE** | **DESCRIPTION** |
530 +====================+==========================================================================+
531 | YARDSTICK-831 | tc053 kill haproxy wrong |
532 +--------------------+--------------------------------------------------------------------------+
533 | YARDSTICK-842 | load image fails when there's cirros image exist |
534 +--------------------+--------------------------------------------------------------------------+
535 | YARDSTICK-857 | tc006 failed due to volume attached to different location "/dev/vdc" |
536 +--------------------+--------------------------------------------------------------------------+
537 | YARDSTICK-874 | Specify supported architecture for Ubuntu backports repository |
538 +--------------------+--------------------------------------------------------------------------+
539 | YARDSTICK-875 | Check if multiverse repository is available in Ubuntu |
540 +--------------------+--------------------------------------------------------------------------+
541 | YARDSTICK-893 | Fix proxy env handling and ansible multinode support |
542 +--------------------+--------------------------------------------------------------------------+
543 | YARDSTICK-899 | Variable local_iface_name is read before it is set |
544 +--------------------+--------------------------------------------------------------------------+
545 | YARDSTICK-900 | Section in "upload_yardstick_image.yml" invalid |
546 +--------------------+--------------------------------------------------------------------------+
547 | YARDSTICK-911 | Remove 'inconsistent-return-statements' from Pylint checks |
548 +--------------------+--------------------------------------------------------------------------+
549 | YARDSTICK-989 | Yardstick real-time influxdb KPI reporting regressions |
550 +--------------------+--------------------------------------------------------------------------+
551 | YARDSTICK-994 | NSB set-up build script for baremetal broken |
552 +--------------------+--------------------------------------------------------------------------+
553 | YARDSTICK-996 | Error in address input format in "_ip_range_action_partial" |
554 +--------------------+--------------------------------------------------------------------------+
555 | YARDSTICK-1003 | Prox vnf descriptor cleanup for tg and vnf |
556 +--------------------+--------------------------------------------------------------------------+
557 | YARDSTICK-1006 | Ansible destroy script will fail if vm has already been undefined |
558 +--------------------+--------------------------------------------------------------------------+
559 | YARDSTICK-1012 | constants: fix pylint warnings for OSError |
560 +--------------------+--------------------------------------------------------------------------+
561 | YARDSTICK-1014 | Remove unused args in |
562 | | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile |
563 +--------------------+--------------------------------------------------------------------------+
564 | YARDSTICK-1016 | Allow vm to access outside world through default gateway |
565 +--------------------+--------------------------------------------------------------------------+
566 | YARDSTICK-1019 | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script |
567 +--------------------+--------------------------------------------------------------------------+
568 | YARDSTICK-1021 | NSB: All Sample VNF test cases timeout after 1 hour of execution |
569 +--------------------+--------------------------------------------------------------------------+
570 | YARDSTICK-1036 | Prox: Addition of storage of extra counters for Grafana |
571 +--------------------+--------------------------------------------------------------------------+
572 | YARDSTICK-1038 | Missing file which is described in the operation_conf.yaml |
573 +--------------------+--------------------------------------------------------------------------+
574 | YARDSTICK-1047 | Error in string format in HeatTemplateError message |
575 +--------------------+--------------------------------------------------------------------------+
576 | YARDSTICK-1056 | yardstick report command print error when run test case |
577 +--------------------+--------------------------------------------------------------------------+
578 | YARDSTICK-1059 | Reduce the log level if TRex client is no connected |
579 +--------------------+--------------------------------------------------------------------------+
580 | YARDSTICK-1073 | Error when retrieving "options" section in "scenario" |
581 +--------------------+--------------------------------------------------------------------------+
582 | YARDSTICK-1080 | Running Test Case in Latest Yardstick Docker Image shows Error |
583 +--------------------+--------------------------------------------------------------------------+
584 | YARDSTICK-1082 | tc043,tc055, tc063, tc075, pass wrong node name in the ci scenario yaml |
585 +--------------------+--------------------------------------------------------------------------+
586 | YARDSTICK-1102 | Don't hide exception traceback from Task.start() |
587 +--------------------+--------------------------------------------------------------------------+
588 | YARDSTICK-1107 | bad exception traceback print due to atexit_handler |
589 +--------------------+--------------------------------------------------------------------------+
590 | YARDSTICK-1120 | HA test case tc050 should start monitor before attack |
591 +--------------------+--------------------------------------------------------------------------+
593 Fraser 6.0.0 known restrictions/issues
594 ======================================
596 +-----------+-----------+----------------------------------------------+
597 | Installer | Scenario | Issue |
598 +===========+===========+==============================================+
600 +-----------+-----------+----------------------------------------------+
605 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
607 - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser
609 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
611 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
613 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
615 - Yardstick IRC chanel: #opnfv-yardstick