5 OPNFV Danube release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
11 experimental framework are opensource software, licensed under the terms of the
12 Apache License, Version 2.0.
14 =========================================
15 OPNFV Danube Release Note for Yardstick
16 =========================================
21 .. _Yardstick: https://wiki.opnfv.org/yardstick
23 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
25 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
31 This document describes the release note of Yardstick project.
37 +----------------+--------------------+---------------------------------+
38 | *Date* | *Version* | *Comment* |
40 +----------------+--------------------+---------------------------------+
41 | | 3.1 | Yardstick for Danube release |
43 | | | Note: The 3.1 tag is due to git |
44 | | | tag issue during Danube 3.0 |
47 +----------------+--------------------+---------------------------------+
48 | May 4th, 2017 | 2.0 | Yardstick for Danube release |
50 +----------------+--------------------+---------------------------------+
51 | Mar 31st, 2017 | 1.0 | Yardstick for Danube release |
53 +----------------+--------------------+---------------------------------+
59 The software delivered in the OPNFV Yardstick_ Project, comprising the
60 *Yardstick framework*, the *Yardstick test cases* and the experimental
61 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
64 The *Yardstick* framework is *installer*, *infrastructure* and *application*
69 ======================
71 This Danube release provides *Yardstick* as a framework for NFVI testing
72 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
74 * Documentation generated with Sphinx
80 * Release notes (this document)
84 * Automated Yardstick test suite (daily, weekly)
86 * Jenkins Jobs for OPNFV community labs
88 * Automated Yardstick test results visualization
90 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
93 * Yardstick framework source code
95 * Yardstick test cases yaml files
97 * Yardstick plug-in configration yaml files, plug-in install/remove scripts
99 For Danube release, the *Yardstick framework* is used for the following
102 * OPNFV platform testing - generic test cases to measure the categories:
110 * OPNFV platform network service benchmarking(NSB)
114 * Test cases for the following OPNFV Projects:
128 * virtual Traffic Classifier
130 The *Yardstick framework* is developed in the OPNFV community, by the
131 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
134 .. note:: The test case description template used for the Yardstick test cases
135 is based on the document ETSI-ISG NFV-TST001_; the results report template
136 used for the Yardstick results is based on the IEEE Std 829-2008.
142 +--------------------------------------+--------------------------------------+
143 | **Project** | Yardstick |
145 +--------------------------------------+--------------------------------------+
146 | **Repo/tag** | yardstick/Danube.3.1 |
148 +--------------------------------------+--------------------------------------+
149 | **Yardstick Docker image tag** | Danube.3.1 |
151 +--------------------------------------+--------------------------------------+
152 | **Release designation** | Danube |
154 +--------------------------------------+--------------------------------------+
155 | **Release date** | July 14th, 2017 |
157 +--------------------------------------+--------------------------------------+
158 | **Purpose of the delivery** | OPNFV Danube release 3.0 |
160 +--------------------------------------+--------------------------------------+
169 - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html
171 - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html
174 Software Deliverables
175 ---------------------
178 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: danube.3.1)
183 +---------------------+-------------------------------------------------------+
184 | **Context** | **Description** |
186 +---------------------+-------------------------------------------------------+
187 | *Heat* | Models orchestration using OpenStack Heat |
189 +---------------------+-------------------------------------------------------+
190 | *Node* | Models Baremetal, Controller, Compute |
192 +---------------------+-------------------------------------------------------+
193 | *Standalone* | Models VM running on Non-Managed NFVi |
195 +---------------------+-------------------------------------------------------+
200 +---------------------+-------------------------------------------------------+
201 | **Runner** | **Description** |
203 +---------------------+-------------------------------------------------------+
204 | *Arithmetic* | Steps every run arithmetically according to specified |
207 +---------------------+-------------------------------------------------------+
208 | *Duration* | Runs for a specified period of time |
210 +---------------------+-------------------------------------------------------+
211 | *Iteration* | Runs for a specified number of iterations |
213 +---------------------+-------------------------------------------------------+
214 | *Sequence* | Selects input value to a scenario from an input file |
215 | | and runs all entries sequentially |
217 +---------------------+-------------------------------------------------------+
222 +---------------------+-------------------------------------------------------+
223 | **Category** | **Delivered** |
225 +---------------------+-------------------------------------------------------+
226 | *Availability* | Attacker: |
228 | | * baremetal, process |
232 | | * check host, openstack, process, service |
234 | | * start/stop service |
238 | | * command, process |
240 +---------------------+-------------------------------------------------------+
241 | *Compute* | * cpuload |
247 | | * lmbench_cache |
259 | | * computecapacity |
261 +---------------------+-------------------------------------------------------+
262 | *Networking* | * iperf3 |
276 | | * sfc with tacker |
278 | | * vtc instantion validation |
280 | | * vtc instantion validation with noisy neighbors |
282 | | * vtc throughput |
284 | | * vtc throughput in the presence of noisy neighbors |
286 | | * networkcapacity |
288 | | * netutilization |
292 +---------------------+-------------------------------------------------------+
293 | *Parser* | Tosca2Heat |
295 +---------------------+-------------------------------------------------------+
298 | | storagecapacity |
300 +---------------------+-------------------------------------------------------+
301 | *StorPerf* | storperf |
303 +---------------------+-------------------------------------------------------+
304 | *NSB* | vPE thoughput test case |
306 +---------------------+-------------------------------------------------------+
309 **API to Other Frameworks**
311 +---------------------+-------------------------------------------------------+
312 | **Framework** | **Description** |
314 +---------------------+-------------------------------------------------------+
315 | *ApexLake* | Experimental framework that enables the user to |
316 | | validate NFVI from the perspective of a VNF. |
317 | | A virtual Traffic Classifier is utilized as VNF. |
318 | | Enables experiments with SR-IOV on Compute Node. |
320 +---------------------+-------------------------------------------------------+
323 **Test Results Output**
325 +-----------------------------+-----------------------------------------------+
326 | **Dispatcher** | **Description** |
328 +-----------------------------+-----------------------------------------------+
329 | file | Log to a file. |
331 +-----------------------------+-----------------------------------------------+
332 | http | Post data to html. |
334 +-----------------------------+-----------------------------------------------+
335 | influxdb | Post data to influxDB. |
337 +-----------------------------+-----------------------------------------------+
343 * Generic NFVI test cases
345 * OPNFV_YARDSTICK_TCOO1 - NW Performance
347 * OPNFV_YARDSTICK_TCOO2 - NW Latency
349 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
351 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
353 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
355 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
357 * OPNFV_YARDSTICK_TCO10 - Memory Latency
359 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
361 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
363 * OPNFV_YARDSTICK_TCO14 - Processing Speed
365 * OPNFV_YARDSTICK_TCO24 - CPU Load
367 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
369 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
372 * OPNFV_YARDSTICK_TCO42 - Network Performance
374 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
376 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
378 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
380 * OPNFV_YARDSTICK_TCO61 - Network Utilization
382 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
384 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
386 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
389 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
391 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
394 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
396 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
398 * OPNFV_YARDSTICK_TCO76 - Network frame error rate
400 * Test Cases for OPNFV HA Project:
402 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
404 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
406 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
409 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
411 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
414 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
417 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
420 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
423 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
426 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
429 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
432 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
434 * Test Case for OPNFV IPv6 Project:
436 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
438 * Test Case for OPNFV KVM Project:
440 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
442 * Test Case for OPNFV Parser Project:
444 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
446 * Test Case for OPNFV StorPerf Project:
448 * OPNFV_YARDSTICK_TCO74 - Storperf
450 * Test Cases for Virtual Traffic Classifier:
452 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
456 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
458 * Benchmarking in presence of noisy neighbors Test:
460 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
462 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
463 presence of noisy neighbors Test
469 Module Version Changes
470 ----------------------
472 This is the fourth tracked release of Yardstick. It is based on following
482 Document Version Changes
483 ------------------------
485 This is the fourth tracked version of the Yardstick framework in OPNFV.
486 It includes the following documentation updates:
488 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
489 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
490 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
492 - Yardstick Developer Guide
494 - Yardstick Release Notes for Yardstick: this document
500 - Yardstick RESTful API support
502 - Introduce Network service benchmarking
504 - Introduce stress testing with Bottlenecks team
506 - Yardstick framework improvement:
508 - Parellel test cases execution support
510 - yardstick report CLI
512 - Node context support openstack configuration via Ansible
522 For Danube 3.0, Yardstick was tested on the following scenarios:
524 +-------------------------+---------+---------+---------+---------+
525 | Scenario | Apex | Compass | Fuel | Joid |
526 +=========================+=========+=========+=========+=========+
527 | os-nosdn-nofeature-noha | | | X | X |
528 +-------------------------+---------+---------+---------+---------+
529 | os-nosdn-nofeature-ha | X | X | X | X |
530 +-------------------------+---------+---------+---------+---------+
531 | os-odl_l2-nofeature-ha | | X | X | X |
532 +-------------------------+---------+---------+---------+---------+
533 | os-odl_l2-nofeature-noha| | | X | |
534 +-------------------------+---------+---------+---------+---------+
535 | os-odl_l3-nofeature-ha | X | X | X | |
536 +-------------------------+---------+---------+---------+---------+
537 | os-odl_l3-nofeature-noha| | | X | |
538 +-------------------------+---------+---------+---------+---------+
539 | os-onos-sfc-ha | | | | |
540 +-------------------------+---------+---------+---------+---------+
541 | os-onos-nofeature-ha | | X | | X |
542 +-------------------------+---------+---------+---------+---------+
543 | os-onos-nofeature-noha | | | | |
544 +-------------------------+---------+---------+---------+---------+
545 | os-odl_l2-sfc-ha | | | X | |
546 +-------------------------+---------+---------+---------+---------+
547 | os-odl_l2-sfc-noha | | | X | |
548 +-------------------------+---------+---------+---------+---------+
549 | os-odl_l2-bgpvpn-ha | X | | X | |
550 +-------------------------+---------+---------+---------+---------+
551 | os-odl_l2-bgpvpn-noha | | | X | |
552 +-------------------------+---------+---------+---------+---------+
553 | os-nosdn-kvm-ha | X | | X | |
554 +-------------------------+---------+---------+---------+---------+
555 | os-nosdn-kvm-noha | | | X | |
556 +-------------------------+---------+---------+---------+---------+
557 | os-nosdn-ovs-ha | | | X | |
558 +-------------------------+---------+---------+---------+---------+
559 | os-nosdn-ovs-noha | | | X | |
560 +-------------------------+---------+---------+---------+---------+
561 | os-ocl-nofeature-ha | | X | | |
562 +-------------------------+---------+---------+---------+---------+
563 | os-nosdn-lxd-ha | | | | X |
564 +-------------------------+---------+---------+---------+---------+
565 | os-nosdn-lxd-noha | | | | X |
566 +-------------------------+---------+---------+---------+---------+
567 | os-nosdn-fdio-ha | X | | | |
568 +-------------------------+---------+---------+---------+---------+
569 | os-odl_l2-fdio-noha | X | | | |
570 +-------------------------+---------+---------+---------+---------+
571 | os-odl-gluon-noha | X | | | |
572 +-------------------------+---------+---------+---------+---------+
573 | os-nosdn-openo-ha | | X | | |
574 +-------------------------+---------+---------+---------+---------+
575 | os-nosdn-kvm_ovs_dpdk | | | X | |
577 +-------------------------+---------+---------+---------+---------+
578 | os-nosdn-kvm_ovs_dpdk-ha| | | X | |
579 +-------------------------+---------+---------+---------+---------+
580 | os-nosdn-kvm_ovs_dpdk | | | X | |
582 +-------------------------+---------+---------+---------+---------+
583 | os-nosdn-kvm_ovs_dpdk | | | X | |
584 | _bar-noha | | | | |
585 +-------------------------+---------+---------+---------+---------+
586 | opnfv_os-ovn-nofeature- | X | | | |
587 | noha_daily | | | | |
588 +-------------------------+---------+---------+---------+---------+
593 Test results are available in:
595 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
597 The reporting pages can be found at:
599 * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html
600 * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
601 * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html
602 * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html
608 - Floating IP not supported in bgpvpn scenario
610 - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario
612 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
613 which were not fully verified, are listed in the OPNFV installer's release
622 +----------------------------+------------------------------------------------+
623 | **JIRA REFERENCE** | **DESCRIPTION** |
625 +----------------------------+------------------------------------------------+
626 | JIRA: YARDSTICK-714 | Add yardstick env influxdb/grafana command for |
628 +----------------------------+------------------------------------------------+
629 | JIRA: YARDSTICK-655 | Monitor command in tc019 may not show the |
630 | | real nova-api service status |
631 +----------------------------+------------------------------------------------+
632 | JIRA: YARDSTICK-397 | HA testing framework improvement |
634 +----------------------------+------------------------------------------------+
635 | JIRA: YARDSTICK-660 | Improve monitor_process pass criteria |
637 +----------------------------+------------------------------------------------+
638 | JIRA: YARDSTICK-657 | HA monitor_multi bug, |
639 | | KeyError: 'max_outage_time' |
640 +----------------------------+------------------------------------------------+
641 | JIRA: YARDSTICK-647 | TC025 fault_type value is wrong when using |
642 | | baremetal pod scripts |
643 +----------------------------+------------------------------------------------+
644 | JIRA: YARDSTICK-659 | Terminate openstack service process using kill |
645 | | command in HA test cases |
646 +----------------------------+------------------------------------------------+
647 | JIRA: ARMBAND-275 | Yardstick TC005 fails with |
648 | | "Cannot map zero-fill pages" error |
649 +----------------------------+------------------------------------------------+
650 | JIRA: YARDSTICK-561 | Bugfix: AttributeError: 'dict' object has no |
651 | | attribute 'split' if run sample/ping-hot.yaml |
652 +----------------------------+------------------------------------------------+
653 | JIRA: ARMBAND-268 | ERROR No JSON object could be decoded from |
654 | | LMBENCH in TC010 |
655 +----------------------------+------------------------------------------------+
656 | JIRA: YARDSTICK-680 | storperf test case tc074 do not get results |
658 +----------------------------+------------------------------------------------+
662 +----------------------------+------------------------------------------------+
663 | **JIRA REFERENCE** | **DESCRIPTION** |
665 +----------------------------+------------------------------------------------+
666 | JIRA: YARDSTICK-608 | Set work directory in Yardstick container |
668 +----------------------------+------------------------------------------------+
669 | JIRA: YARDSTICK-616 | Bugfix: https support should adapt insecure |
671 +----------------------------+------------------------------------------------+
672 | JIRA: YARDSTICK-620 | Yardstick virtualenv support |
674 +----------------------------+------------------------------------------------+
675 | JIRA: YARDSTICK-621 | Bugfix: fix query job status in TC074 |
677 +----------------------------+------------------------------------------------+
678 | JIRA: YARDSTICK-622 | Bugfix: take test case modification into |
679 | | effect in load_images.sh |
680 +----------------------------+------------------------------------------------+
681 | JIRA: YARDSTICK-623 | change openrc file path to |
682 | | /etc/yardstick/openstack.creds |
683 +----------------------------+------------------------------------------------+
684 | JIRA: YARDSTICK-624 | Add opnfv_os-ovn-nofeature-noha_daily test |
686 +----------------------------+------------------------------------------------+
687 | JIRA: YARDSTICK-628 | Bugfix: Make tc019 and tc025 accept |
688 | | --task-args options |
689 +----------------------------+------------------------------------------------+
690 | JIRA: YARDSTICK-629 | Bugfix: yardstick env prepare cmd do not |
691 | | support other installer |
692 +----------------------------+------------------------------------------------+
693 | JIRA: YARDSTICK-632 | Bugfix: KeyError when using http dispatcher |
695 +----------------------------+------------------------------------------------+
696 | JIRA: YARDSTICK-633 | Bugfix: Environment Compatibility Issues in HA |
698 +----------------------------+------------------------------------------------+
699 | JIRA: YARDSTICK-634 | fix ha issue when run tc050~tc054 in ci |
701 +----------------------------+------------------------------------------------+
702 | JIRA: YARDSTICK-635 | Bugfix: Local Openstack Operation in HA test |
704 +----------------------------+------------------------------------------------+
708 +----------------------------+------------------------------------------------+
709 | **JIRA REFERENCE** | **DESCRIPTION** |
711 +----------------------------+------------------------------------------------+
712 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
714 +----------------------------+------------------------------------------------+
715 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
717 +----------------------------+------------------------------------------------+
720 Danube 3.1 known restrictions/issues
721 ====================================
722 +-----------+-----------+----------------------------------------------+
723 | Installer | Scenario | Issue |
724 +===========+===========+==============================================+
725 | any | *-bgpvpn | Floating ips not supported. Some Test cases |
726 | | | related to floating ips are excluded. |
727 +-----------+-----------+----------------------------------------------+
728 | any | odl_l3-* | Some test cases related to using floating IP |
729 | | | addresses fail because of a known ODL bug. |
731 +-----------+-----------+----------------------------------------------+
732 | compass | odl_l2-* | In some test cases, VM instance will failed |
733 | | | raising network interfaces. |
735 +-----------+-----------+----------------------------------------------+
741 +----------------------------+------------------------------------------------+
742 | **JIRA REFERENCE** | **DESCRIPTION** |
744 +----------------------------+------------------------------------------------+
745 | JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 |
747 +----------------------------+------------------------------------------------+
748 | JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
749 | | (which actually pingable) |
750 +----------------------------+------------------------------------------------+
751 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
753 +----------------------------+------------------------------------------------+
759 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
761 - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning
763 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
765 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
767 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
769 - Yardstick IRC chanel: #opnfv-yardstick