5 OPNFV Danube release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
11 experimental framework are opensource software, licensed under the terms of the
12 Apache License, Version 2.0.
14 =========================================
15 OPNFV Danube Release Note for Yardstick
16 =========================================
21 .. _Yardstick: https://wiki.opnfv.org/yardstick
23 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
25 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
31 This document describes the release note of Yardstick project.
37 +----------------+--------------------+---------------------------------+
38 | *Date* | *Version* | *Comment* |
40 +----------------+--------------------+---------------------------------+
41 | | 3.0 | Yardstick for Danube release |
43 +----------------+--------------------+---------------------------------+
44 | May 4th, 2017 | 2.0 | Yardstick for Danube release |
46 +----------------+--------------------+---------------------------------+
47 | Mar 31st, 2017 | 1.0 | Yardstick for Danube release |
49 +----------------+--------------------+---------------------------------+
55 The software delivered in the OPNFV Yardstick_ Project, comprising the
56 *Yardstick framework*, the *Yardstick test cases* and the experimental
57 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
60 The *Yardstick* framework is *installer*, *infrastructure* and *application*
65 ======================
67 This Danube release provides *Yardstick* as a framework for NFVI testing
68 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
70 * Documentation generated with Sphinx
76 * Release notes (this document)
80 * Automated Yardstick test suite (daily, weekly)
82 * Jenkins Jobs for OPNFV community labs
84 * Automated Yardstick test results visualization
86 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
89 * Yardstick framework source code
91 * Yardstick test cases yaml files
93 * Yardstick plug-in configration yaml files, plug-in install/remove scripts
95 For Danube release, the *Yardstick framework* is used for the following
98 * OPNFV platform testing - generic test cases to measure the categories:
106 * OPNFV platform network service benchmarking(NSB)
110 * Test cases for the following OPNFV Projects:
124 * virtual Traffic Classifier
126 The *Yardstick framework* is developed in the OPNFV community, by the
127 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
130 .. note:: The test case description template used for the Yardstick test cases
131 is based on the document ETSI-ISG NFV-TST001_; the results report template
132 used for the Yardstick results is based on the IEEE Std 829-2008.
138 +--------------------------------------+--------------------------------------+
139 | **Project** | Yardstick |
141 +--------------------------------------+--------------------------------------+
142 | **Repo/tag** | yardstick/Danube.2.0 |
144 +--------------------------------------+--------------------------------------+
145 | **Yardstick Docker image tag** | Danube.2.0 |
147 +--------------------------------------+--------------------------------------+
148 | **Release designation** | Danube |
150 +--------------------------------------+--------------------------------------+
151 | **Release date** | May 4th, 2017 |
153 +--------------------------------------+--------------------------------------+
154 | **Purpose of the delivery** | OPNFV Danube release 2.0 |
156 +--------------------------------------+--------------------------------------+
165 - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html
167 - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html
170 Software Deliverables
171 ---------------------
174 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: danube.2.0)
179 +---------------------+-------------------------------------------------------+
180 | **Context** | **Description** |
182 +---------------------+-------------------------------------------------------+
183 | *Heat* | Models orchestration using OpenStack Heat |
185 +---------------------+-------------------------------------------------------+
186 | *Node* | Models Baremetal, Controller, Compute |
188 +---------------------+-------------------------------------------------------+
189 | *Standalone* | Models VM running on Non-Managed NFVi |
191 +---------------------+-------------------------------------------------------+
196 +---------------------+-------------------------------------------------------+
197 | **Runner** | **Description** |
199 +---------------------+-------------------------------------------------------+
200 | *Arithmetic* | Steps every run arithmetically according to specified |
203 +---------------------+-------------------------------------------------------+
204 | *Duration* | Runs for a specified period of time |
206 +---------------------+-------------------------------------------------------+
207 | *Iteration* | Runs for a specified number of iterations |
209 +---------------------+-------------------------------------------------------+
210 | *Sequence* | Selects input value to a scenario from an input file |
211 | | and runs all entries sequentially |
213 +---------------------+-------------------------------------------------------+
218 +---------------------+-------------------------------------------------------+
219 | **Category** | **Delivered** |
221 +---------------------+-------------------------------------------------------+
222 | *Availability* | Attacker: |
224 | | * baremetal, process |
228 | | * check host, openstack, process, service |
230 | | * start/stop service |
234 | | * command, process |
236 +---------------------+-------------------------------------------------------+
237 | *Compute* | * cpuload |
243 | | * lmbench_cache |
255 | | * computecapacity |
257 +---------------------+-------------------------------------------------------+
258 | *Networking* | * iperf3 |
272 | | * sfc with tacker |
274 | | * vtc instantion validation |
276 | | * vtc instantion validation with noisy neighbors |
278 | | * vtc throughput |
280 | | * vtc throughput in the presence of noisy neighbors |
282 | | * networkcapacity |
284 | | * netutilization |
288 +---------------------+-------------------------------------------------------+
289 | *Parser* | Tosca2Heat |
291 +---------------------+-------------------------------------------------------+
294 | | storagecapacity |
296 +---------------------+-------------------------------------------------------+
297 | *StorPerf* | storperf |
299 +---------------------+-------------------------------------------------------+
300 | *NSB* | vPE thoughput test case |
302 +---------------------+-------------------------------------------------------+
305 **API to Other Frameworks**
307 +---------------------+-------------------------------------------------------+
308 | **Framework** | **Description** |
310 +---------------------+-------------------------------------------------------+
311 | *ApexLake* | Experimental framework that enables the user to |
312 | | validate NFVI from the perspective of a VNF. |
313 | | A virtual Traffic Classifier is utilized as VNF. |
314 | | Enables experiments with SR-IOV on Compute Node. |
316 +---------------------+-------------------------------------------------------+
319 **Test Results Output**
321 +-----------------------------+-----------------------------------------------+
322 | **Dispatcher** | **Description** |
324 +-----------------------------+-----------------------------------------------+
325 | file | Log to a file. |
327 +-----------------------------+-----------------------------------------------+
328 | http | Post data to html. |
330 +-----------------------------+-----------------------------------------------+
331 | influxdb | Post data to influxDB. |
333 +-----------------------------+-----------------------------------------------+
339 * Generic NFVI test cases
341 * OPNFV_YARDSTICK_TCOO1 - NW Performance
343 * OPNFV_YARDSTICK_TCOO2 - NW Latency
345 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
347 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
349 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
351 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
353 * OPNFV_YARDSTICK_TCO10 - Memory Latency
355 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
357 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
359 * OPNFV_YARDSTICK_TCO14 - Processing Speed
361 * OPNFV_YARDSTICK_TCO24 - CPU Load
363 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
365 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
368 * OPNFV_YARDSTICK_TCO42 - Network Performance
370 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
372 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
374 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
376 * OPNFV_YARDSTICK_TCO61 - Network Utilization
378 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
380 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
382 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
385 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
387 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
390 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
392 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
394 * OPNFV_YARDSTICK_TCO76 - Network frame error rate
396 * Test Cases for OPNFV HA Project:
398 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
400 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
402 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
405 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
407 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
410 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
413 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
416 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
419 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
422 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
425 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
428 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
430 * Test Case for OPNFV IPv6 Project:
432 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
434 * Test Case for OPNFV KVM Project:
436 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
438 * Test Case for OPNFV Parser Project:
440 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
442 * Test Case for OPNFV StorPerf Project:
444 * OPNFV_YARDSTICK_TCO74 - Storperf
446 * Test Cases for Virtual Traffic Classifier:
448 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
452 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
454 * Benchmarking in presence of noisy neighbors Test:
456 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
458 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
459 presence of noisy neighbors Test
465 Module Version Changes
466 ----------------------
468 This is the fourth tracked release of Yardstick. It is based on following
478 Document Version Changes
479 ------------------------
481 This is the fourth tracked version of the Yardstick framework in OPNFV.
482 It includes the following documentation updates:
484 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
485 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
486 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
488 - Yardstick Developer Guide
490 - Yardstick Release Notes for Yardstick: this document
496 - Yardstick RESTful API support
498 - Introduce Network service benchmarking
500 - Introduce stress testing with Bottlenecks team
502 - Yardstick framework improvement:
504 - Parellel test cases execution support
506 - yardstick report CLI
508 - Node context support openstack configuration via Ansible
518 For Danube 2.0, Yardstick was tested on the following scenarios:
520 +-------------------------+---------+---------+---------+---------+
521 | Scenario | Apex | Compass | Fuel | Joid |
522 +=========================+=========+=========+=========+=========+
523 | os-nosdn-nofeature-noha | | | X | X |
524 +-------------------------+---------+---------+---------+---------+
525 | os-nosdn-nofeature-ha | X | X | X | X |
526 +-------------------------+---------+---------+---------+---------+
527 | os-odl_l2-nofeature-ha | | X | X | X |
528 +-------------------------+---------+---------+---------+---------+
529 | os-odl_l2-nofeature-noha| | | X | |
530 +-------------------------+---------+---------+---------+---------+
531 | os-odl_l3-nofeature-ha | X | X | X | |
532 +-------------------------+---------+---------+---------+---------+
533 | os-odl_l3-nofeature-noha| | | X | |
534 +-------------------------+---------+---------+---------+---------+
535 | os-onos-sfc-ha | | | | |
536 +-------------------------+---------+---------+---------+---------+
537 | os-onos-nofeature-ha | | X | | X |
538 +-------------------------+---------+---------+---------+---------+
539 | os-onos-nofeature-noha | | | | |
540 +-------------------------+---------+---------+---------+---------+
541 | os-odl_l2-sfc-ha | | | X | |
542 +-------------------------+---------+---------+---------+---------+
543 | os-odl_l2-sfc-noha | | | X | |
544 +-------------------------+---------+---------+---------+---------+
545 | os-odl_l2-bgpvpn-ha | X | | X | |
546 +-------------------------+---------+---------+---------+---------+
547 | os-odl_l2-bgpvpn-noha | | | X | |
548 +-------------------------+---------+---------+---------+---------+
549 | os-nosdn-kvm-ha | X | | X | |
550 +-------------------------+---------+---------+---------+---------+
551 | os-nosdn-kvm-noha | | | X | |
552 +-------------------------+---------+---------+---------+---------+
553 | os-nosdn-ovs-ha | | | X | |
554 +-------------------------+---------+---------+---------+---------+
555 | os-nosdn-ovs-noha | | | X | |
556 +-------------------------+---------+---------+---------+---------+
557 | os-ocl-nofeature-ha | | X | | |
558 +-------------------------+---------+---------+---------+---------+
559 | os-nosdn-lxd-ha | | | | X |
560 +-------------------------+---------+---------+---------+---------+
561 | os-nosdn-lxd-noha | | | | X |
562 +-------------------------+---------+---------+---------+---------+
563 | os-nosdn-fdio-ha | X | | | |
564 +-------------------------+---------+---------+---------+---------+
565 | os-odl_l2-fdio-noha | X | | | |
566 +-------------------------+---------+---------+---------+---------+
567 | os-odl-gluon-noha | X | | | |
568 +-------------------------+---------+---------+---------+---------+
569 | os-nosdn-openo-ha | | X | | |
570 +-------------------------+---------+---------+---------+---------+
571 | os-nosdn-kvm_ovs_dpdk | | | X | |
573 +-------------------------+---------+---------+---------+---------+
574 | os-nosdn-kvm_ovs_dpdk-ha| | | X | |
575 +-------------------------+---------+---------+---------+---------+
576 | os-nosdn-kvm_ovs_dpdk | | | X | |
578 +-------------------------+---------+---------+---------+---------+
579 | os-nosdn-kvm_ovs_dpdk | | | X | |
580 | _bar-noha | | | | |
581 +-------------------------+---------+---------+---------+---------+
582 | opnfv_os-ovn-nofeature- | X | | | |
583 | noha_daily | | | | |
584 +-------------------------+---------+---------+---------+---------+
589 Test results are available in:
591 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
593 The reporting pages can be found at:
595 * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html
596 * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
597 * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html
598 * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html
604 - Floating IP not supported in bgpvpn scenario
606 - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario
608 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
609 which were not fully verified, are listed in the OPNFV installer's release
618 +----------------------------+------------------------------------------------+
619 | **JIRA REFERENCE** | **SLOGAN** |
621 +----------------------------+------------------------------------------------+
622 | JIRA: YARDSTICK-608 | Set work directory in Yardstick container |
624 +----------------------------+------------------------------------------------+
625 | JIRA: YARDSTICK-616 | Bugfix: https support should adapt insecure |
627 +----------------------------+------------------------------------------------+
628 | JIRA: YARDSTICK-620 | Yardstick virtualenv support |
630 +----------------------------+------------------------------------------------+
631 | JIRA: YARDSTICK-621 | Bugfix: fix query job status in TC074 |
633 +----------------------------+------------------------------------------------+
634 | JIRA: YARDSTICK-622 | Bugfix: take test case modification into |
635 | | effect in load_images.sh |
636 +----------------------------+------------------------------------------------+
637 | JIRA: YARDSTICK-623 | change openrc file path to |
638 | | /etc/yardstick/openstack.creds |
639 +----------------------------+------------------------------------------------+
640 | JIRA: YARDSTICK-624 | Add opnfv_os-ovn-nofeature-noha_daily test |
642 +----------------------------+------------------------------------------------+
643 | JIRA: YARDSTICK-628 | Bugfix: Make tc019 and tc025 accept |
644 | | --task-args options |
645 +----------------------------+------------------------------------------------+
646 | JIRA: YARDSTICK-629 | Bugfix: yardstick env prepare cmd do not |
647 | | support other installer |
648 +----------------------------+------------------------------------------------+
649 | JIRA: YARDSTICK-632 | Bugfix: KeyError when using http dispatcher |
651 +----------------------------+------------------------------------------------+
652 | JIRA: YARDSTICK-633 | Bugfix: Environment Compatibility Issues in HA |
654 +----------------------------+------------------------------------------------+
655 | JIRA: YARDSTICK-634 | fix ha issue when run tc050~tc054 in ci |
657 +----------------------------+------------------------------------------------+
658 | JIRA: YARDSTICK-635 | Bugfix: Local Openstack Operation in HA test |
660 +----------------------------+------------------------------------------------+
664 +----------------------------+------------------------------------------------+
665 | **JIRA REFERENCE** | **SLOGAN** |
667 +----------------------------+------------------------------------------------+
668 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
670 +----------------------------+------------------------------------------------+
671 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
673 +----------------------------+------------------------------------------------+
676 Danube 2.0 known restrictions/issues
677 ====================================
678 +-----------+-----------+----------------------------------------------+
679 | Installer | Scenario | Issue |
680 +===========+===========+==============================================+
681 | any | *-bgpvpn | Floating ips not supported. Some Test cases |
682 | | | related to floating ips are excluded. |
683 +-----------+-----------+----------------------------------------------+
684 | any | odl_l3-* | Some test cases related to using floating IP |
685 | | | addresses fail because of a known ODL bug. |
687 +-----------+-----------+----------------------------------------------+
688 | compass | odl_l2-* | In some test cases, VM instance will failed |
689 | | | raising network interfaces. |
691 +-----------+-----------+----------------------------------------------+
697 +----------------------------+------------------------------------------------+
698 | **JIRA REFERENCE** | **SLOGAN** |
700 +----------------------------+------------------------------------------------+
701 | JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 |
703 +----------------------------+------------------------------------------------+
704 | JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
705 | | (which actually pingable) |
706 +----------------------------+------------------------------------------------+
707 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
709 +----------------------------+------------------------------------------------+
715 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
717 - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning
719 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
721 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
723 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
725 - Yardstick IRC chanel: #opnfv-yardstick