5 OPNFV Danube release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* are opensource software,
11 licensed under the terms of the Apache License, Version 2.0.
13 =========================================
14 OPNFV Danube Release Note for Yardstick
15 =========================================
20 .. _Yardstick: https://wiki.opnfv.org/yardstick
22 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
24 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
30 This document describes the release note of Yardstick project.
36 +----------------+--------------------+---------------------------------+
37 | *Date* | *Version* | *Comment* |
39 +----------------+--------------------+---------------------------------+
40 | | 3.2 | Yardstick for Danube release |
42 | | | Note: The 3.2 tag is due to a |
43 | | | code issue during Danube 3.1 |
46 +----------------+--------------------+---------------------------------+
47 | May 4th, 2017 | 2.0 | Yardstick for Danube release |
49 +----------------+--------------------+---------------------------------+
50 | Mar 31st, 2017 | 1.0 | Yardstick for Danube release |
52 +----------------+--------------------+---------------------------------+
58 The software delivered in the OPNFV Yardstick_ Project, comprising the
59 *Yardstick framework*, the *Yardstick test cases* and the experimental
60 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
63 The *Yardstick* framework is *installer*, *infrastructure* and *application*
68 ======================
70 This Danube release provides *Yardstick* as a framework for NFVI testing
71 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
73 * Documentation generated with Sphinx
79 * Release notes (this document)
83 * Automated Yardstick test suite (daily, weekly)
85 * Jenkins Jobs for OPNFV community labs
87 * Automated Yardstick test results visualization
89 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
92 * Yardstick framework source code
94 * Yardstick test cases yaml files
96 * Yardstick plug-in configration yaml files, plug-in install/remove scripts
98 For Danube release, the *Yardstick framework* is used for the following
101 * OPNFV platform testing - generic test cases to measure the categories:
109 * OPNFV platform network service benchmarking(NSB)
113 * Test cases for the following OPNFV Projects:
127 * virtual Traffic Classifier
129 The *Yardstick framework* is developed in the OPNFV community, by the
130 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
133 .. note:: The test case description template used for the Yardstick test cases
134 is based on the document ETSI-ISG NFV-TST001_; the results report template
135 used for the Yardstick results is based on the IEEE Std 829-2008.
141 +--------------------------------------+--------------------------------------+
142 | **Project** | Yardstick |
144 +--------------------------------------+--------------------------------------+
145 | **Repo/tag** | yardstick/Danube.3.2 |
147 +--------------------------------------+--------------------------------------+
148 | **Yardstick Docker image tag** | Danube.3.2 |
150 +--------------------------------------+--------------------------------------+
151 | **Release designation** | Danube |
153 +--------------------------------------+--------------------------------------+
154 | **Release date** | August 15th, 2017 |
156 +--------------------------------------+--------------------------------------+
157 | **Purpose of the delivery** | OPNFV Danube release 3.0 |
159 +--------------------------------------+--------------------------------------+
168 - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html
170 - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html
173 Software Deliverables
174 ---------------------
177 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: danube.3.2)
182 +---------------------+-------------------------------------------------------+
183 | **Context** | **Description** |
185 +---------------------+-------------------------------------------------------+
186 | *Heat* | Models orchestration using OpenStack Heat |
188 +---------------------+-------------------------------------------------------+
189 | *Node* | Models Baremetal, Controller, Compute |
191 +---------------------+-------------------------------------------------------+
192 | *Standalone* | Models VM running on Non-Managed NFVi |
194 +---------------------+-------------------------------------------------------+
199 +---------------------+-------------------------------------------------------+
200 | **Runner** | **Description** |
202 +---------------------+-------------------------------------------------------+
203 | *Arithmetic* | Steps every run arithmetically according to specified |
206 +---------------------+-------------------------------------------------------+
207 | *Duration* | Runs for a specified period of time |
209 +---------------------+-------------------------------------------------------+
210 | *Iteration* | Runs for a specified number of iterations |
212 +---------------------+-------------------------------------------------------+
213 | *Sequence* | Selects input value to a scenario from an input file |
214 | | and runs all entries sequentially |
216 +---------------------+-------------------------------------------------------+
221 +---------------------+-------------------------------------------------------+
222 | **Category** | **Delivered** |
224 +---------------------+-------------------------------------------------------+
225 | *Availability* | Attacker: |
227 | | * baremetal, process |
231 | | * check host, openstack, process, service |
233 | | * start/stop service |
237 | | * command, process |
239 +---------------------+-------------------------------------------------------+
240 | *Compute* | * cpuload |
246 | | * lmbench_cache |
258 | | * computecapacity |
260 +---------------------+-------------------------------------------------------+
261 | *Networking* | * iperf3 |
275 | | * sfc with tacker |
277 | | * vtc instantion validation |
279 | | * vtc instantion validation with noisy neighbors |
281 | | * vtc throughput |
283 | | * vtc throughput in the presence of noisy neighbors |
285 | | * networkcapacity |
287 | | * netutilization |
291 +---------------------+-------------------------------------------------------+
292 | *Parser* | Tosca2Heat |
294 +---------------------+-------------------------------------------------------+
297 | | storagecapacity |
299 +---------------------+-------------------------------------------------------+
300 | *StorPerf* | storperf |
302 +---------------------+-------------------------------------------------------+
303 | *NSB* | vPE thoughput test case |
305 +---------------------+-------------------------------------------------------+
308 **Test Results Output**
310 +-----------------------------+-----------------------------------------------+
311 | **Dispatcher** | **Description** |
313 +-----------------------------+-----------------------------------------------+
314 | file | Log to a file. |
316 +-----------------------------+-----------------------------------------------+
317 | http | Post data to html. |
319 +-----------------------------+-----------------------------------------------+
320 | influxdb | Post data to influxDB. |
322 +-----------------------------+-----------------------------------------------+
328 * Generic NFVI test cases
330 * OPNFV_YARDSTICK_TCOO1 - NW Performance
332 * OPNFV_YARDSTICK_TCOO2 - NW Latency
334 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
336 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
338 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
340 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
342 * OPNFV_YARDSTICK_TCO10 - Memory Latency
344 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
346 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
348 * OPNFV_YARDSTICK_TCO14 - Processing Speed
350 * OPNFV_YARDSTICK_TCO24 - CPU Load
352 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
354 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
357 * OPNFV_YARDSTICK_TCO42 - Network Performance
359 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
361 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
363 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
365 * OPNFV_YARDSTICK_TCO61 - Network Utilization
367 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
369 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
371 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
374 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
376 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
379 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
381 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
383 * OPNFV_YARDSTICK_TCO76 - Network frame error rate
385 * Test Cases for OPNFV HA Project:
387 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
389 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
391 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
394 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
396 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
399 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
402 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
405 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
408 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
411 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
414 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
417 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
419 * Test Case for OPNFV IPv6 Project:
421 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
423 * Test Case for OPNFV KVM Project:
425 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
427 * Test Case for OPNFV Parser Project:
429 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
431 * Test Case for OPNFV StorPerf Project:
433 * OPNFV_YARDSTICK_TCO74 - Storperf
435 * Test Cases for Virtual Traffic Classifier:
437 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
441 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
443 * Benchmarking in presence of noisy neighbors Test:
445 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
447 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
448 presence of noisy neighbors Test
454 Module Version Changes
455 ----------------------
457 This is the fourth tracked release of Yardstick. It is based on following
467 Document Version Changes
468 ------------------------
470 This is the fourth tracked version of the Yardstick framework in OPNFV.
471 It includes the following documentation updates:
473 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
474 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
475 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
477 - Yardstick Developer Guide
479 - Yardstick Release Notes for Yardstick: this document
485 - Yardstick RESTful API support
487 - Introduce Network service benchmarking
489 - Introduce stress testing with Bottlenecks team
491 - Yardstick framework improvement:
493 - Parellel test cases execution support
495 - yardstick report CLI
497 - Node context support openstack configuration via Ansible
507 For Danube 3.0, Yardstick was tested on the following scenarios:
509 +-------------------------+---------+---------+---------+---------+
510 | Scenario | Apex | Compass | Fuel | Joid |
511 +=========================+=========+=========+=========+=========+
512 | os-nosdn-nofeature-noha | | | X | X |
513 +-------------------------+---------+---------+---------+---------+
514 | os-nosdn-nofeature-ha | X | X | X | X |
515 +-------------------------+---------+---------+---------+---------+
516 | os-odl_l2-nofeature-ha | | X | X | X |
517 +-------------------------+---------+---------+---------+---------+
518 | os-odl_l2-nofeature-noha| | | X | |
519 +-------------------------+---------+---------+---------+---------+
520 | os-odl_l3-nofeature-ha | X | X | X | |
521 +-------------------------+---------+---------+---------+---------+
522 | os-odl_l3-nofeature-noha| | | X | |
523 +-------------------------+---------+---------+---------+---------+
524 | os-onos-sfc-ha | | | | |
525 +-------------------------+---------+---------+---------+---------+
526 | os-onos-nofeature-ha | | X | | X |
527 +-------------------------+---------+---------+---------+---------+
528 | os-onos-nofeature-noha | | | | |
529 +-------------------------+---------+---------+---------+---------+
530 | os-odl_l2-sfc-ha | | | X | |
531 +-------------------------+---------+---------+---------+---------+
532 | os-odl_l2-sfc-noha | | | X | |
533 +-------------------------+---------+---------+---------+---------+
534 | os-odl_l2-bgpvpn-ha | X | | X | |
535 +-------------------------+---------+---------+---------+---------+
536 | os-odl_l2-bgpvpn-noha | | | X | |
537 +-------------------------+---------+---------+---------+---------+
538 | os-nosdn-kvm-ha | X | | X | |
539 +-------------------------+---------+---------+---------+---------+
540 | os-nosdn-kvm-noha | | | X | |
541 +-------------------------+---------+---------+---------+---------+
542 | os-nosdn-ovs-ha | | | X | |
543 +-------------------------+---------+---------+---------+---------+
544 | os-nosdn-ovs-noha | | | X | |
545 +-------------------------+---------+---------+---------+---------+
546 | os-ocl-nofeature-ha | | X | | |
547 +-------------------------+---------+---------+---------+---------+
548 | os-nosdn-lxd-ha | | | | X |
549 +-------------------------+---------+---------+---------+---------+
550 | os-nosdn-lxd-noha | | | | X |
551 +-------------------------+---------+---------+---------+---------+
552 | os-nosdn-fdio-ha | X | | | |
553 +-------------------------+---------+---------+---------+---------+
554 | os-odl_l2-fdio-noha | X | | | |
555 +-------------------------+---------+---------+---------+---------+
556 | os-odl-gluon-noha | X | | | |
557 +-------------------------+---------+---------+---------+---------+
558 | os-nosdn-openo-ha | | X | | |
559 +-------------------------+---------+---------+---------+---------+
560 | os-nosdn-kvm_ovs_dpdk | | | X | |
562 +-------------------------+---------+---------+---------+---------+
563 | os-nosdn-kvm_ovs_dpdk-ha| | | X | |
564 +-------------------------+---------+---------+---------+---------+
565 | os-nosdn-kvm_ovs_dpdk | | | X | |
567 +-------------------------+---------+---------+---------+---------+
568 | os-nosdn-kvm_ovs_dpdk | | | X | |
569 | _bar-noha | | | | |
570 +-------------------------+---------+---------+---------+---------+
571 | opnfv_os-ovn-nofeature- | X | | | |
572 | noha_daily | | | | |
573 +-------------------------+---------+---------+---------+---------+
578 Test results are available in:
580 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
582 The reporting pages can be found at:
584 * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html
585 * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
586 * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html
587 * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html
593 - Floating IP not supported in bgpvpn scenario
595 - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario
597 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
598 which were not fully verified, are listed in the OPNFV installer's release
607 +----------------------------+------------------------------------------------+
608 | **JIRA REFERENCE** | **DESCRIPTION** |
610 +----------------------------+------------------------------------------------+
611 | JIRA: YARDSTICK-776 | Bugfix: cannot run task if without |
612 | | yardstick.conf in danube |
613 +----------------------------+------------------------------------------------+
618 +----------------------------+------------------------------------------------+
619 | **JIRA REFERENCE** | **DESCRIPTION** |
621 +----------------------------+------------------------------------------------+
622 | JIRA: YARDSTICK-714 | Add yardstick env influxdb/grafana command for |
624 +----------------------------+------------------------------------------------+
625 | JIRA: YARDSTICK-655 | Monitor command in tc019 may not show the |
626 | | real nova-api service status |
627 +----------------------------+------------------------------------------------+
628 | JIRA: YARDSTICK-397 | HA testing framework improvement |
630 +----------------------------+------------------------------------------------+
631 | JIRA: YARDSTICK-660 | Improve monitor_process pass criteria |
633 +----------------------------+------------------------------------------------+
634 | JIRA: YARDSTICK-657 | HA monitor_multi bug, |
635 | | KeyError: 'max_outage_time' |
636 +----------------------------+------------------------------------------------+
637 | JIRA: YARDSTICK-647 | TC025 fault_type value is wrong when using |
638 | | baremetal pod scripts |
639 +----------------------------+------------------------------------------------+
640 | JIRA: YARDSTICK-659 | Terminate openstack service process using kill |
641 | | command in HA test cases |
642 +----------------------------+------------------------------------------------+
643 | JIRA: ARMBAND-275 | Yardstick TC005 fails with |
644 | | "Cannot map zero-fill pages" error |
645 +----------------------------+------------------------------------------------+
646 | JIRA: YARDSTICK-561 | Bugfix: AttributeError: 'dict' object has no |
647 | | attribute 'split' if run sample/ping-hot.yaml |
648 +----------------------------+------------------------------------------------+
649 | JIRA: ARMBAND-268 | ERROR No JSON object could be decoded from |
650 | | LMBENCH in TC010 |
651 +----------------------------+------------------------------------------------+
652 | JIRA: YARDSTICK-680 | storperf test case tc074 do not get results |
654 +----------------------------+------------------------------------------------+
658 +----------------------------+------------------------------------------------+
659 | **JIRA REFERENCE** | **DESCRIPTION** |
661 +----------------------------+------------------------------------------------+
662 | JIRA: YARDSTICK-608 | Set work directory in Yardstick container |
664 +----------------------------+------------------------------------------------+
665 | JIRA: YARDSTICK-616 | Bugfix: https support should adapt insecure |
667 +----------------------------+------------------------------------------------+
668 | JIRA: YARDSTICK-620 | Yardstick virtualenv support |
670 +----------------------------+------------------------------------------------+
671 | JIRA: YARDSTICK-621 | Bugfix: fix query job status in TC074 |
673 +----------------------------+------------------------------------------------+
674 | JIRA: YARDSTICK-622 | Bugfix: take test case modification into |
675 | | effect in load_images.sh |
676 +----------------------------+------------------------------------------------+
677 | JIRA: YARDSTICK-623 | change openrc file path to |
678 | | /etc/yardstick/openstack.creds |
679 +----------------------------+------------------------------------------------+
680 | JIRA: YARDSTICK-624 | Add opnfv_os-ovn-nofeature-noha_daily test |
682 +----------------------------+------------------------------------------------+
683 | JIRA: YARDSTICK-628 | Bugfix: Make tc019 and tc025 accept |
684 | | --task-args options |
685 +----------------------------+------------------------------------------------+
686 | JIRA: YARDSTICK-629 | Bugfix: yardstick env prepare cmd do not |
687 | | support other installer |
688 +----------------------------+------------------------------------------------+
689 | JIRA: YARDSTICK-632 | Bugfix: KeyError when using http dispatcher |
691 +----------------------------+------------------------------------------------+
692 | JIRA: YARDSTICK-633 | Bugfix: Environment Compatibility Issues in HA |
694 +----------------------------+------------------------------------------------+
695 | JIRA: YARDSTICK-634 | fix ha issue when run tc050~tc054 in ci |
697 +----------------------------+------------------------------------------------+
698 | JIRA: YARDSTICK-635 | Bugfix: Local Openstack Operation in HA test |
700 +----------------------------+------------------------------------------------+
704 +----------------------------+------------------------------------------------+
705 | **JIRA REFERENCE** | **DESCRIPTION** |
707 +----------------------------+------------------------------------------------+
708 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
710 +----------------------------+------------------------------------------------+
711 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
713 +----------------------------+------------------------------------------------+
716 Danube 3.2 known restrictions/issues
717 ====================================
718 +-----------+-----------+----------------------------------------------+
719 | Installer | Scenario | Issue |
720 +===========+===========+==============================================+
721 | any | *-bgpvpn | Floating ips not supported. Some Test cases |
722 | | | related to floating ips are excluded. |
723 +-----------+-----------+----------------------------------------------+
724 | any | odl_l3-* | Some test cases related to using floating IP |
725 | | | addresses fail because of a known ODL bug. |
727 +-----------+-----------+----------------------------------------------+
728 | compass | odl_l2-* | In some test cases, VM instance will failed |
729 | | | raising network interfaces. |
731 +-----------+-----------+----------------------------------------------+
737 +----------------------------+------------------------------------------------+
738 | **JIRA REFERENCE** | **DESCRIPTION** |
740 +----------------------------+------------------------------------------------+
741 | JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 |
743 +----------------------------+------------------------------------------------+
744 | JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
745 | | (which actually pingable) |
746 +----------------------------+------------------------------------------------+
747 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
749 +----------------------------+------------------------------------------------+
755 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
757 - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning
759 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
761 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
763 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
765 - Yardstick IRC chanel: #opnfv-yardstick