5 OPNFV Danube release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
11 experimental framework are opensource software, licensed under the terms of the
12 Apache License, Version 2.0.
14 =========================================
15 OPNFV Danube Release Note for Yardstick
16 =========================================
21 .. _Yardstick: https://wiki.opnfv.org/yardstick
23 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
25 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
31 This document describes the release note of Yardstick project.
37 +----------------+--------------------+---------------------------------+
38 | *Date* | *Version* | *Comment* |
40 +----------------+--------------------+---------------------------------+
41 | | 3.0 | Yardstick for Danube release |
43 +----------------+--------------------+---------------------------------+
44 | | 2.0 | Yardstick for Danube release |
46 +----------------+--------------------+---------------------------------+
47 | Mar 31st, 2017 | 1.0 | Yardstick for Danube release |
49 +----------------+--------------------+---------------------------------+
55 The software delivered in the OPNFV Yardstick_ Project, comprising the
56 *Yardstick framework*, the *Yardstick test cases* and the experimental
57 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
60 The *Yardstick* framework is *installer*, *infrastructure* and *application*
65 ======================
67 This Danube release provides *Yardstick* as a framework for NFVI testing
68 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
70 * Documentation generated with Sphinx
76 * Release notes (this document)
80 * Automated Yardstick test suite (daily, weekly)
82 * Jenkins Jobs for OPNFV community labs
84 * Automated Yardstick test results visualization
86 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
89 * Yardstick framework source code
91 * Yardstick test cases yaml files
93 * Yardstick plug-in configration yaml files, plug-in install/remove scripts
95 For Danube release, the *Yardstick framework* is used for the following
98 * OPNFV platform testing - generic test cases to measure the categories:
106 * OPNFV platform network service benchmarking(NSB)
110 * Test cases for the following OPNFV Projects:
124 * virtual Traffic Classifier
126 The *Yardstick framework* is developed in the OPNFV community, by the
127 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
130 .. note:: The test case description template used for the Yardstick test cases
131 is based on the document ETSI-ISG NFV-TST001_; the results report template
132 used for the Yardstick results is based on the IEEE Std 829-2008.
138 +--------------------------------------+--------------------------------------+
139 | **Project** | Yardstick |
141 +--------------------------------------+--------------------------------------+
142 | **Repo/tag** | yardstick/Danube.1.0 |
144 +--------------------------------------+--------------------------------------+
145 | **Yardstick Docker image tag** | Danube.1.0 |
147 +--------------------------------------+--------------------------------------+
148 | **Release designation** | Danube |
150 +--------------------------------------+--------------------------------------+
151 | **Release date** | March 31st, 2017 |
153 +--------------------------------------+--------------------------------------+
154 | **Purpose of the delivery** | OPNFV Danube release 1.0 |
156 +--------------------------------------+--------------------------------------+
165 - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html
167 - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html
170 Software Deliverables
171 ---------------------
173 **Yardstick framework source code <Danube.1.0>**
175 +--------------------------------------+--------------------------------------+
176 | **Project** | Yardstick |
178 +--------------------------------------+--------------------------------------+
179 | **Repo/tag** | yardstick/Danube.1.0 |
181 +--------------------------------------+--------------------------------------+
182 | **Yardstick Docker image tag** | Danube.1.0 |
184 +--------------------------------------+--------------------------------------+
185 | **Release designation** | Danube |
187 +--------------------------------------+--------------------------------------+
188 | **Release date** | March 31st, 2017 |
190 +--------------------------------------+--------------------------------------+
191 | **Purpose of the delivery** | OPNFV Danube release |
193 +--------------------------------------+--------------------------------------+
198 +---------------------+-------------------------------------------------------+
199 | **Context** | **Description** |
201 +---------------------+-------------------------------------------------------+
202 | *Heat* | Models orchestration using OpenStack Heat |
204 +---------------------+-------------------------------------------------------+
205 | *Node* | Models Baremetal, Controller, Compute |
207 +---------------------+-------------------------------------------------------+
208 | *Standalone* | Models VM running on Non-Managed NFVi |
210 +---------------------+-------------------------------------------------------+
215 +---------------------+-------------------------------------------------------+
216 | **Runner** | **Description** |
218 +---------------------+-------------------------------------------------------+
219 | *Arithmetic* | Steps every run arithmetically according to specified |
222 +---------------------+-------------------------------------------------------+
223 | *Duration* | Runs for a specified period of time |
225 +---------------------+-------------------------------------------------------+
226 | *Iteration* | Runs for a specified number of iterations |
228 +---------------------+-------------------------------------------------------+
229 | *Sequence* | Selects input value to a scenario from an input file |
230 | | and runs all entries sequentially |
232 +---------------------+-------------------------------------------------------+
237 +---------------------+-------------------------------------------------------+
238 | **Category** | **Delivered** |
240 +---------------------+-------------------------------------------------------+
241 | *Availability* | Attacker: |
243 | | * baremetal, process |
247 | | * check host, openstack, process, service |
249 | | * start/stop service |
253 | | * command, process |
255 +---------------------+-------------------------------------------------------+
256 | *Compute* | * cpuload |
262 | | * lmbench_cache |
274 | | * computecapacity |
276 +---------------------+-------------------------------------------------------+
277 | *Networking* | * iperf3 |
291 | | * sfc with tacker |
293 | | * vtc instantion validation |
295 | | * vtc instantion validation with noisy neighbors |
297 | | * vtc throughput |
299 | | * vtc throughput in the presence of noisy neighbors |
301 | | * networkcapacity |
303 | | * netutilization |
307 +---------------------+-------------------------------------------------------+
308 | *Parser* | Tosca2Heat |
310 +---------------------+-------------------------------------------------------+
313 | | storagecapacity |
315 +---------------------+-------------------------------------------------------+
316 | *StorPerf* | storperf |
318 +---------------------+-------------------------------------------------------+
319 | *NSB* | vPE thoughput test case |
321 +---------------------+-------------------------------------------------------+
324 **API to Other Frameworks**
326 +---------------------+-------------------------------------------------------+
327 | **Framework** | **Description** |
329 +---------------------+-------------------------------------------------------+
330 | *ApexLake* | Experimental framework that enables the user to |
331 | | validate NFVI from the perspective of a VNF. |
332 | | A virtual Traffic Classifier is utilized as VNF. |
333 | | Enables experiments with SR-IOV on Compute Node. |
335 +---------------------+-------------------------------------------------------+
338 **Test Results Output**
340 +-----------------------------+-----------------------------------------------+
341 | **Dispatcher** | **Description** |
343 +-----------------------------+-----------------------------------------------+
344 | file | Log to a file. |
346 +-----------------------------+-----------------------------------------------+
347 | http | Post data to html. |
349 +-----------------------------+-----------------------------------------------+
350 | influxdb | Post data to influxDB. |
352 +-----------------------------+-----------------------------------------------+
358 * Generic NFVI test cases
360 * OPNFV_YARDSTICK_TCOO1 - NW Performance
362 * OPNFV_YARDSTICK_TCOO2 - NW Latency
364 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
366 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
368 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
370 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
372 * OPNFV_YARDSTICK_TCO10 - Memory Latency
374 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
376 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
378 * OPNFV_YARDSTICK_TCO14 - Processing Speed
380 * OPNFV_YARDSTICK_TCO24 - CPU Load
382 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
384 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
387 * OPNFV_YARDSTICK_TCO42 - Network Performance
389 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
391 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
393 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
395 * OPNFV_YARDSTICK_TCO61 - Network Utilization
397 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
399 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
401 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
404 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
406 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
409 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
411 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
413 * OPNFV_YARDSTICK_TCO76 - Network frame error rate
415 * Test Cases for OPNFV HA Project:
417 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
419 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
421 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
424 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
426 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
429 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
432 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
435 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
438 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
441 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
444 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
447 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
449 * Test Case for OPNFV IPv6 Project:
451 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
453 * Test Case for OPNFV KVM Project:
455 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
457 * Test Case for OPNFV Parser Project:
459 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
461 * Test Case for OPNFV StorPerf Project:
463 * OPNFV_YARDSTICK_TCO74 - Storperf
465 * Test Cases for Virtual Traffic Classifier:
467 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
471 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
473 * Benchmarking in presence of noisy neighbors Test:
475 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
477 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
478 presence of noisy neighbors Test
484 Module Version Changes
485 ----------------------
487 This is the fourth tracked release of Yardstick. It is based on following
497 Document Version Changes
498 ------------------------
500 This is the fourth tracked version of the Yardstick framework in OPNFV.
501 It includes the following documentation updates:
503 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
504 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
505 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
507 - Yardstick Developer Guide
509 - Yardstick Release Notes for Yardstick: this document
515 - Yardstick RESTful API support
517 - Introduce Network service benchmarking
519 - Introduce stress testing with Bottlenecks team
521 - Yardstick framework improvement:
523 - Parellel test cases execution support
525 - yardstick report CLI
527 - Node context support openstack configuration via Ansible
537 For Danube 1.0, Yardstick was tested on the following scenarios:
539 +-------------------------+---------+---------+---------+---------+
540 | Scenario | Apex | Compass | Fuel | Joid |
541 +=========================+=========+=========+=========+=========+
542 | os-nosdn-nofeature-noha | | | X | X |
543 +-------------------------+---------+---------+---------+---------+
544 | os-nosdn-nofeature-ha | X | X | X | X |
545 +-------------------------+---------+---------+---------+---------+
546 | os-odl_l2-nofeature-ha | | X | X | X |
547 +-------------------------+---------+---------+---------+---------+
548 | os-odl_l2-nofeature-noha| | | X | |
549 +-------------------------+---------+---------+---------+---------+
550 | os-odl_l3-nofeature-ha | X | X | X | |
551 +-------------------------+---------+---------+---------+---------+
552 | os-odl_l3-nofeature-noha| | | X | |
553 +-------------------------+---------+---------+---------+---------+
554 | os-onos-sfc-ha | | | | |
555 +-------------------------+---------+---------+---------+---------+
556 | os-onos-nofeature-ha | | X | | X |
557 +-------------------------+---------+---------+---------+---------+
558 | os-onos-nofeature-noha | | | | |
559 +-------------------------+---------+---------+---------+---------+
560 | os-odl_l2-sfc-ha | | | X | |
561 +-------------------------+---------+---------+---------+---------+
562 | os-odl_l2-sfc-noha | | | X | |
563 +-------------------------+---------+---------+---------+---------+
564 | os-odl_l2-bgpvpn-ha | X | | X | |
565 +-------------------------+---------+---------+---------+---------+
566 | os-odl_l2-bgpvpn-noha | | | X | |
567 +-------------------------+---------+---------+---------+---------+
568 | os-nosdn-kvm-ha | X | | X | |
569 +-------------------------+---------+---------+---------+---------+
570 | os-nosdn-kvm-noha | | | X | |
571 +-------------------------+---------+---------+---------+---------+
572 | os-nosdn-ovs-ha | | | X | |
573 +-------------------------+---------+---------+---------+---------+
574 | os-nosdn-ovs-noha | | | X | |
575 +-------------------------+---------+---------+---------+---------+
576 | os-ocl-nofeature-ha | | X | | |
577 +-------------------------+---------+---------+---------+---------+
578 | os-nosdn-lxd-ha | | | | X |
579 +-------------------------+---------+---------+---------+---------+
580 | os-nosdn-lxd-noha | | | | X |
581 +-------------------------+---------+---------+---------+---------+
582 | os-nosdn-fdio-ha | X | | | |
583 +-------------------------+---------+---------+---------+---------+
584 | os-odl_l2-fdio-noha | X | | | |
585 +-------------------------+---------+---------+---------+---------+
586 | os-odl-gluon-noha | X | | | |
587 +-------------------------+---------+---------+---------+---------+
588 | os-nosdn-openo-ha | | X | | |
589 +-------------------------+---------+---------+---------+---------+
590 | os-nosdn-kvm_ovs_dpdk | | | X | |
592 +-------------------------+---------+---------+---------+---------+
593 | os-nosdn-kvm_ovs_dpdk-ha| | | X | |
594 +-------------------------+---------+---------+---------+---------+
595 | os-nosdn-kvm_ovs_dpdk | | | X | |
597 +-------------------------+---------+---------+---------+---------+
598 | os-nosdn-kvm_ovs_dpdk | | | X | |
599 | _bar-noha | | | | |
600 +-------------------------+---------+---------+---------+---------+
606 Test results are available in:
608 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
610 The reporting pages can be found at:
612 * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html
613 * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
614 * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html
615 * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html
621 - Floating IP not supported in bgpvpn scenario
623 - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario
625 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
626 which were not fully verified, are listed in the OPNFV installer's release
635 +----------------------------+------------------------------------------------+
636 | **JIRA REFERENCE** | **SLOGAN** |
638 +----------------------------+------------------------------------------------+
639 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
641 +----------------------------+------------------------------------------------+
642 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
644 +----------------------------+------------------------------------------------+
647 Danube 1.0 known restrictions/issues
648 ====================================
649 +-----------+-----------+----------------------------------------------+
650 | Installer | Scenario | Issue |
651 +===========+===========+==============================================+
652 | any | *-bgpvpn | Floating ips not supported. Some Test cases |
653 | | | related to floating ips are excluded. |
654 +-----------+-----------+----------------------------------------------+
655 | any | odl_l3-* | Some test cases related to using floating IP |
656 | | | addresses fail because of a known ODL bug. |
658 +-----------+-----------+----------------------------------------------+
659 | compass | odl_l2-* | In some test cases, VM instance will failed |
660 | | | raising network interfaces. |
662 +-----------+-----------+----------------------------------------------+
669 +----------------------------+------------------------------------------------+
670 | **JIRA REFERENCE** | **SLOGAN** |
672 +----------------------------+------------------------------------------------+
673 | JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
674 | | (which actually pingable) |
675 +----------------------------+------------------------------------------------+
676 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
678 +----------------------------+------------------------------------------------+
684 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
686 - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning
688 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
690 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
692 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
694 - Yardstick IRC chanel: #opnfv-yardstick