5 OPNFV Danube release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
11 experimental framework are opensource software, licensed under the terms of the
12 Apache License, Version 2.0.
14 =========================================
15 OPNFV Danube Release Note for Yardstick
16 =========================================
21 .. _Yardstick: https://wiki.opnfv.org/yardstick
23 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
25 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
31 This document describes the release note of Yardstick project.
37 +----------------+--------------------+---------------------------------+
38 | *Date* | *Version* | *Comment* |
40 +----------------+--------------------+---------------------------------+
41 | | 3.0 | Yardstick for Danube release |
43 +----------------+--------------------+---------------------------------+
44 | | 2.0 | Yardstick for Danube release |
46 +----------------+--------------------+---------------------------------+
47 | Mar 31st, 2017 | 1.0 | Yardstick for Danube release |
49 +----------------+--------------------+---------------------------------+
55 The software delivered in the OPNFV Yardstick_ Project, comprising the
56 *Yardstick framework*, the *Yardstick test cases* and the experimental
57 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
60 The *Yardstick* framework is *installer*, *infrastructure* and *application*
65 ======================
67 This Danube release provides *Yardstick* as a framework for NFVI testing
68 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
70 * Documentation generated with Sphinx
76 * Release notes (this document)
80 * Automated Yardstick test suite (daily, weekly)
82 * Jenkins Jobs for OPNFV community labs
84 * Automated Yardstick test results visualization
86 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
89 * Yardstick framework source code
91 * Yardstick test cases yaml files
93 * Yardstick pliug-in configration yaml files, plug-in install/remove scripts
95 For Danube release, the *Yardstick framework* is used for the following
98 * OPNFV platform testing - generic test cases to measure the categories:
106 * OPNFV platform network service benchmarking(NSB)
110 * Test cases for the following OPNFV Projects:
124 * virtual Traffic Classifier
126 The *Yardstick framework* is developed in the OPNFV community, by the
127 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
130 .. note:: The test case description template used for the Yardstick test cases
131 is based on the document ETSI-ISG NFV-TST001_; the results report template
132 used for the Yardstick results is based on the IEEE Std 829-2008.
138 +--------------------------------------+--------------------------------------+
139 | **Project** | Yardstick |
141 +--------------------------------------+--------------------------------------+
142 | **Repo/tag** | yardstick/Danube.1.0 |
144 +--------------------------------------+--------------------------------------+
145 | **Yardstick Docker image tag** | Danube.1.0 |
147 +--------------------------------------+--------------------------------------+
148 | **Release designation** | Danube |
150 +--------------------------------------+--------------------------------------+
151 | **Release date** | March 31st, 2017 |
153 +--------------------------------------+--------------------------------------+
154 | **Purpose of the delivery** | OPNFV Danube release 1.0 |
156 +--------------------------------------+--------------------------------------+
165 - User Guide: http://artifacts.opnfv.org/yardstick/Danube/docs/userguide/index.html
167 - Test Results: http://artifacts.opnfv.org/yardstick/Danube/docs/results/overview.html
170 Software Deliverables
171 ---------------------
173 **Yardstick framework source code <Danube.1.0>**
175 +--------------------------------------+--------------------------------------+
176 | **Project** | Yardstick |
178 +--------------------------------------+--------------------------------------+
179 | **Repo/tag** | yardstick/Danube.1.0 |
181 +--------------------------------------+--------------------------------------+
182 | **Yardstick Docker image tag** | Danube.1.0 |
184 +--------------------------------------+--------------------------------------+
185 | **Release designation** | Danube |
187 +--------------------------------------+--------------------------------------+
188 | **Release date** | March 31st, 2017 |
190 +--------------------------------------+--------------------------------------+
191 | **Purpose of the delivery** | OPNFV Danube release |
193 +--------------------------------------+--------------------------------------+
198 +---------------------+-------------------------------------------------------+
199 | **Context** | **Description** |
201 +---------------------+-------------------------------------------------------+
202 | *Heat* | Models orchestration using OpenStack Heat |
204 +---------------------+-------------------------------------------------------+
205 | *Node* | Models Baremetal, Controller, Compute |
207 +---------------------+-------------------------------------------------------+
208 | *Standalone* | Models to deploy/undeploy NFVi infrastructure to run |
211 +---------------------+-------------------------------------------------------+
216 +---------------------+-------------------------------------------------------+
217 | **Runner** | **Description** |
219 +---------------------+-------------------------------------------------------+
220 | *Arithmetic* | Steps every run arithmetically according to specified |
223 +---------------------+-------------------------------------------------------+
224 | *Duration* | Runs for a specified period of time |
226 +---------------------+-------------------------------------------------------+
227 | *Iteration* | Runs for a specified number of iterations |
229 +---------------------+-------------------------------------------------------+
230 | *Sequence* | Selects input value to a scenario from an input file |
231 | | and runs all entries sequentially |
233 +---------------------+-------------------------------------------------------+
238 +---------------------+-------------------------------------------------------+
239 | **Category** | **Delivered** |
241 +---------------------+-------------------------------------------------------+
242 | *Availability* | Attacker: |
244 | | * baremetal, process |
248 | | * check host, openstack, process, service |
250 | | * start/stop service |
254 | | * command, process |
256 +---------------------+-------------------------------------------------------+
257 | *Compute* | * cpuload |
263 | | * lmbench_cache |
275 | | * computecapacity |
277 +---------------------+-------------------------------------------------------+
278 | *Networking* | * iperf3 |
292 | | * sfc with tacker |
294 | | * vtc instantion validation |
296 | | * vtc instantion validation with noisy neighbors |
298 | | * vtc throughput |
300 | | * vtc throughput in the presence of noisy neighbors |
302 | | * networkcapacity |
304 | | * netutilization |
308 +---------------------+-------------------------------------------------------+
309 | *Parser* | Tosca2Heat |
311 +---------------------+-------------------------------------------------------+
314 | | storagecapacity |
316 +---------------------+-------------------------------------------------------+
317 | *StorPerf* | storperf |
319 +---------------------+-------------------------------------------------------+
320 | *NSB* | vPE thoughput test case |
322 +---------------------+-------------------------------------------------------+
325 **API to Other Frameworks**
327 +---------------------+-------------------------------------------------------+
328 | **Framework** | **Description** |
330 +---------------------+-------------------------------------------------------+
331 | *ApexLake* | Experimental framework that enables the user to |
332 | | validate NFVI from the perspective of a VNF. |
333 | | A virtual Traffic Classifier is utilized as VNF. |
334 | | Enables experiments with SR-IOV on Compute Node. |
336 +---------------------+-------------------------------------------------------+
339 **Test Results Output**
341 +-----------------------------+-----------------------------------------------+
342 | **Dispatcher** | **Description** |
344 +-----------------------------+-----------------------------------------------+
345 | file | Log to a file. |
347 +-----------------------------+-----------------------------------------------+
348 | http | Post data to html. |
350 +-----------------------------+-----------------------------------------------+
351 | influxdb | Post data to influxDB. |
353 +-----------------------------+-----------------------------------------------+
359 * Generic NFVI test cases
361 * OPNFV_YARDSTICK_TCOO1 - NW Performance
363 * OPNFV_YARDSTICK_TCOO2 - NW Latency
365 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
367 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
369 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
371 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
373 * OPNFV_YARDSTICK_TCO10 - Memory Latency
375 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
377 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
379 * OPNFV_YARDSTICK_TCO14 - Processing Speed
381 * OPNFV_YARDSTICK_TCO24 - CPU Load
383 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
385 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
388 * OPNFV_YARDSTICK_TCO42 - Network Performance
390 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
392 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
394 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
396 * OPNFV_YARDSTICK_TCO61 - Network Utilization
398 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
400 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
402 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
405 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
407 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
410 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
412 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
414 * OPNFV_YARDSTICK_TCO76 - Network frame error rate
416 * Test Cases for OPNFV HA Project:
418 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
420 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
422 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
425 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
427 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
430 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
433 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
436 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
439 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
442 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
445 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
448 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
450 * Test Case for OPNFV IPv6 Project:
452 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
454 * Test Case for OPNFV KVM Project:
456 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
458 * Test Case for OPNFV Parser Project:
460 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
462 * Test Case for OPNFV StorPerf Project:
464 * OPNFV_YARDSTICK_TCO74 - Storperf
466 * Test Cases for Virtual Traffic Classifier:
468 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
471 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
472 Benchmarking in presence of noisy neighbors Test
474 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
476 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
477 presence of noisy neighbors Test
483 Module Version Changes
484 ----------------------
486 This is the second tracked release of Yardstick. It is based on following
493 - OpenDaylight Beryllium
496 Document Version Changes
497 ------------------------
499 This is the fouth tracked version of the Yardstick framework in OPNFV.
500 It includes the following documentation updates:
502 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
503 add "Yardstick - NSB Testing -Installation" chapter;
504 Update Yardstick installation chapter;
506 - Yardstick Developer Guide
508 - Yardstick Code Documentation: no changes
510 - Yardstick Release Notes for Yardstick: this document
512 - Test Results report for Danube testing with Yardstick: updated listed of
513 verified scenarios and limitations
518 - Yardstick ReST API support
519 - Network service benchmarking
524 For Danube 1.0, Yardstick was tested on the following scenarios:
526 +-------------------------+---------+---------+---------+---------+
527 | Scenario | Apex | Compass | Fuel | Joid |
528 +=========================+=========+=========+=========+=========+
529 | os-nosdn-nofeature-noha | | | X | X |
530 +-------------------------+---------+---------+---------+---------+
531 | os-nosdn-nofeature-ha | X | X | X | X |
532 +-------------------------+---------+---------+---------+---------+
533 | os-odl_l2-nofeature-ha | | X | X | X |
534 +-------------------------+---------+---------+---------+---------+
535 | os-odl_l2-nofeature-noha| | | X | |
536 +-------------------------+---------+---------+---------+---------+
537 | os-odl_l3-nofeature-ha | X | X | X | |
538 +-------------------------+---------+---------+---------+---------+
539 | os-odl_l3-nofeature-noha| | | X | |
540 +-------------------------+---------+---------+---------+---------+
541 | os-onos-sfc-ha | | | | |
542 +-------------------------+---------+---------+---------+---------+
543 | os-onos-nofeature-ha | | X | | X |
544 +-------------------------+---------+---------+---------+---------+
545 | os-onos-nofeature-noha | | | | |
546 +-------------------------+---------+---------+---------+---------+
547 | os-odl_l2-sfc-ha | | | X | |
548 +-------------------------+---------+---------+---------+---------+
549 | os-odl_l2-sfc-noha | | | X | |
550 +-------------------------+---------+---------+---------+---------+
551 | os-odl_l2-bgpvpn-ha | X | | X | |
552 +-------------------------+---------+---------+---------+---------+
553 | os-odl_l2-bgpvpn-noha | | | X | |
554 +-------------------------+---------+---------+---------+---------+
555 | os-nosdn-kvm-ha | X | | X | |
556 +-------------------------+---------+---------+---------+---------+
557 | os-nosdn-kvm-noha | | | X | |
558 +-------------------------+---------+---------+---------+---------+
559 | os-nosdn-ovs-ha | | | X | |
560 +-------------------------+---------+---------+---------+---------+
561 | os-nosdn-ovs-noha | | | X | |
562 +-------------------------+---------+---------+---------+---------+
563 | os-ocl-nofeature-ha | | X | | |
564 +-------------------------+---------+---------+---------+---------+
565 | os-nosdn-lxd-ha | | | | X |
566 +-------------------------+---------+---------+---------+---------+
567 | os-nosdn-lxd-noha | | | | X |
568 +-------------------------+---------+---------+---------+---------+
569 | os-nosdn-fdio-ha | X | | | |
570 +-------------------------+---------+---------+---------+---------+
571 | os-odl_l2-fdio-noha | X | | | |
572 +-------------------------+---------+---------+---------+---------+
573 | os-odl-gluon-noha | X | | | |
574 +-------------------------+---------+---------+---------+---------+
575 | os-nosdn-openo-ha | | X | | |
576 +-------------------------+---------+---------+---------+---------+
577 | os-nosdn-kvm_ovs_dpdk | | | X | |
579 +-------------------------+---------+---------+---------+---------+
580 | os-nosdn-kvm_ovs_dpdk-ha| | | X | |
581 +-------------------------+---------+---------+---------+---------+
582 | os-nosdn-kvm_ovs_dpdk | | | X | |
584 +-------------------------+---------+---------+---------+---------+
585 | os-nosdn-kvm_ovs_dpdk | | | X | |
586 | _bar-noha | | | | |
587 +-------------------------+---------+---------+---------+---------+
593 Test results are available in:
595 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
597 The reporting pages can be found at:
599 * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html
600 * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
601 * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html
602 * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html
604 You can get additional details through test logs on http://artifacts.opnfv.org/.
605 As no search engine is available on the OPNFV artifact web site you must
606 retrieve the pod identifier on which the tests have been executed (see
607 field pod in any of the results) then click on the selected POD and look
608 for the date of the test you are interested in.
613 - Floating IP not supported in bgpvpn scenario
614 - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario
616 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
617 which were not fully verified, are listed in the OPNFV installer's release
627 +----------------------------+------------------------------------------------+
628 | **JIRA REFERENCE** | **SLOGAN** |
630 +----------------------------+------------------------------------------------+
631 | JIRA: YARDSTICK-593 | fix yardstick-image built without test tools |
633 +----------------------------+------------------------------------------------+
634 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
636 +----------------------------+------------------------------------------------+
637 | JIRA: YARDSTICK-600 | netperf_bottleneck execute failed because |
638 | | is_same_heat_context return True |
639 +----------------------------+------------------------------------------------+
640 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
642 +----------------------------+------------------------------------------------+
643 | JIRA: YARDSTICK-604 | add storperf flavor for TC074 |
645 +----------------------------+------------------------------------------------+
646 | JIRA: YARDSTICK-605 | pathspec 'master' did not match any file(s) |
648 +----------------------------+------------------------------------------------+
649 | JIRA: YARDSTICK-607 | Error validating value 'cirros-0.3.5' |
651 +----------------------------+------------------------------------------------+
652 | JIRA: YARDSTICK-609 | Remove improper license header in swagger docs |
654 +----------------------------+------------------------------------------------+
657 Danube 1.0 known restrictions/issues
658 ====================================
659 +-----------+-----------+----------------------------------------------+
660 | Installer | Scenario | Issue |
661 +===========+===========+==============================================+
662 | any | *-bgpvpn | Floating ips not supported. Some Test cases |
663 | | | related to floating ips are excluded. |
664 +-----------+-----------+----------------------------------------------+
665 | any | odl_l3-* | Some test cases related to using floating IP |
666 | | | addresses fail because of a known ODL bug. |
668 +-----------+-----------+----------------------------------------------+
669 | compass | odl_l2-* | In some test cases, VM instance will failed |
670 | | | raising network interfaces. |
672 +-----------+-----------+----------------------------------------------+
679 +----------------------------+------------------------------------------------+
680 | **JIRA REFERENCE** | **SLOGAN** |
682 +----------------------------+------------------------------------------------+
683 | JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
684 | | (which actually pingable) |
685 +----------------------------+------------------------------------------------+
686 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
688 +----------------------------+------------------------------------------------+
693 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
695 - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning
697 - wiki Yardstick Danube release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Danube
699 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
701 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
703 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
705 - Yardstick IRC chanel: #opnfv-yardstick