5 OPNFV Colorado release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
11 experimental framework are opensource software, licensed under the terms of the
12 Apache License, Version 2.0.
14 =========================================
15 OPNFV Colorado Release Note for Yardstick
16 =========================================
21 .. _Yardstick: https://wiki.opnfv.org/yardstick
23 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
25 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
31 This document describes the release note of Yardstick project.
37 +----------------+--------------------+---------------------------------+
38 | *Date* | *Version* | *Comment* |
40 +----------------+--------------------+---------------------------------+
41 | Oct 27nd, 2016 | 2.0 | Yardstick for Colorado release |
43 +----------------+--------------------+---------------------------------+
44 | Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release |
46 +----------------+--------------------+---------------------------------+
52 The software delivered in the OPNFV Yardstick_ Project, comprising the
53 *Yardstick framework*, the *Yardstick test cases* and the experimental
54 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
57 The *Yardstick* framework is *installer*, *infrastructure* and *application*
61 OPNFV Colorado Release
62 ======================
64 This Colorado release provides *Yardstick* as a framework for NFVI testing
65 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
67 * Documentation generated with Sphinx
73 * Release notes (this document)
77 * Automated Yardstick test suite (daily, weekly)
79 * Jenkins Jobs for OPNFV community labs
81 * Automated Yardstick test results visualization
83 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
86 * Yardstick framework source code
88 * Yardstick test cases yaml files
90 * Yardstick pliug-in configration yaml files, plug-in install/remove scripts
92 For Colorado release, the *Yardstick framework* is used for the following
95 * OPNFV platform testing - generic test cases to measure the categories:
103 * Test cases for the following OPNFV Projects:
117 * virtual Traffic Classifier
119 The *Yardstick framework* is developed in the OPNFV community, by the
120 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
123 .. note:: The test case description template used for the Yardstick test cases
124 is based on the document ETSI-ISG NFV-TST001_; the results report template
125 used for the Yardstick results is based on the IEEE Std 829-2008.
131 +--------------------------------------+--------------------------------------+
132 | **Project** | Yardstick |
134 +--------------------------------------+--------------------------------------+
135 | **Repo/tag** | yardstick/colorado.2.0 |
137 +--------------------------------------+--------------------------------------+
138 | **Yardstick Docker image tag** | colorado.2.0 |
140 +--------------------------------------+--------------------------------------+
141 | **Release designation** | Colorado |
143 +--------------------------------------+--------------------------------------+
144 | **Release date** | October 27 2016 |
146 +--------------------------------------+--------------------------------------+
147 | **Purpose of the delivery** | OPNFV Colorado release 2.0 |
149 +--------------------------------------+--------------------------------------+
158 - User Guide: http://artifacts.opnfv.org/yardstick/colorado/docs/userguide/index.html
160 - Test Results: http://artifacts.opnfv.org/yardstick/colorado/docs/results/overview.html
163 Software Deliverables
164 ---------------------
166 **Yardstick framework source code <colorado.2.0>**
168 +--------------------------------------+--------------------------------------+
169 | **Project** | Yardstick |
171 +--------------------------------------+--------------------------------------+
172 | **Repo/tag** | yardstick/colorado.2.0 |
174 +--------------------------------------+--------------------------------------+
175 | **Yardstick Docker image tag** | colorado.2.0 |
177 +--------------------------------------+--------------------------------------+
178 | **Release designation** | Colorado |
180 +--------------------------------------+--------------------------------------+
181 | **Release date** | October 27th, 2016 |
183 +--------------------------------------+--------------------------------------+
184 | **Purpose of the delivery** | OPNFV Colorado release |
186 +--------------------------------------+--------------------------------------+
191 +---------------------+-------------------------------------------------------+
192 | **Context** | **Description** |
194 +---------------------+-------------------------------------------------------+
195 | *Heat* | Models orchestration using OpenStack Heat |
197 +---------------------+-------------------------------------------------------+
198 | *Node* | Models Baremetal, Controller, Compute |
200 +---------------------+-------------------------------------------------------+
205 +---------------------+-------------------------------------------------------+
206 | **Runner** | **Description** |
208 +---------------------+-------------------------------------------------------+
209 | *Arithmetic* | Steps every run arithmetically according to specified |
212 +---------------------+-------------------------------------------------------+
213 | *Duration* | Runs for a specified period of time |
215 +---------------------+-------------------------------------------------------+
216 | *Iteration* | Runs for a specified number of iterations |
218 +---------------------+-------------------------------------------------------+
219 | *Sequence* | Selects input value to a scenario from an input file |
220 | | and runs all entries sequentially |
222 +---------------------+-------------------------------------------------------+
227 +---------------------+-------------------------------------------------------+
228 | **Category** | **Delivered** |
230 +---------------------+-------------------------------------------------------+
231 | *Availability* | Attacker: |
233 | | * baremetal, process |
237 | | * check host, openstack, process, service |
239 | | * start/stop service |
243 | | * command, process |
245 +---------------------+-------------------------------------------------------+
246 | *Compute* | * cpuload |
252 | | * lmbench_cache |
264 | | * computecapacity |
266 +---------------------+-------------------------------------------------------+
267 | *Networking* | * iperf3 |
281 | | * sfc with tacker |
283 | | * vtc instantion validation |
285 | | * vtc instantion validation with noisy neighbors |
287 | | * vtc throughput |
289 | | * vtc throughput in the presence of noisy neighbors |
291 | | * networkcapacity |
293 | | * netutilization |
295 +---------------------+-------------------------------------------------------+
296 | *Parser* | Tosca2Heat |
298 +---------------------+-------------------------------------------------------+
301 | | storagecapacity |
303 +---------------------+-------------------------------------------------------+
304 | *StorPerf* | storperf |
306 +---------------------+-------------------------------------------------------+
309 **API to Other Frameworks**
311 +---------------------+-------------------------------------------------------+
312 | **Framework** | **Description** |
314 +---------------------+-------------------------------------------------------+
315 | *ApexLake* | Experimental framework that enables the user to |
316 | | validate NFVI from the perspective of a VNF. |
317 | | A virtual Traffic Classifier is utilized as VNF. |
318 | | Enables experiments with SR-IOV on Compute Node. |
320 +---------------------+-------------------------------------------------------+
323 **Test Results Output**
325 +-----------------------------+-----------------------------------------------+
326 | **Dispatcher** | **Description** |
328 +-----------------------------+-----------------------------------------------+
329 | file | Log to a file. |
331 +-----------------------------+-----------------------------------------------+
332 | http | Post data to html. |
334 +-----------------------------+-----------------------------------------------+
335 | influxdb | Post data to influxDB. |
337 +-----------------------------+-----------------------------------------------+
343 * Generic NFVI test cases
345 * OPNFV_YARDSTICK_TCOO1 - NW Performance
347 * OPNFV_YARDSTICK_TCOO2 - NW Latency
349 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
351 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
353 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
355 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
357 * OPNFV_YARDSTICK_TCO10 - Memory Latency
359 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
361 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
363 * OPNFV_YARDSTICK_TCO14 - Processing Speed
365 * OPNFV_YARDSTICK_TCO24 - CPU Load
367 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
369 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
372 * OPNFV_YARDSTICK_TCO42 - Network Performance
374 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
376 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
378 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
380 * OPNFV_YARDSTICK_TCO61 - Network Utilization
382 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
384 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
386 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
389 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
391 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
394 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
396 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
398 * Test Cases for OPNFV HA Project:
400 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
402 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
404 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
407 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
409 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
412 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
415 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
418 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
421 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
424 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
427 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
430 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
432 * Test Case for OPNFV IPv6 Project:
434 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
436 * Test Case for OPNFV KVM Project:
438 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
440 * Test Case for OPNFV Parser Project:
442 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
444 * Test Case for OPNFV StorPerf Project:
446 * OPNFV_YARDSTICK_TCO74 - Storperf
448 * Test Cases for Virtual Traffic Classifier:
450 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
453 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
454 Benchmarking in presence of noisy neighbors Test
456 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
458 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
459 presence of noisy neighbors Test
465 Module Version Changes
466 ----------------------
468 This is the second tracked release of Yardstick. It is based on following
475 - OpenDaylight Beryllium
478 Document Version Changes
479 ------------------------
481 This is the second tracked version of the Yardstick framework in OPNFV.
482 It includes the following documentation updates:
484 - Yardstick User Guide: added yardstick plugin chapter; added Store Other
485 Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter.
487 - Yardstick Code Documentation: no changes
489 - Yardstick Release Notes for Yardstick: this document
491 - Test Results report for Colorado testing with Yardstick: updated listed of
492 verified scenarios and limitations
498 - Yardstick reporting
499 - StorPerf Integration
505 For Colorado 2.0, Yardstick was tested on the following scenarios:
507 +-------------------------+---------+---------+---------+---------+
508 | Scenario | Apex | Compass | Fuel | Joid |
509 +=========================+=========+=========+=========+=========+
510 | os-nosdn-nofeature-noha | | | | X |
511 +-------------------------+---------+---------+---------+---------+
512 | os-nosdn-nofeature-ha | X | | X | X |
513 +-------------------------+---------+---------+---------+---------+
514 | os-odl_l2-nofeature-ha | X | X | X | X |
515 +-------------------------+---------+---------+---------+---------+
516 | os-odl_l2-nofeature-noha| | X | | |
517 +-------------------------+---------+---------+---------+---------+
518 | os-odl_l3-nofeature-ha | X | | X | |
519 +-------------------------+---------+---------+---------+---------+
520 | os-odl_l3-nofeature-ha | | X | | |
521 +-------------------------+---------+---------+---------+---------+
522 | os-onos-sfc-ha | X | | X | X |
523 +-------------------------+---------+---------+---------+---------+
524 | os-onos-nofeature-ha | X | | X | X |
525 +-------------------------+---------+---------+---------+---------+
526 | os-onos-nofeature-noha | | X | | |
527 +-------------------------+---------+---------+---------+---------+
528 | os-odl_l2-sfc-ha | | | X | |
529 +-------------------------+---------+---------+---------+---------+
530 | os-odl_l2-sfc-noha | X | X | | |
531 +-------------------------+---------+---------+---------+---------+
532 | os-odl_l2-bgpvpn-ha | X | | X | |
533 +-------------------------+---------+---------+---------+---------+
534 | os-odl_l2-bgpvpn-noha | | X | | |
535 +-------------------------+---------+---------+---------+---------+
536 | os-nosdn-kvm-ha | | | X | |
537 +-------------------------+---------+---------+---------+---------+
538 | os-nosdn-kvm-noha | | X | | |
539 +-------------------------+---------+---------+---------+---------+
540 | os-nosdn-ovs-ha | | | X | |
541 +-------------------------+---------+---------+---------+---------+
542 | os-nosdn-ovs-noha | X | | X | |
543 +-------------------------+---------+---------+---------+---------+
544 | os-ocl-nofeature-ha | | | | |
545 +-------------------------+---------+---------+---------+---------+
546 | os-nosdn-lxd-ha | | | | X |
547 +-------------------------+---------+---------+---------+---------+
548 | os-nosdn-lxd-noha | | | | X |
549 +-------------------------+---------+---------+---------+---------+
550 | os-odl_l2-fdio-noha | X | | | |
551 +-------------------------+---------+---------+---------+---------+
557 Test results are available in:
559 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
561 The reporting pages can be found at:
563 * apex: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html
564 * compass: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html
565 * fuel: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-fuel.html
566 * joid: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-joid.html
568 You can get additional details through test logs on http://artifacts.opnfv.org/.
569 As no search engine is available on the OPNFV artifact web site you must
570 retrieve the pod identifier on which the tests have been executed (see
571 field pod in any of the results) then click on the selected POD and look
572 for the date of the test you are interested in.
577 - Floating IP not supported in bgpvpn scenario
578 - Floating IP not supported in apex-os-odl_l3-nofeature-ha scenario
580 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
581 which were not fully verified, are listed in the OPNFV installer's release
590 +----------------------------+------------------------------------------------+
591 | **JIRA REFERENCE** | **SLOGAN** |
593 +----------------------------+------------------------------------------------+
594 | JIRA: YARDSTICK-325 | Provide raw format yardstick vm image for |
595 | | nova-lxd scenario. |
597 +----------------------------+------------------------------------------------+
598 | JIRA: YARDSTICK-358 | tc027 ipv6 test case to de-coupling to the |
601 +----------------------------+------------------------------------------------+
602 | JIRA: YARDSTICK-359 | ipv6 testcase disable port-security on |
605 +----------------------------+------------------------------------------------+
606 | JIRA: YARDSTICK-363 | ipv6 testcase to support fuel. |
608 +----------------------------+------------------------------------------------+
609 | JIRA: YARDSTICK-367 | Add d3 graph presentation to yardstick |
612 +----------------------------+------------------------------------------------+
613 | JIRA: YARDSTICK-371 | Provide raw format yardstick vm image for |
614 | | nova-lxd scenario. |
616 +----------------------------+------------------------------------------------+
617 | JIRA: YARDSTICK-372 | cannot find yardstick-img-dpdk-modify and |
618 | | yardstick-img-lxd-modify in environment |
621 +----------------------------+------------------------------------------------+
624 Colorado 2.0 known restrictions/issues
625 ==================================
626 +-----------+-----------+----------------------------------------------+
627 | Installer | Scenario | Issue |
628 +===========+===========+==============================================+
629 | any | *-bgpvpn | Floating ips not supported. Some Test cases |
630 | | | related to floating ips are excluded. |
631 +-----------+-----------+----------------------------------------------+
632 | any | odl_l3-* | Some test cases related to using floating IP |
633 | | | addresses fail because of a known ODL bug. |
634 | | | https://jira.opnfv.org/browse/APEX-112 |
635 +-----------+-----------+----------------------------------------------+
645 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
647 - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
649 - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
651 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
653 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
655 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
657 - Yardstick IRC chanel: #opnfv-yardstick
659 .. _`YARDSTICK-325` : https://jira.opnfv.org/browse/YARDSTICK-325
661 .. _`YARDSTICK-358` : https://jira.opnfv.org/browse/YARDSTICK-358
663 .. _`YARDSTICK-359` : https://jira.opnfv.org/browse/YARDSTICK-359
665 .. _`YARDSTICK-363` : https://jira.opnfv.org/browse/YARDSTICK-363
667 .. _`YARDSTICK-367` : https://jira.opnfv.org/browse/YARDSTICK-367
669 .. _`YARDSTICK-371` : https://jira.opnfv.org/browse/YARDSTICK-371
671 .. _`YARDSTICK-372` : https://jira.opnfv.org/browse/YARDSTICK-372