5 OPNFV Colorado release note for Yardstick Docs
6 are licensed under a Creative Commons Attribution 4.0 International License.
7 You should have received a copy of the license along with this.
8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
10 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
11 experimental framework are opensource software, licensed under the terms of the
12 Apache License, Version 2.0.
14 =========================================
15 OPNFV Colorado Release Note for Yardstick
16 =========================================
21 .. _Yardstick: https://wiki.opnfv.org/yardstick
23 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
25 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
31 This document describes the release note of Yardstick project.
37 +----------------+--------------------+---------------------------------+
38 | *Date* | *Version* | *Comment* |
40 +----------------+--------------------+---------------------------------+
41 | Dec 5th, 2016 | 3.0 | Yardstick for Colorado release |
43 +----------------+--------------------+---------------------------------+
44 | Oct 27th, 2016 | 2.0 | Yardstick for Colorado release |
46 +----------------+--------------------+---------------------------------+
47 | Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release |
49 +----------------+--------------------+---------------------------------+
55 The software delivered in the OPNFV Yardstick_ Project, comprising the
56 *Yardstick framework*, the *Yardstick test cases* and the experimental
57 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
60 The *Yardstick* framework is *installer*, *infrastructure* and *application*
64 OPNFV Colorado Release
65 ======================
67 This Colorado release provides *Yardstick* as a framework for NFVI testing
68 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
70 * Documentation generated with Sphinx
76 * Release notes (this document)
80 * Automated Yardstick test suite (daily, weekly)
82 * Jenkins Jobs for OPNFV community labs
84 * Automated Yardstick test results visualization
86 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
89 * Yardstick framework source code
91 * Yardstick test cases yaml files
93 * Yardstick pliug-in configration yaml files, plug-in install/remove scripts
95 For Colorado release, the *Yardstick framework* is used for the following
98 * OPNFV platform testing - generic test cases to measure the categories:
106 * Test cases for the following OPNFV Projects:
120 * virtual Traffic Classifier
122 The *Yardstick framework* is developed in the OPNFV community, by the
123 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
126 .. note:: The test case description template used for the Yardstick test cases
127 is based on the document ETSI-ISG NFV-TST001_; the results report template
128 used for the Yardstick results is based on the IEEE Std 829-2008.
134 +--------------------------------------+--------------------------------------+
135 | **Project** | Yardstick |
137 +--------------------------------------+--------------------------------------+
138 | **Repo/tag** | yardstick/colorado.3.0 |
140 +--------------------------------------+--------------------------------------+
141 | **Yardstick Docker image tag** | colorado.3.0 |
143 +--------------------------------------+--------------------------------------+
144 | **Release designation** | Colorado |
146 +--------------------------------------+--------------------------------------+
147 | **Release date** | December 5th, 2016 |
149 +--------------------------------------+--------------------------------------+
150 | **Purpose of the delivery** | OPNFV Colorado release 3.0 |
152 +--------------------------------------+--------------------------------------+
161 - User Guide: http://artifacts.opnfv.org/yardstick/colorado/docs/userguide/index.html
163 - Test Results: http://artifacts.opnfv.org/yardstick/colorado/docs/results/overview.html
166 Software Deliverables
167 ---------------------
169 **Yardstick framework source code <colorado.3.0>**
171 +--------------------------------------+--------------------------------------+
172 | **Project** | Yardstick |
174 +--------------------------------------+--------------------------------------+
175 | **Repo/tag** | yardstick/colorado.3.0 |
177 +--------------------------------------+--------------------------------------+
178 | **Yardstick Docker image tag** | colorado.3.0 |
180 +--------------------------------------+--------------------------------------+
181 | **Release designation** | Colorado |
183 +--------------------------------------+--------------------------------------+
184 | **Release date** | December 5th, 2016 |
186 +--------------------------------------+--------------------------------------+
187 | **Purpose of the delivery** | OPNFV Colorado release |
189 +--------------------------------------+--------------------------------------+
194 +---------------------+-------------------------------------------------------+
195 | **Context** | **Description** |
197 +---------------------+-------------------------------------------------------+
198 | *Heat* | Models orchestration using OpenStack Heat |
200 +---------------------+-------------------------------------------------------+
201 | *Node* | Models Baremetal, Controller, Compute |
203 +---------------------+-------------------------------------------------------+
208 +---------------------+-------------------------------------------------------+
209 | **Runner** | **Description** |
211 +---------------------+-------------------------------------------------------+
212 | *Arithmetic* | Steps every run arithmetically according to specified |
215 +---------------------+-------------------------------------------------------+
216 | *Duration* | Runs for a specified period of time |
218 +---------------------+-------------------------------------------------------+
219 | *Iteration* | Runs for a specified number of iterations |
221 +---------------------+-------------------------------------------------------+
222 | *Sequence* | Selects input value to a scenario from an input file |
223 | | and runs all entries sequentially |
225 +---------------------+-------------------------------------------------------+
230 +---------------------+-------------------------------------------------------+
231 | **Category** | **Delivered** |
233 +---------------------+-------------------------------------------------------+
234 | *Availability* | Attacker: |
236 | | * baremetal, process |
240 | | * check host, openstack, process, service |
242 | | * start/stop service |
246 | | * command, process |
248 +---------------------+-------------------------------------------------------+
249 | *Compute* | * cpuload |
255 | | * lmbench_cache |
267 | | * computecapacity |
269 +---------------------+-------------------------------------------------------+
270 | *Networking* | * iperf3 |
284 | | * sfc with tacker |
286 | | * vtc instantion validation |
288 | | * vtc instantion validation with noisy neighbors |
290 | | * vtc throughput |
292 | | * vtc throughput in the presence of noisy neighbors |
294 | | * networkcapacity |
296 | | * netutilization |
298 +---------------------+-------------------------------------------------------+
299 | *Parser* | Tosca2Heat |
301 +---------------------+-------------------------------------------------------+
304 | | storagecapacity |
306 +---------------------+-------------------------------------------------------+
307 | *StorPerf* | storperf |
309 +---------------------+-------------------------------------------------------+
312 **API to Other Frameworks**
314 +---------------------+-------------------------------------------------------+
315 | **Framework** | **Description** |
317 +---------------------+-------------------------------------------------------+
318 | *ApexLake* | Experimental framework that enables the user to |
319 | | validate NFVI from the perspective of a VNF. |
320 | | A virtual Traffic Classifier is utilized as VNF. |
321 | | Enables experiments with SR-IOV on Compute Node. |
323 +---------------------+-------------------------------------------------------+
326 **Test Results Output**
328 +-----------------------------+-----------------------------------------------+
329 | **Dispatcher** | **Description** |
331 +-----------------------------+-----------------------------------------------+
332 | file | Log to a file. |
334 +-----------------------------+-----------------------------------------------+
335 | http | Post data to html. |
337 +-----------------------------+-----------------------------------------------+
338 | influxdb | Post data to influxDB. |
340 +-----------------------------+-----------------------------------------------+
346 * Generic NFVI test cases
348 * OPNFV_YARDSTICK_TCOO1 - NW Performance
350 * OPNFV_YARDSTICK_TCOO2 - NW Latency
352 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
354 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
356 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
358 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
360 * OPNFV_YARDSTICK_TCO10 - Memory Latency
362 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
364 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
366 * OPNFV_YARDSTICK_TCO14 - Processing Speed
368 * OPNFV_YARDSTICK_TCO24 - CPU Load
370 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
372 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
375 * OPNFV_YARDSTICK_TCO42 - Network Performance
377 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
379 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
381 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
383 * OPNFV_YARDSTICK_TCO61 - Network Utilization
385 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
387 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
389 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
392 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
394 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
397 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
399 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
401 * Test Cases for OPNFV HA Project:
403 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
405 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
407 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
410 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
412 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
415 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
418 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
421 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
424 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
427 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
430 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
433 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
435 * Test Case for OPNFV IPv6 Project:
437 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
439 * Test Case for OPNFV KVM Project:
441 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
443 * Test Case for OPNFV Parser Project:
445 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
447 * Test Case for OPNFV StorPerf Project:
449 * OPNFV_YARDSTICK_TCO74 - Storperf
451 * Test Cases for Virtual Traffic Classifier:
453 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
456 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
457 Benchmarking in presence of noisy neighbors Test
459 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
461 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
462 presence of noisy neighbors Test
468 Module Version Changes
469 ----------------------
471 This is the second tracked release of Yardstick. It is based on following
478 - OpenDaylight Beryllium
481 Document Version Changes
482 ------------------------
484 This is the second tracked version of the Yardstick framework in OPNFV.
485 It includes the following documentation updates:
487 - Yardstick User Guide: added yardstick plugin chapter; added Store Other
488 Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter.
490 - Yardstick Code Documentation: no changes
492 - Yardstick Release Notes for Yardstick: this document
494 - Test Results report for Colorado testing with Yardstick: updated listed of
495 verified scenarios and limitations
501 - Yardstick reporting
502 - StorPerf Integration
508 For Colorado 3.0, Yardstick was tested on the following scenarios:
510 +-------------------------+---------+---------+---------+---------+
511 | Scenario | Apex | Compass | Fuel | Joid |
512 +=========================+=========+=========+=========+=========+
513 | os-nosdn-nofeature-noha | | | | X |
514 +-------------------------+---------+---------+---------+---------+
515 | os-nosdn-nofeature-ha | X | | X | X |
516 +-------------------------+---------+---------+---------+---------+
517 | os-odl_l2-nofeature-ha | X | X | X | X |
518 +-------------------------+---------+---------+---------+---------+
519 | os-odl_l2-nofeature-noha| | X | | |
520 +-------------------------+---------+---------+---------+---------+
521 | os-odl_l3-nofeature-ha | X | | X | |
522 +-------------------------+---------+---------+---------+---------+
523 | os-odl_l3-nofeature-ha | | X | | |
524 +-------------------------+---------+---------+---------+---------+
525 | os-onos-sfc-ha | X | | X | X |
526 +-------------------------+---------+---------+---------+---------+
527 | os-onos-nofeature-ha | X | | X | X |
528 +-------------------------+---------+---------+---------+---------+
529 | os-onos-nofeature-noha | | X | | |
530 +-------------------------+---------+---------+---------+---------+
531 | os-odl_l2-sfc-ha | | | X | |
532 +-------------------------+---------+---------+---------+---------+
533 | os-odl_l2-sfc-noha | X | X | | |
534 +-------------------------+---------+---------+---------+---------+
535 | os-odl_l2-bgpvpn-ha | X | | X | |
536 +-------------------------+---------+---------+---------+---------+
537 | os-odl_l2-bgpvpn-noha | | X | | |
538 +-------------------------+---------+---------+---------+---------+
539 | os-nosdn-kvm-ha | | | X | |
540 +-------------------------+---------+---------+---------+---------+
541 | os-nosdn-kvm-noha | | X | | |
542 +-------------------------+---------+---------+---------+---------+
543 | os-nosdn-ovs-ha | | | X | |
544 +-------------------------+---------+---------+---------+---------+
545 | os-nosdn-ovs-noha | X | | X | |
546 +-------------------------+---------+---------+---------+---------+
547 | os-ocl-nofeature-ha | | | | |
548 +-------------------------+---------+---------+---------+---------+
549 | os-nosdn-lxd-ha | | | | X |
550 +-------------------------+---------+---------+---------+---------+
551 | os-nosdn-lxd-noha | | | | X |
552 +-------------------------+---------+---------+---------+---------+
553 | os-odl_l2-fdio-noha | X | | | |
554 +-------------------------+---------+---------+---------+---------+
560 Test results are available in:
562 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
564 The reporting pages can be found at:
566 * apex: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html
567 * compass: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html
568 * fuel: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-fuel.html
569 * joid: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-joid.html
571 You can get additional details through test logs on http://artifacts.opnfv.org/.
572 As no search engine is available on the OPNFV artifact web site you must
573 retrieve the pod identifier on which the tests have been executed (see
574 field pod in any of the results) then click on the selected POD and look
575 for the date of the test you are interested in.
580 - Floating IP not supported in bgpvpn scenario
581 - Floating IP not supported in apex-os-odl_l3-nofeature-ha scenario
583 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
584 which were not fully verified, are listed in the OPNFV installer's release
593 +----------------------------+------------------------------------------------+
594 | **JIRA REFERENCE** | **SLOGAN** |
596 +----------------------------+------------------------------------------------+
597 | JIRA: YARDSTICK-239 | Define process for working with Yardstick |
598 | | Grafana dashboard. |
600 +----------------------------+------------------------------------------------+
601 | JIRA: YARDSTICK-373 | Add os-odl_l2-fdio-ha scenario support. |
603 +----------------------------+------------------------------------------------+
608 +----------------------------+------------------------------------------------+
609 | **JIRA REFERENCE** | **SLOGAN** |
611 +----------------------------+------------------------------------------------+
612 | JIRA: YARDSTICK-325 | Provide raw format yardstick vm image for |
613 | | nova-lxd scenario. |
615 +----------------------------+------------------------------------------------+
616 | JIRA: YARDSTICK-358 | tc027 ipv6 test case to de-coupling to the |
619 +----------------------------+------------------------------------------------+
620 | JIRA: YARDSTICK-359 | ipv6 testcase disable port-security on |
623 +----------------------------+------------------------------------------------+
624 | JIRA: YARDSTICK-363 | ipv6 testcase to support fuel. |
626 +----------------------------+------------------------------------------------+
627 | JIRA: YARDSTICK-367 | Add d3 graph presentation to yardstick |
630 +----------------------------+------------------------------------------------+
631 | JIRA: YARDSTICK-371 | Provide raw format yardstick vm image for |
632 | | nova-lxd scenario. |
634 +----------------------------+------------------------------------------------+
635 | JIRA: YARDSTICK-372 | cannot find yardstick-img-dpdk-modify and |
636 | | yardstick-img-lxd-modify in environment |
639 +----------------------------+------------------------------------------------+
642 Colorado 3.0 known restrictions/issues
643 ==================================
644 +-----------+-----------+----------------------------------------------+
645 | Installer | Scenario | Issue |
646 +===========+===========+==============================================+
647 | any | *-bgpvpn | Floating ips not supported. Some Test cases |
648 | | | related to floating ips are excluded. |
649 +-----------+-----------+----------------------------------------------+
650 | any | odl_l3-* | Some test cases related to using floating IP |
651 | | | addresses fail because of a known ODL bug. |
652 | | | https://jira.opnfv.org/browse/APEX-112 |
653 +-----------+-----------+----------------------------------------------+
663 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
665 - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
667 - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
669 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
671 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
673 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
675 - Yardstick IRC chanel: #opnfv-yardstick
677 .. _`YARDSTICK-239` : https://jira.opnfv.org/browse/YARDSTICK-239
679 .. _`YARDSTICK-325` : https://jira.opnfv.org/browse/YARDSTICK-325
681 .. _`YARDSTICK-358` : https://jira.opnfv.org/browse/YARDSTICK-358
683 .. _`YARDSTICK-359` : https://jira.opnfv.org/browse/YARDSTICK-359
685 .. _`YARDSTICK-363` : https://jira.opnfv.org/browse/YARDSTICK-363
687 .. _`YARDSTICK-367` : https://jira.opnfv.org/browse/YARDSTICK-367
689 .. _`YARDSTICK-371` : https://jira.opnfv.org/browse/YARDSTICK-371
691 .. _`YARDSTICK-372` : https://jira.opnfv.org/browse/YARDSTICK-372
693 .. _`YARDSTICK-373` : https://jira.opnfv.org/browse/YARDSTICK-373