1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, Ericsson AB and others.
7 ============================================
8 OPNFV Colorado Release Note for Yardstick
9 ============================================
14 .. _Yardstick: https://wiki.opnfv.org/yardstick
16 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
18 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
24 This document describes the release note of Yardstick project.
30 OPNFV Colorado release note for Yardstick Docs
31 are licensed under a Creative Commons Attribution 4.0 International License.
32 You should have received a copy of the license along with this.
33 If not, see <http://creativecommons.org/licenses/by/4.0/>.
35 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
36 experimental framework are opensource software, licensed under the terms of the
37 Apache License, Version 2.0.
43 +----------------+--------------------+---------------------------------+
44 | *Date* | *Version* | *Comment* |
46 +----------------+--------------------+---------------------------------+
47 | Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release |
49 +----------------+--------------------+---------------------------------+
55 The software delivered in the OPNFV Yardstick_ Project, comprising the
56 *Yardstick framework*, the *Yardstick test cases* and the experimental
57 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
60 The *Yardstick* framework is *installer*, *infrastructure* and *application*
67 This Colorado release provides *Yardstick* as a framework for NFVI testing
68 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
70 * Documentation generated with Sphinx
76 * Release notes (this document)
80 * Automated Yardstick test suite (daily, weekly)
82 * Jenkins Jobs for OPNFV community labs
84 * Automated Yardstick test results visualization
86 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
89 * Yardstick framework source code
91 * Yardstick test cases yaml files
93 * Yardstick pliug-in configration yaml files, plug-in install/remove scripts
95 For Colorado release, the *Yardstick framework* is used for the following
98 * OPNFV platform testing - generic test cases to measure the categories:
106 * Test cases for the following OPNFV Projects:
120 * virtual Traffic Classifier
122 The *Yardstick framework* is developed in the OPNFV community, by the
123 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
126 .. note:: The test case description template used for the Yardstick test cases
127 is based on the document ETSI-ISG NFV-TST001_; the results report template
128 used for the Yardstick results is based on the IEEE Std 829-2008.
134 +--------------------------------------+--------------------------------------+
135 | **Project** | Yardstick |
137 +--------------------------------------+--------------------------------------+
138 | **Repo/tag** | yardstick/colorado.1.0 |
140 +--------------------------------------+--------------------------------------+
141 | **Yardstick Docker image tag** | colorado.1.0 |
143 +--------------------------------------+--------------------------------------+
144 | **Release designation** | Colorado base release |
146 +--------------------------------------+--------------------------------------+
147 | **Release date** | September 22 2016 |
149 +--------------------------------------+--------------------------------------+
150 | **Purpose of the delivery** | Colorado base release |
152 +--------------------------------------+--------------------------------------+
158 Software Deliverables
159 ---------------------
161 **Yardstick framework source code <colorado.1.0>**
163 +--------------------------------------+--------------------------------------+
164 | **Project** | Yardstick |
166 +--------------------------------------+--------------------------------------+
167 | **Repo/tag** | yardstick/colorado.1.0 |
169 +--------------------------------------+--------------------------------------+
170 | **Yardstick Docker image tag** | colorado.1.0 |
172 +--------------------------------------+--------------------------------------+
173 | **Release designation** | colorado |
175 +--------------------------------------+--------------------------------------+
176 | **Release date** | September 22th, 2016 |
178 +--------------------------------------+--------------------------------------+
179 | **Purpose of the delivery** | OPNFV Colorado release |
181 +--------------------------------------+--------------------------------------+
186 +---------------------+-------------------------------------------------------+
187 | **Context** | **Description** |
189 +---------------------+-------------------------------------------------------+
190 | *Heat* | Models orchestration using OpenStack Heat |
192 +---------------------+-------------------------------------------------------+
193 | *Node* | Models Baremetal, Controller, Compute |
195 +---------------------+-------------------------------------------------------+
200 +---------------------+-------------------------------------------------------+
201 | **Runner** | **Description** |
203 +---------------------+-------------------------------------------------------+
204 | *Arithmetic* | Steps every run arithmetically according to specified |
207 +---------------------+-------------------------------------------------------+
208 | *Duration* | Runs for a specified period of time |
210 +---------------------+-------------------------------------------------------+
211 | *Iteration* | Runs for a specified number of iterations |
213 +---------------------+-------------------------------------------------------+
214 | *Sequence* | Selects input value to a scenario from an input file |
215 | | and runs all entries sequentially |
217 +---------------------+-------------------------------------------------------+
222 +---------------------+-------------------------------------------------------+
223 | **Category** | **Delivered** |
225 +---------------------+-------------------------------------------------------+
226 | *Availability* | Attacker: |
228 | | * baremetal, process |
232 | | * check host, openstack, process, service |
234 | | * start/stop service |
238 | | * command, process |
240 +---------------------+-------------------------------------------------------+
241 | *Compute* | * cpuload |
247 | | * lmbench_cache |
259 | | * computecapacity |
261 +---------------------+-------------------------------------------------------+
262 | *Networking* | * iperf3 |
276 | | * sfc with tacker |
278 | | * vtc instantion validation |
280 | | * vtc instantion validation with noisy neighbors |
282 | | * vtc throughput |
284 | | * vtc throughput in the presence of noisy neighbors |
286 | | * networkcapacity |
288 | | * netutilization |
290 +---------------------+-------------------------------------------------------+
291 | *Parser* | Tosca2Heat |
293 +---------------------+-------------------------------------------------------+
296 | | storagecapacity |
298 +---------------------+-------------------------------------------------------+
299 | *StorPerf* | storperf |
301 +---------------------+-------------------------------------------------------+
304 **API to Other Frameworks**
306 +---------------------+-------------------------------------------------------+
307 | **Framework** | **Description** |
309 +---------------------+-------------------------------------------------------+
310 | *ApexLake* | Experimental framework that enables the user to |
311 | | validate NFVI from the perspective of a VNF. |
312 | | A virtual Traffic Classifier is utilized as VNF. |
313 | | Enables experiments with SR-IOV on Compute Node. |
315 +---------------------+-------------------------------------------------------+
318 **Test Results Output**
320 +-----------------------------+-----------------------------------------------+
321 | **Dispatcher** | **Description** |
323 +-----------------------------+-----------------------------------------------+
324 | file | Log to a file. |
326 +-----------------------------+-----------------------------------------------+
327 | http | Post data to html. |
329 +-----------------------------+-----------------------------------------------+
330 | influxdb | Post data to influxDB. |
332 +-----------------------------+-----------------------------------------------+
338 * Generic NFVI test cases
340 * OPNFV_YARDSTICK_TCOO1 - NW Performance
342 * OPNFV_YARDSTICK_TCOO2 - NW Latency
344 * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
346 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
348 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
350 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
352 * OPNFV_YARDSTICK_TCO10 - Memory Latency
354 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
356 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
358 * OPNFV_YARDSTICK_TCO14 - Processing Speed
360 * OPNFV_YARDSTICK_TCO24 - CPU Load
362 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
364 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
367 * OPNFV_YARDSTICK_TCO42 - Network Performance
369 * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
371 * OPNFV_YARDSTICK_TCO44 - Memory Utilization
373 * OPNFV_YARDSTICK_TCO55 - Compute Capacity
375 * OPNFV_YARDSTICK_TCO61 - Network Utilization
377 * OPNFV_YARDSTICK_TCO63 - Storage Capacity
379 * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
381 * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
384 * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
386 * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
389 * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
391 * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
393 * Test Cases for OPNFV HA Project:
395 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
397 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
399 * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
402 * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
404 * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
407 * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
410 * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
413 * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
416 * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
419 * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
422 * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
425 * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
427 * Test Case for OPNFV IPv6 Project:
429 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
431 * Test Case for OPNFV KVM Project:
433 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
435 * Test Case for OPNFV Parser Project:
437 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
439 * Test Case for OPNFV StorPerf Project:
441 * OPNFV_YARDSTICK_TCO74 - Storperf
443 * Test Cases for Virtual Traffic Classifier:
445 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
448 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
449 Benchmarking in presence of noisy neighbors Test
451 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
453 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
454 presence of noisy neighbors Test
460 Module Version Changes
461 ----------------------
463 This is the third tracked release of Yardstick. It is based on following
468 - OpenDaylight Beryllium
471 Document Version Changes
472 ------------------------
474 This is the first tracked version of the Yardstick framework in OPNFV.
475 It includes the following documentation updates:
477 - Yardstick User Guide: added yardstick plugin chapter; added Store Other
478 Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter.
480 - Yardstick Code Documentation: no changes
482 - Yardstick Release Notes for Yardstick: this document
484 - Test Results report for Colorado testing with Yardstick: updated listed of
485 verified scenarios and limitations
503 - Boot up VM failed in joid-os-nosdn-lxd-ha and joid-os-nosdn-lxd-noha scenarios
504 - Yardstick CI job timeout in fuel-os-onos-nofeature-ha scenario
507 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
508 which were not fully verified, are listed in the OPNFV installer's release
512 Colorado known restrictions/issues
513 ==================================
514 +-----------+-----------+----------------------------------------------+
515 | Installer | Scenario | Issue |
516 +===========+===========+==============================================+
517 | any | *-bgpvpn | floating ips not supported. Some Test cases |
518 | | | related to floating ips are excluded. |
519 +-----------+-----------+----------------------------------------------+
526 Test results are available in:
528 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
534 +------------------+-----------------------------------------------+
535 | JIRA | Description |
536 +==================+===============================================+
537 +------------------+-----------------------------------------------+
538 +------------------+-----------------------------------------------+
539 +------------------+-----------------------------------------------+
540 +------------------+-----------------------------------------------+
541 +------------------+-----------------------------------------------+
547 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
549 - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
551 - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
553 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
555 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
557 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
559 - Yardstick IRC chanel: #opnfv-yardstick