1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
4 .. (c) OPNFV, Ericsson AB and others.
7 ============================================
8 OPNFV Brahmaputra Release Note for Yardstick
9 ============================================
14 .. _Yardstick: https://wiki.opnfv.org/yardstick
16 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
18 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
24 This document compiles the release notes for the OPNFV Brahmaputra release
25 for Yardstick framework as well as Yardstick_ Project deliverables.
30 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
31 experimental framework are opensource software, licensed under the terms of the
32 Apache License, Version 2.0.
38 +---------------+--------------------+---------------------------------+
39 | *Date* | *Version* | *Comment* |
41 +---------------+--------------------+---------------------------------+
42 | Apr 27th,2016 | 3.0 | Brahmaputra release |
44 +---------------+--------------------+---------------------------------+
45 | Mar 30th,2016 | 2.0 | Brahmaputra release |
47 +---------------+--------------------+---------------------------------+
48 | Feb 25th,2016 | 1.0 | Brahmaputra release |
50 +---------------+--------------------+---------------------------------+
56 The software delivered in the OPNFV Yardstick_ Project, comprising the
57 *Yardstick framework*, the *Yardstick test cases* and the experimental
58 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
61 The *Yardstick* framework is *installer*, *infrastructure* and *application*
68 This Brahmaputra release provides *Yardstick* as a framework for NFVI testing
69 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
71 * Documentation generated with Sphinx
77 * Release notes (this document)
81 * Automated Yardstick test suite (daily, weekly)
83 * Jenkins Jobs for OPNFV community labs
85 * Automated Yardstick test results visualization
87 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB used as
90 * Yardstick framework source code
92 * Yardstick test cases yaml files
94 For Brahmaputra release, the *Yardstick framework* is used for the following
97 * OPNFV platform testing - generic test cases to measure the categories:
105 * Test cases for the following OPNFV Projects:
115 * Test cases added in Brahmaputra2.0:
117 * virtual Traffic Classifier
119 The *Yardstick framework* is developed in the OPNFV community, by the
120 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
123 .. note:: The test case description template used for the Yardstick test cases
124 is based on the document ETSI-ISG NFV-TST001_; the results report template
125 used for the Yardstick results is based on the IEEE Std 829-2008.
131 +--------------------------------------+--------------------------------------+
132 | **Project** | Yardstick |
134 +--------------------------------------+--------------------------------------+
135 | **Repo/tag** | yardstick/brahmaputra.3.0 |
137 +--------------------------------------+--------------------------------------+
138 | **Yardstick Docker image tag** | brahmaputra.3.0 |
140 +--------------------------------------+--------------------------------------+
141 | **Release designation** | Brahmaputra |
143 +--------------------------------------+--------------------------------------+
144 | **Release date** | Apr 27th, 2016 |
146 +--------------------------------------+--------------------------------------+
147 | **Purpose of the delivery** | OPNFV Brahmaputra release |
149 +--------------------------------------+--------------------------------------+
155 Module Version Changes
156 ~~~~~~~~~~~~~~~~~~~~~~
158 This is the third tracked release of Yardstick. It is based on following
163 - OpenDaylight Beryllium
166 Document Version Changes
167 ~~~~~~~~~~~~~~~~~~~~~~~~
169 This is the third tracked version of the Yardstick framework in OPNFV.
170 It includes the following documentation updates:
172 - Yardstick User Guide: corrected faulty links
174 - Yardstick Code Documentation: no changes
176 - Yardstick Release Notes for Yardstick: this document
178 - Test Results report for Brahmaputra testing with Yardstick: updated listed of
179 verified scenarios and limitations
181 Documentation updates on the second tracked version:
183 - Yardstick User Guide: added software architecture chapter
185 - Yardstick Code Documentation: no changes
187 - Yardstick Release Notes for Yardstick: this document
189 - Test Results report for Brahmaputra testing with Yardstick: added test cases
190 and results for virtual Traffic Classifier
203 +----------------------------+------------------------------------------------+
204 | **JIRA REFERENCE** | **SLOGAN** |
206 +----------------------------+------------------------------------------------+
207 | JIRA: YARDSTICK-227 | Heat HTTPS SSL support. |
209 +----------------------------+------------------------------------------------+
219 +----------------------------+------------------------------------------------+
220 | **JIRA REFERENCE** | **SLOGAN** |
222 +----------------------------+------------------------------------------------+
223 | JIRA: - | Change copyrights for base scenario, runners, |
224 | | dispatchers, cover. |
226 +----------------------------+------------------------------------------------+
227 | JIRA: - | Update setup.py and dependencies |
229 +----------------------------+------------------------------------------------+
230 | JIRA: - | Add missing dependencies to docker file |
232 +----------------------------+------------------------------------------------+
233 | JIRA: - | Fix Heat template for noisy neighbors deploy |
235 +----------------------------+------------------------------------------------+
241 +----------------------------+------------------------------------------------+
242 | **JIRA REFERENCE** | **SLOGAN** |
244 +----------------------------+------------------------------------------------+
245 | JIRA: YARDSTICK-175 | Running test suite, if a test cases running |
246 | | failed, the test is stopped. |
248 +----------------------------+------------------------------------------------+
249 | JIRA: YARDSTICK-176 | Fix plotter bug since Output format has been |
252 +----------------------------+------------------------------------------------+
253 | JIRA: YARDSTICK-216 | ArgsAlreadyParsedError: arguments already |
254 | | parsed: cannot register CLI option. |
256 +----------------------------+------------------------------------------------+
257 | JIRA: YARDSTICK-231 | Installation instructions on Wiki not accurate |
259 +----------------------------+------------------------------------------------+
261 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
262 which were not fully verified, are listed in the OPNFV installer's release
269 Software Deliverables
270 ~~~~~~~~~~~~~~~~~~~~~
272 **Yardstick framework source code <brahmaputra.3.0>**
274 +--------------------------------------+--------------------------------------+
275 | **Project** | Yardstick |
277 +--------------------------------------+--------------------------------------+
278 | **Repo/tag** | yardstick/brahmaputra.3.0 |
280 +--------------------------------------+--------------------------------------+
281 | **Yardstick Docker image tag** | brahmaputra.3.0 |
283 +--------------------------------------+--------------------------------------+
284 | **Release designation** | Brahmaputra |
286 +--------------------------------------+--------------------------------------+
287 | **Release date** | Apr 27th, 2016 |
289 +--------------------------------------+--------------------------------------+
290 | **Purpose of the delivery** | OPNFV Brahmaputra release |
292 +--------------------------------------+--------------------------------------+
296 +---------------------+-------------------------------------------------------+
297 | **Context** | **Description** |
299 +---------------------+-------------------------------------------------------+
300 | *Heat* | Models orchestration using OpenStack Heat |
302 +---------------------+-------------------------------------------------------+
303 | *Node* | Models Baremetal, Controller, Compute |
305 +---------------------+-------------------------------------------------------+
310 +---------------------+-------------------------------------------------------+
311 | **Runner** | **Description** |
313 +---------------------+-------------------------------------------------------+
314 | *Arithmetic* | Steps every run arithmetically according to specified |
317 +---------------------+-------------------------------------------------------+
318 | *Duration* | Runs for a specified period of time |
320 +---------------------+-------------------------------------------------------+
321 | *Iteration* | Runs for a specified number of iterations |
323 +---------------------+-------------------------------------------------------+
324 | *Sequence* | Selects input value to a scenario from an input file |
325 | | and runs all entries sequentially |
327 +---------------------+-------------------------------------------------------+
331 +---------------------+-------------------------------------------------------+
332 | **Category** | **Delivered** |
334 +---------------------+-------------------------------------------------------+
335 | *Availability* | Attacker: |
337 | | * baremetal, process |
341 | | * check host, openstack, process, service |
343 | | * start/stop service |
347 | | * command, process |
349 +---------------------+-------------------------------------------------------+
350 | *Compute* | * cpuload |
360 +---------------------+-------------------------------------------------------+
361 | *Networking* | * iperf3 |
373 | | * sfc with tacker |
375 | | * vtc instantion validation |
377 | | * vtc instantion validation with noisy neighbors |
379 | | * vtc throughput |
381 | | * vtc throughput in the presence of noisy neighbors |
383 +---------------------+-------------------------------------------------------+
384 | *Parser* | Tosca2Heat |
386 +---------------------+-------------------------------------------------------+
389 +---------------------+-------------------------------------------------------+
391 **API to Other Frameworks**
393 +---------------------+-------------------------------------------------------+
394 | **Framework** | **Description** |
396 +---------------------+-------------------------------------------------------+
397 | *ApexLake* | Experimental framework that enables the user to |
398 | | validate NFVI from the perspective of a VNF. |
399 | | A virtual Traffic Classifier is utilized as VNF. |
400 | | Enables experiments with SR-IOV on Compute Node. |
402 +---------------------+-------------------------------------------------------+
404 **Test Results Output**
406 +-----------------------------+-----------------------------------------------+
407 | **Dispatcher** | **Description** |
409 +-----------------------------+-----------------------------------------------+
410 | file | Log to a file. |
412 +-----------------------------+-----------------------------------------------+
413 | http | Post data to html. |
415 +-----------------------------+-----------------------------------------------+
416 | influxdb | Post data to influxdB. |
418 +-----------------------------+-----------------------------------------------+
424 * Generic NFVI test cases
426 * OPNFV_YARDSTICK_TCOO1 - NW Performance
428 * OPNFV_YARDSTICK_TCOO2 - NW Latency
430 * OPNFV_YARDSTICK_TCOO5 - Storage Performance
432 * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
434 * OPNFV_YARDSTICK_TCOO9 - Packet Loss
436 * OPNFV_YARDSTICK_TCO10 - Memory Latency
438 * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
440 * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
442 * OPNFV_YARDSTICK_TCO14 - Processing Speed
444 * OPNFV_YARDSTICK_TCO24 - CPU Load
446 * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
448 * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
452 * Test Cases for OPNFV HA Project:
454 * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
456 * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
458 * Test Case for OPNFV IPv6 Project:
460 * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
462 * Test Case for OPNFV KVM Project:
464 * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
466 * Test Case for OPNFV Parser Project:
468 * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
470 * Test Cases for Virtual Traffic Classifier:
472 * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
475 * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
476 Benchmarking in presence of noisy neighbors Test
478 * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
480 * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
481 presence of noisy neighbors Test