4 OPNFV Euphrates release note for Yardstick Docs
5 are licensed under a Creative Commons Attribution 4.0 International License.
6 You should have received a copy of the license along with this.
7 If not, see <http://creativecommons.org/licenses/by/4.0/>.
9 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
10 licensed under the terms of the Apache License, Version 2.0.
12 OPNFV Euphrates Release Note for Yardstick
13 ==========================================
18 .. _Yardstick: https://wiki.opnfv.org/yardstick
20 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
22 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
28 This document describes the release note of Yardstick project.
33 +------------------+-----------+---------------------------------+
34 | *Date* | *Version* | *Comment* |
36 +------------------+-----------+---------------------------------+
37 | October 20, 2017 | 5.0.0 | Yardstick for Euphrates release |
39 +------------------+-----------+---------------------------------+
45 The software delivered in the OPNFV Yardstick_ Project, comprising the
46 *Yardstick framework*, the *Yardstick test cases* and the experimental
47 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
50 The *Yardstick* framework is *installer*, *infrastructure* and *application*
53 OPNFV Euphrates Release
54 -----------------------
56 This Euphrates release provides *Yardstick* as a framework for NFVI testing
57 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
59 * Documentation generated with Sphinx
65 * Release notes (this document)
69 * Automated Yardstick test suite (daily, weekly)
71 * Jenkins Jobs for OPNFV community labs
73 * Automated Yardstick test results visualization
75 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
78 * Yardstick framework source code
80 * Yardstick test cases yaml files
82 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
84 For Euphrates release, the *Yardstick framework* is used for the following
87 * OPNFV platform testing - generic test cases to measure the categories:
95 * OPNFV platform network service benchmarking(NSB)
99 * Test cases for the following OPNFV Projects:
113 * virtual Traffic Classifier
115 The *Yardstick framework* is developed in the OPNFV community, by the
118 .. note:: The test case description template used for the Yardstick test cases
119 is based on the document ETSI-ISG NFV-TST001_; the results report template
120 used for the Yardstick results is based on the IEEE Std 829-2008.
126 +--------------------------------+-----------------------+
127 | **Project** | Yardstick |
129 +--------------------------------+-----------------------+
130 | **Repo/tag** | yardstick/opnfv-5.0.0 |
132 +--------------------------------+-----------------------+
133 | **Yardstick Docker image tag** | opnfv-5.0.0 |
135 +--------------------------------+-----------------------+
136 | **Release designation** | Euphrates |
138 +--------------------------------+-----------------------+
139 | **Release date** | October 20, 2017 |
141 +--------------------------------+-----------------------+
142 | **Purpose of the delivery** | OPNFV Euphrates 5.0.0 |
144 +--------------------------------+-----------------------+
153 - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html
155 - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html
158 Software Deliverables
159 ^^^^^^^^^^^^^^^^^^^^^
162 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.0.0)
168 +--------------+-------------------------------------------+
169 | **Context** | **Description** |
171 +--------------+-------------------------------------------+
172 | *Heat* | Models orchestration using OpenStack Heat |
174 +--------------+-------------------------------------------+
175 | *Node* | Models Baremetal, Controller, Compute |
177 +--------------+-------------------------------------------+
178 | *Standalone* | Models VM running on Non-Managed NFVi |
180 +--------------+-------------------------------------------+
181 | *Kubernetes* | Models VM running on Non-Managed NFVi |
183 +--------------+-------------------------------------------+
189 +--------------+-------------------------------------------------------+
190 | **Runner** | **Description** |
192 +--------------+-------------------------------------------------------+
193 | *Arithmetic* | Steps every run arithmetically according to specified |
196 +--------------+-------------------------------------------------------+
197 | *Duration* | Runs for a specified period of time |
199 +--------------+-------------------------------------------------------+
200 | *Iteration* | Runs for a specified number of iterations |
202 +--------------+-------------------------------------------------------+
203 | *Sequence* | Selects input value to a scenario from an input file |
204 | | and runs all entries sequentially |
206 +--------------+-------------------------------------------------------+
212 +----------------+-----------------------------------------------------+
213 | **Category** | **Delivered** |
215 +----------------+-----------------------------------------------------+
216 | *Availability* | Attacker: |
218 | | * baremetal, process |
222 | | * check host, openstack, process, service |
224 | | * start/stop service |
228 | | * command, process |
230 +----------------+-----------------------------------------------------+
231 | *Compute* | * cpuload |
237 | | * lmbench_cache |
249 | | * computecapacity |
251 +----------------+-----------------------------------------------------+
252 | *Networking* | * iperf3 |
266 | | * sfc with tacker |
268 | | * vtc instantion validation |
270 | | * vtc instantion validation with noisy neighbors |
272 | | * vtc throughput |
274 | | * vtc throughput in the presence of noisy neighbors |
276 | | * networkcapacity |
278 | | * netutilization |
282 +----------------+-----------------------------------------------------+
283 | *Parser* | Tosca2Heat |
285 +----------------+-----------------------------------------------------+
288 | | storagecapacity |
290 +----------------+-----------------------------------------------------+
291 | *StorPerf* | storperf |
293 +----------------+-----------------------------------------------------+
294 | *NSB* | vPE thoughput test case |
296 +----------------+-----------------------------------------------------+
308 Module Version Changes
309 ^^^^^^^^^^^^^^^^^^^^^^
311 This is the fourth tracked release of Yardstick. It is based on following
317 Document Version Changes
318 ^^^^^^^^^^^^^^^^^^^^^^^^
320 This is the fourth tracked version of the Yardstick framework in OPNFV.
321 It includes the following documentation updates:
323 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
324 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
325 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
327 - Yardstick Developer Guide
329 - Yardstick Release Notes for Yardstick: this document
335 - Yardstick RESTful API support
337 - Introduce Network service benchmarking
339 - Introduce stress testing with Bottlenecks team
341 - Yardstick framework improvement:
343 - Parellel test cases execution support
345 - yardstick report CLI
347 - Node context support OpenStack configuration via Ansible
357 For Euphrates 5.0.0, Yardstick was tested on the following scenarios:
359 +--------------------------+------+---------+------+------+
360 | Scenario | Apex | Compass | Fuel | Joid |
361 +==========================+======+=========+======+======+
362 | os-nosdn-nofeature-noha | | | X | X |
363 +--------------------------+------+---------+------+------+
364 | os-nosdn-nofeature-ha | X | X | X | X |
365 +--------------------------+------+---------+------+------+
366 | os-odl_l2-nofeature-ha | | X | X | X |
367 +--------------------------+------+---------+------+------+
368 | os-odl_l2-nofeature-noha | | | X | |
369 +--------------------------+------+---------+------+------+
370 | os-odl_l3-nofeature-ha | X | X | X | |
371 +--------------------------+------+---------+------+------+
372 | os-odl_l3-nofeature-noha | | | X | |
373 +--------------------------+------+---------+------+------+
374 | os-onos-sfc-ha | | | | |
375 +--------------------------+------+---------+------+------+
376 | os-onos-nofeature-ha | | X | | X |
377 +--------------------------+------+---------+------+------+
378 | os-onos-nofeature-noha | | | | |
379 +--------------------------+------+---------+------+------+
380 | os-odl_l2-sfc-ha | | | X | |
381 +--------------------------+------+---------+------+------+
382 | os-odl_l2-sfc-noha | | | X | |
383 +--------------------------+------+---------+------+------+
384 | os-odl_l2-bgpvpn-ha | X | | X | |
385 +--------------------------+------+---------+------+------+
386 | os-odl_l2-bgpvpn-noha | | | X | |
387 +--------------------------+------+---------+------+------+
388 | os-nosdn-kvm-ha | X | | X | |
389 +--------------------------+------+---------+------+------+
390 | os-nosdn-kvm-noha | | | X | |
391 +--------------------------+------+---------+------+------+
392 | os-nosdn-ovs-ha | | | X | |
393 +--------------------------+------+---------+------+------+
394 | os-nosdn-ovs-noha | | | X | |
395 +--------------------------+------+---------+------+------+
396 | os-ocl-nofeature-ha | | X | | |
397 +--------------------------+------+---------+------+------+
398 | os-nosdn-lxd-ha | | | | X |
399 +--------------------------+------+---------+------+------+
400 | os-nosdn-lxd-noha | | | | X |
401 +--------------------------+------+---------+------+------+
402 | os-nosdn-fdio-ha | X | | | |
403 +--------------------------+------+---------+------+------+
404 | os-odl_l2-fdio-noha | X | | | |
405 +--------------------------+------+---------+------+------+
406 | os-odl-gluon-noha | X | | | |
407 +--------------------------+------+---------+------+------+
408 | os-nosdn-openo-ha | | X | | |
409 +--------------------------+------+---------+------+------+
410 | os-nosdn-kvm_ovs_dpdk | | | X | |
412 +--------------------------+------+---------+------+------+
413 | os-nosdn-kvm_ovs_dpdk-ha | | | X | |
414 +--------------------------+------+---------+------+------+
415 | os-nosdn-kvm_ovs_dpdk | | | X | |
417 +--------------------------+------+---------+------+------+
418 | os-nosdn-kvm_ovs_dpdk | | | X | |
419 | _bar-noha | | | | |
420 +--------------------------+------+---------+------+------+
421 | opnfv_os-ovn-nofeature- | X | | | |
422 | noha_daily | | | | |
423 +--------------------------+------+---------+------+------+
428 Test results are available in:
430 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
432 The reporting pages can be found at:
434 +---------------+-------------------------------------------------------------------------------------+
435 | apex | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html |
436 +---------------+-------------------------------------------------------------------------------------+
437 | compass | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html |
438 +---------------+-------------------------------------------------------------------------------------+
439 | fuel\@x86 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html |
440 +---------------+-------------------------------------------------------------------------------------+
441 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html |
442 +---------------+-------------------------------------------------------------------------------------+
443 | joid | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html |
444 +---------------+-------------------------------------------------------------------------------------+
455 +---------------------+--------------------------------------------+
456 | **JIRA REFERENCE** | **DESCRIPTION** |
458 +---------------------+--------------------------------------------+
459 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
461 +---------------------+--------------------------------------------+
462 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
464 +---------------------+--------------------------------------------+
467 Euphratess 5.0.0 known restrictions/issues
468 ------------------------------------------
469 +-----------+-----------+----------------------------------------------+
470 | Installer | Scenario | Issue |
471 +===========+===========+==============================================+
472 | any | \*-bgpvpn | Floating ips not supported. Some Test cases |
473 | | | related to floating ips are excluded. |
474 +-----------+-----------+----------------------------------------------+
475 | any | odl_l3-\* | Some test cases related to using floating IP |
476 | | | addresses fail because of a known ODL bug. |
478 +-----------+-----------+----------------------------------------------+
479 | compass | odl_l2-\* | In some test cases, VM instance will failed |
480 | | | raising network interfaces. |
482 +-----------+-----------+----------------------------------------------+
488 +---------------------+----------------------------------------------+
489 | **JIRA REFERENCE** | **DESCRIPTION** |
491 +---------------------+----------------------------------------------+
492 | JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 |
494 +---------------------+----------------------------------------------+
495 | JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
496 | | (which actually pingable) |
497 +---------------------+----------------------------------------------+
498 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
500 +---------------------+----------------------------------------------+
506 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
508 - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning
510 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
512 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
514 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
516 - Yardstick IRC chanel: #opnfv-yardstick