4 OPNFV Euphrates release note for Yardstick Docs
5 are licensed under a Creative Commons Attribution 4.0 International License.
6 You should have received a copy of the license along with this.
7 If not, see <http://creativecommons.org/licenses/by/4.0/>.
9 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
10 licensed under the terms of the Apache License, Version 2.0.
12 OPNFV Euphrates Release Note for Yardstick
13 ==========================================
18 .. _Yardstick: https://wiki.opnfv.org/yardstick
20 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
22 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
28 This document describes the release note of Yardstick project.
33 +------------------+-----------+---------------------------------+
34 | *Date* | *Version* | *Comment* |
36 +------------------+-----------+---------------------------------+
37 | October 20, 2017 | 5.0.0 | Yardstick for Euphrates release |
39 +------------------+-----------+---------------------------------+
45 The software delivered in the OPNFV Yardstick_ Project, comprising the
46 *Yardstick framework*, the *Yardstick test cases* and the experimental
47 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
50 The *Yardstick* framework is *installer*, *infrastructure* and *application*
53 OPNFV Euphrates Release
54 -----------------------
56 This Euphrates release provides *Yardstick* as a framework for NFVI testing
57 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
59 * Documentation generated with Sphinx
65 * Release notes (this document)
69 * Automated Yardstick test suite (daily, weekly)
71 * Jenkins Jobs for OPNFV community labs
73 * Automated Yardstick test results visualization
75 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
78 * Yardstick framework source code
80 * Yardstick test cases yaml files
82 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
84 For Euphrates release, the *Yardstick framework* is used for the following
87 * OPNFV platform testing - generic test cases to measure the categories:
95 * OPNFV platform network service benchmarking(NSB)
99 * Test cases for the following OPNFV Projects:
115 * virtual Traffic Classifier
117 The *Yardstick framework* is developed in the OPNFV community, by the
120 .. note:: The test case description template used for the Yardstick test cases
121 is based on the document ETSI-ISG NFV-TST001_; the results report template
122 used for the Yardstick results is based on the IEEE Std 829-2008.
128 +--------------------------------+-----------------------+
129 | **Project** | Yardstick |
131 +--------------------------------+-----------------------+
132 | **Repo/tag** | yardstick/opnfv-5.0.0 |
134 +--------------------------------+-----------------------+
135 | **Yardstick Docker image tag** | opnfv-5.0.0 |
137 +--------------------------------+-----------------------+
138 | **Release designation** | Euphrates |
140 +--------------------------------+-----------------------+
141 | **Release date** | October 20, 2017 |
143 +--------------------------------+-----------------------+
144 | **Purpose of the delivery** | OPNFV Euphrates 5.0.0 |
146 +--------------------------------+-----------------------+
155 - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html
157 - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html
160 Software Deliverables
161 ^^^^^^^^^^^^^^^^^^^^^
164 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.0.0)
170 +--------------+-------------------------------------------+
171 | **Context** | **Description** |
173 +--------------+-------------------------------------------+
174 | *Heat* | Models orchestration using OpenStack Heat |
176 +--------------+-------------------------------------------+
177 | *Node* | Models Baremetal, Controller, Compute |
179 +--------------+-------------------------------------------+
180 | *Standalone* | Models VM running on Non-Managed NFVi |
182 +--------------+-------------------------------------------+
183 | *Kubernetes* | Models VM running on Non-Managed NFVi |
185 +--------------+-------------------------------------------+
191 +--------------+-------------------------------------------------------+
192 | **Runner** | **Description** |
194 +--------------+-------------------------------------------------------+
195 | *Arithmetic* | Steps every run arithmetically according to specified |
198 +--------------+-------------------------------------------------------+
199 | *Duration* | Runs for a specified period of time |
201 +--------------+-------------------------------------------------------+
202 | *Iteration* | Runs for a specified number of iterations |
204 +--------------+-------------------------------------------------------+
205 | *Sequence* | Selects input value to a scenario from an input file |
206 | | and runs all entries sequentially |
208 +--------------+-------------------------------------------------------+
214 +----------------+-----------------------------------------------------+
215 | **Category** | **Delivered** |
217 +----------------+-----------------------------------------------------+
218 | *Availability* | Attacker: |
220 | | * baremetal, process |
224 | | * check host, openstack, process, service |
226 | | * start/stop service |
230 | | * command, process |
232 +----------------+-----------------------------------------------------+
233 | *Compute* | * cpuload |
239 | | * lmbench_cache |
251 | | * computecapacity |
255 +----------------+-----------------------------------------------------+
256 | *Networking* | * iperf3 |
270 | | * sfc with tacker |
272 | | * vtc instantion validation |
274 | | * vtc instantion validation with noisy neighbors |
276 | | * vtc throughput |
278 | | * vtc throughput in the presence of noisy neighbors |
280 | | * networkcapacity |
282 | | * netutilization |
288 +----------------+-----------------------------------------------------+
289 | *Parser* | Tosca2Heat |
291 +----------------+-----------------------------------------------------+
296 | | storagecapacity |
298 +----------------+-----------------------------------------------------+
299 | *StorPerf* | storperf |
301 +----------------+-----------------------------------------------------+
302 | *NSB* | vPE thoughput test case |
304 +----------------+-----------------------------------------------------+
311 * Generic NFVI test cases
313 * OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006
315 * OPNFV_YARDSTICK_TCO79 - Bonnie++
317 * Kubernetes Test cases
319 * OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER
321 * OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM
327 Module Version Changes
328 ^^^^^^^^^^^^^^^^^^^^^^
330 This is the fifth tracked release of Yardstick. It is based on following
335 - OpenDayLight Nitrogen
340 Document Version Changes
341 ^^^^^^^^^^^^^^^^^^^^^^^^
343 This is the fifth tracked version of the Yardstick framework in OPNFV.
344 It includes the following documentation updates:
346 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
347 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
348 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
350 - Yardstick Developer Guide
352 - Yardstick Release Notes for Yardstick: this document
358 - Yardstick RESTful API support
360 - Network service benchmarking
362 - Stress testing with Bottlenecks team
364 - Yardstick framework improvement:
366 - yardstick report CLI
368 - Node context support OpenStack configuration via Ansible
372 - Kubernetes context type
374 - Yardstick container local GUI
382 For Euphrates 5.0.0, Yardstick was tested on the following scenarios:
384 +--------------------------+------+---------+------+------+
385 | Scenario | Apex | Compass | Fuel | Joid |
386 +==========================+======+=========+======+======+
387 | os-nosdn-nofeature-noha | | | X | X |
388 +--------------------------+------+---------+------+------+
389 | os-nosdn-nofeature-ha | X | X | X | X |
390 +--------------------------+------+---------+------+------+
391 | os-odl_l2-nofeature-ha | | X | X | X |
392 +--------------------------+------+---------+------+------+
393 | os-odl_l2-nofeature-noha | | | X | |
394 +--------------------------+------+---------+------+------+
395 | os-odl_l3-nofeature-ha | X | X | X | |
396 +--------------------------+------+---------+------+------+
397 | os-odl_l3-nofeature-noha | | | X | |
398 +--------------------------+------+---------+------+------+
399 | os-onos-sfc-ha | | | | |
400 +--------------------------+------+---------+------+------+
401 | os-onos-nofeature-ha | | X | | X |
402 +--------------------------+------+---------+------+------+
403 | os-onos-nofeature-noha | | | | |
404 +--------------------------+------+---------+------+------+
405 | os-odl_l2-sfc-ha | | | X | |
406 +--------------------------+------+---------+------+------+
407 | os-odl_l2-sfc-noha | | | X | |
408 +--------------------------+------+---------+------+------+
409 | os-odl_l2-bgpvpn-ha | X | | X | |
410 +--------------------------+------+---------+------+------+
411 | os-odl_l2-bgpvpn-noha | | | X | |
412 +--------------------------+------+---------+------+------+
413 | os-nosdn-kvm-ha | X | | X | |
414 +--------------------------+------+---------+------+------+
415 | os-nosdn-kvm-noha | | | X | |
416 +--------------------------+------+---------+------+------+
417 | os-nosdn-ovs-ha | | | X | |
418 +--------------------------+------+---------+------+------+
419 | os-nosdn-ovs-noha | | | X | |
420 +--------------------------+------+---------+------+------+
421 | os-ocl-nofeature-ha | | X | | |
422 +--------------------------+------+---------+------+------+
423 | os-nosdn-lxd-ha | | | | X |
424 +--------------------------+------+---------+------+------+
425 | os-nosdn-lxd-noha | | | | X |
426 +--------------------------+------+---------+------+------+
427 | os-nosdn-fdio-ha | X | | | |
428 +--------------------------+------+---------+------+------+
429 | os-odl_l2-fdio-noha | X | | | |
430 +--------------------------+------+---------+------+------+
431 | os-odl-gluon-noha | X | | | |
432 +--------------------------+------+---------+------+------+
433 | os-nosdn-openo-ha | | X | | |
434 +--------------------------+------+---------+------+------+
435 | os-nosdn-kvm_ovs_dpdk | | | X | |
437 +--------------------------+------+---------+------+------+
438 | os-nosdn-kvm_ovs_dpdk-ha | | | X | |
439 +--------------------------+------+---------+------+------+
440 | os-nosdn-kvm_ovs_dpdk | | | X | |
442 +--------------------------+------+---------+------+------+
443 | os-nosdn-kvm_ovs_dpdk | | | X | |
444 | _bar-noha | | | | |
445 +--------------------------+------+---------+------+------+
446 | opnfv_os-ovn-nofeature- | X | | | |
447 | noha_daily | | | | |
448 +--------------------------+------+---------+------+------+
453 Test results are available in:
455 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
457 The reporting pages can be found at:
459 +---------------+-------------------------------------------------------------------------------------+
460 | apex | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html |
461 +---------------+-------------------------------------------------------------------------------------+
462 | compass | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html |
463 +---------------+-------------------------------------------------------------------------------------+
464 | fuel\@x86 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html |
465 +---------------+-------------------------------------------------------------------------------------+
466 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html |
467 +---------------+-------------------------------------------------------------------------------------+
468 | joid | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html |
469 +---------------+-------------------------------------------------------------------------------------+
480 +---------------------+--------------------------------------------+
481 | **JIRA REFERENCE** | **DESCRIPTION** |
483 +---------------------+--------------------------------------------+
484 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
486 +---------------------+--------------------------------------------+
487 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
489 +---------------------+--------------------------------------------+
492 Euphratess 5.0.0 known restrictions/issues
493 ------------------------------------------
494 +-----------+-----------+----------------------------------------------+
495 | Installer | Scenario | Issue |
496 +===========+===========+==============================================+
497 | any | \*-bgpvpn | Floating ips not supported. Some Test cases |
498 | | | related to floating ips are excluded. |
499 +-----------+-----------+----------------------------------------------+
500 | any | odl_l3-\* | Some test cases related to using floating IP |
501 | | | addresses fail because of a known ODL bug. |
503 +-----------+-----------+----------------------------------------------+
504 | compass | odl_l2-\* | In some test cases, VM instance will failed |
505 | | | raising network interfaces. |
507 +-----------+-----------+----------------------------------------------+
513 +---------------------+----------------------------------------------+
514 | **JIRA REFERENCE** | **DESCRIPTION** |
516 +---------------------+----------------------------------------------+
517 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
519 +---------------------+----------------------------------------------+
525 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
527 - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning
529 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
531 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
533 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
535 - Yardstick IRC chanel: #opnfv-yardstick