4 OPNFV Euphrates release note for Yardstick Docs
5 are licensed under a Creative Commons Attribution 4.0 International License.
6 You should have received a copy of the license along with this.
7 If not, see <http://creativecommons.org/licenses/by/4.0/>.
9 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
10 licensed under the terms of the Apache License, Version 2.0.
12 OPNFV Euphrates Release Note for Yardstick
13 ==========================================
18 .. _Yardstick: https://wiki.opnfv.org/yardstick
20 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
22 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
28 This document describes the release note of Yardstick project.
33 +-------------------+-----------+---------------------------------+
34 | *Date* | *Version* | *Comment* |
36 +-------------------+-----------+---------------------------------+
37 | December 15, 2017 | 5.1.0 | Yardstick for Euphrates release |
39 +-------------------+-----------+---------------------------------+
40 | October 20, 2017 | 5.0.0 | Yardstick for Euphrates release |
42 +-------------------+-----------+---------------------------------+
48 The software delivered in the OPNFV Yardstick_ Project, comprising the
49 *Yardstick framework*, the *Yardstick test cases* and the experimental
50 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
53 The *Yardstick* framework is *installer*, *infrastructure* and *application*
56 OPNFV Euphrates Release
57 -----------------------
59 This Euphrates release provides *Yardstick* as a framework for NFVI testing
60 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
62 * Documentation generated with Sphinx
68 * Release notes (this document)
72 * Automated Yardstick test suite (daily, weekly)
74 * Jenkins Jobs for OPNFV community labs
76 * Automated Yardstick test results visualization
78 * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
81 * Yardstick framework source code
83 * Yardstick test cases yaml files
85 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
87 For Euphrates release, the *Yardstick framework* is used for the following
90 * OPNFV platform testing - generic test cases to measure the categories:
98 * OPNFV platform network service benchmarking(NSB)
102 * Test cases for the following OPNFV Projects:
118 * virtual Traffic Classifier
120 The *Yardstick framework* is developed in the OPNFV community, by the
123 .. note:: The test case description template used for the Yardstick test cases
124 is based on the document ETSI-ISG NFV-TST001_; the results report template
125 used for the Yardstick results is based on the IEEE Std 829-2008.
131 +--------------------------------+-----------------------+
132 | **Project** | Yardstick |
134 +--------------------------------+-----------------------+
135 | **Repo/tag** | yardstick/opnfv-5.1.0 |
137 +--------------------------------+-----------------------+
138 | **Yardstick Docker image tag** | opnfv-5.1.0 |
140 +--------------------------------+-----------------------+
141 | **Release designation** | Euphrates |
143 +--------------------------------+-----------------------+
144 | **Release date** | December 15, 2017 |
146 +--------------------------------+-----------------------+
147 | **Purpose of the delivery** | OPNFV Euphrates 5.1.0 |
149 +--------------------------------+-----------------------+
158 - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html
160 - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html
163 Software Deliverables
164 ^^^^^^^^^^^^^^^^^^^^^
167 - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.1.0)
173 +--------------+-------------------------------------------+
174 | **Context** | **Description** |
176 +--------------+-------------------------------------------+
177 | *Heat* | Models orchestration using OpenStack Heat |
179 +--------------+-------------------------------------------+
180 | *Node* | Models Baremetal, Controller, Compute |
182 +--------------+-------------------------------------------+
183 | *Standalone* | Models VM running on Non-Managed NFVi |
185 +--------------+-------------------------------------------+
186 | *Kubernetes* | Models VM running on Non-Managed NFVi |
188 +--------------+-------------------------------------------+
194 +--------------+-------------------------------------------------------+
195 | **Runner** | **Description** |
197 +--------------+-------------------------------------------------------+
198 | *Arithmetic* | Steps every run arithmetically according to specified |
201 +--------------+-------------------------------------------------------+
202 | *Duration* | Runs for a specified period of time |
204 +--------------+-------------------------------------------------------+
205 | *Iteration* | Runs for a specified number of iterations |
207 +--------------+-------------------------------------------------------+
208 | *Sequence* | Selects input value to a scenario from an input file |
209 | | and runs all entries sequentially |
211 +--------------+-------------------------------------------------------+
217 +----------------+-----------------------------------------------------+
218 | **Category** | **Delivered** |
220 +----------------+-----------------------------------------------------+
221 | *Availability* | Attacker: |
223 | | * baremetal, process |
227 | | * check host, openstack, process, service |
229 | | * start/stop service |
233 | | * command, process |
235 +----------------+-----------------------------------------------------+
236 | *Compute* | * cpuload |
242 | | * lmbench_cache |
254 | | * computecapacity |
258 +----------------+-----------------------------------------------------+
259 | *Networking* | * iperf3 |
273 | | * sfc with tacker |
275 | | * vtc instantion validation |
277 | | * vtc instantion validation with noisy neighbors |
279 | | * vtc throughput |
281 | | * vtc throughput in the presence of noisy neighbors |
283 | | * networkcapacity |
285 | | * netutilization |
291 +----------------+-----------------------------------------------------+
292 | *Parser* | Tosca2Heat |
294 +----------------+-----------------------------------------------------+
299 | | storagecapacity |
301 +----------------+-----------------------------------------------------+
302 | *StorPerf* | storperf |
304 +----------------+-----------------------------------------------------+
305 | *NSB* | vPE thoughput test case |
307 +----------------+-----------------------------------------------------+
314 * Generic NFVI test cases
316 * OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006
318 * OPNFV_YARDSTICK_TCO79 - Bonnie++
320 * Kubernetes Test cases
322 * OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER
324 * OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM
330 Module Version Changes
331 ^^^^^^^^^^^^^^^^^^^^^^
333 This is the fifth tracked release of Yardstick. It is based on following
338 - OpenDayLight Nitrogen
343 Document Version Changes
344 ^^^^^^^^^^^^^^^^^^^^^^^^
346 This is the fifth tracked version of the Yardstick framework in OPNFV.
347 It includes the following documentation updates:
349 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
350 add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
351 add "Yardstick user interface" chapter; Update Yardstick installation chapter;
353 - Yardstick Developer Guide
355 - Yardstick Release Notes for Yardstick: this document
361 - Yardstick RESTful API support
363 - Network service benchmarking
365 - Stress testing with Bottlenecks team
367 - Yardstick framework improvement:
369 - yardstick report CLI
371 - Node context support OpenStack configuration via Ansible
375 - Kubernetes context type
377 - Yardstick container local GUI
385 For Euphrates 5.0.0, Yardstick was tested on the following scenarios:
387 +--------------------------+------+---------+------+------+
388 | Scenario | Apex | Compass | Fuel | Joid |
389 +==========================+======+=========+======+======+
390 | os-nosdn-nofeature-noha | | | X | X |
391 +--------------------------+------+---------+------+------+
392 | os-nosdn-nofeature-ha | X | X | X | X |
393 +--------------------------+------+---------+------+------+
394 | os-odl_l2-nofeature-ha | | X | X | X |
395 +--------------------------+------+---------+------+------+
396 | os-odl_l2-nofeature-noha | | | X | |
397 +--------------------------+------+---------+------+------+
398 | os-odl_l3-nofeature-ha | X | X | X | |
399 +--------------------------+------+---------+------+------+
400 | os-odl_l3-nofeature-noha | | | X | |
401 +--------------------------+------+---------+------+------+
402 | os-onos-sfc-ha | | | | |
403 +--------------------------+------+---------+------+------+
404 | os-onos-nofeature-ha | | X | | X |
405 +--------------------------+------+---------+------+------+
406 | os-onos-nofeature-noha | | | | |
407 +--------------------------+------+---------+------+------+
408 | os-odl_l2-sfc-ha | | | X | |
409 +--------------------------+------+---------+------+------+
410 | os-odl_l2-sfc-noha | | | X | |
411 +--------------------------+------+---------+------+------+
412 | os-odl_l2-bgpvpn-ha | X | | X | |
413 +--------------------------+------+---------+------+------+
414 | os-odl_l2-bgpvpn-noha | | | X | |
415 +--------------------------+------+---------+------+------+
416 | os-nosdn-kvm-ha | X | | X | |
417 +--------------------------+------+---------+------+------+
418 | os-nosdn-kvm-noha | | | X | |
419 +--------------------------+------+---------+------+------+
420 | os-nosdn-ovs-ha | | | X | |
421 +--------------------------+------+---------+------+------+
422 | os-nosdn-ovs-noha | | | X | |
423 +--------------------------+------+---------+------+------+
424 | os-ocl-nofeature-ha | | X | | |
425 +--------------------------+------+---------+------+------+
426 | os-nosdn-lxd-ha | | | | X |
427 +--------------------------+------+---------+------+------+
428 | os-nosdn-lxd-noha | | | | X |
429 +--------------------------+------+---------+------+------+
430 | os-nosdn-fdio-ha | X | | | |
431 +--------------------------+------+---------+------+------+
432 | os-odl_l2-fdio-noha | X | | | |
433 +--------------------------+------+---------+------+------+
434 | os-odl-gluon-noha | X | | | |
435 +--------------------------+------+---------+------+------+
436 | os-nosdn-openo-ha | | X | | |
437 +--------------------------+------+---------+------+------+
438 | os-nosdn-kvm_ovs_dpdk | | | X | |
440 +--------------------------+------+---------+------+------+
441 | os-nosdn-kvm_ovs_dpdk-ha | | | X | |
442 +--------------------------+------+---------+------+------+
443 | os-nosdn-kvm_ovs_dpdk | | | X | |
445 +--------------------------+------+---------+------+------+
446 | os-nosdn-kvm_ovs_dpdk | | | X | |
447 | _bar-noha | | | | |
448 +--------------------------+------+---------+------+------+
449 | opnfv_os-ovn-nofeature- | X | | | |
450 | noha_daily | | | | |
451 +--------------------------+------+---------+------+------+
456 Test results are available in:
458 - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
460 The reporting pages can be found at:
462 +---------------+-------------------------------------------------------------------------------------+
463 | apex | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html |
464 +---------------+-------------------------------------------------------------------------------------+
465 | compass | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html |
466 +---------------+-------------------------------------------------------------------------------------+
467 | fuel\@x86 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html |
468 +---------------+-------------------------------------------------------------------------------------+
469 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html |
470 +---------------+-------------------------------------------------------------------------------------+
471 | joid | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html |
472 +---------------+-------------------------------------------------------------------------------------+
483 +---------------------+-------------------------------------------------------------------------+
484 | **JIRA REFERENCE** | **DESCRIPTION** |
486 +---------------------+-------------------------------------------------------------------------+
487 | JIRA: YARDSTICK-841 | Fix various NSB license issues |
488 +---------------------+-------------------------------------------------------------------------+
489 | JIRA: YARDSTICK-73 | How To Work with Test Cases |
490 +---------------------+-------------------------------------------------------------------------+
491 | JIRA: YARDSTICK-500 | VNF testing documentation |
492 +---------------------+-------------------------------------------------------------------------+
493 | JIRA: YARDSTICK-826 | Allow overriding Heat IP addresses to match traffic generator profile |
494 +---------------------+-------------------------------------------------------------------------+
495 | JIRA: YARDSTICK-828 | Refactor doc/testing/user/userguide "Yardstick Installation" |
496 +---------------------+-------------------------------------------------------------------------+
497 | JIRA: YARDSTICK-830 | build_yardstick_image Ansible mount module doesn't work on Ubuntu 14.04 |
498 +---------------------+-------------------------------------------------------------------------+
499 | JIRA: YARDSTICK-833 | ansible_common transform password into lower case |
500 +---------------------+-------------------------------------------------------------------------+
501 | JIRA: YARDSTICK-847 | tc006, tc079, tc082 miss grafana dashboard in local deployment |
502 +---------------------+-------------------------------------------------------------------------+
503 | JIRA: YARDSTICK-849 | kill process do not accurately kill the process like "nova-api" |
504 +---------------------+-------------------------------------------------------------------------+
505 | JIRA: YARDSTICK-850 | tc023 miss description and tc050-58 wrong description |
506 +---------------------+-------------------------------------------------------------------------+
507 | JIRA: YARDSTICK-852 | tc078 cpu2006 fails in some situation |
508 +---------------------+-------------------------------------------------------------------------+
509 | JIRA: YARDSTICK-854 | yardstick docker lack of trex_client |
510 +---------------------+-------------------------------------------------------------------------+
511 | JIRA: YARDSTICK-867 | testcase tc078 have no data stored or dashboard to show results |
512 +---------------------+-------------------------------------------------------------------------+
513 | JIRA: YARDSTICK-871 | Remove img_modify_playbook assignation in build_yardstick_image.yml |
514 +---------------------+-------------------------------------------------------------------------+
515 | JIRA: YARDSTICK-829 | "nsb_setup.sh" doesn't parse the controller IP correctly |
516 +---------------------+-------------------------------------------------------------------------+
517 | JIRA: YARDSTICK-839 | NSB Prox BM test cases to be fixed for incorporating scale-up |
518 +---------------------+-------------------------------------------------------------------------+
519 | JIRA: YARDSTICK-840 | NSB Prox test documentation of vPE and LW-AFTR test cases |
520 +---------------------+-------------------------------------------------------------------------+
521 | JIRA: YARDSTICK-848 | NSB "Prox" : Cleanup duplicated traffic profile |
522 +---------------------+-------------------------------------------------------------------------+
529 +---------------------+--------------------------------------------+
530 | **JIRA REFERENCE** | **DESCRIPTION** |
532 +---------------------+--------------------------------------------+
533 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
535 +---------------------+--------------------------------------------+
536 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
538 +---------------------+--------------------------------------------+
541 Euphratess 5.0.0 known restrictions/issues
542 ------------------------------------------
543 +-----------+-----------+----------------------------------------------+
544 | Installer | Scenario | Issue |
545 +===========+===========+==============================================+
546 | any | \*-bgpvpn | Floating ips not supported. Some Test cases |
547 | | | related to floating ips are excluded. |
548 +-----------+-----------+----------------------------------------------+
549 | any | odl_l3-\* | Some test cases related to using floating IP |
550 | | | addresses fail because of a known ODL bug. |
552 +-----------+-----------+----------------------------------------------+
553 | compass | odl_l2-\* | In some test cases, VM instance will failed |
554 | | | raising network interfaces. |
556 +-----------+-----------+----------------------------------------------+
562 - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
564 - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning
566 - Yardstick repo: https://git.opnfv.org/cgit/yardstick
568 - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
570 - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
572 - Yardstick IRC chanel: #opnfv-yardstick