License
=======
-OPNFV Danube release note for Yardstick Docs
+OPNFV Fraser release note for Yardstick Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
-The *Yardstick framework*, the *Yardstick test cases* are opensource software,
+The *Yardstick framework*, the *Yardstick test cases* are open-source software,
licensed under the terms of the Apache License, Version 2.0.
-=========================================
-OPNFV Danube Release Note for Yardstick
-=========================================
+=======================================
+OPNFV Fraser Release Note for Yardstick
+=======================================
.. toctree::
:maxdepth: 2
Version History
===============
-
-+----------------+--------------------+---------------------------------+
-| *Date* | *Version* | *Comment* |
-| | | |
-+----------------+--------------------+---------------------------------+
-| | 3.1 | Yardstick for Danube release |
-| | | |
-| | | Note: The 3.1 tag is due to git |
-| | | tag issue during Danube 3.0 |
-| | | release |
-| | | |
-+----------------+--------------------+---------------------------------+
-| May 4th, 2017 | 2.0 | Yardstick for Danube release |
-| | | |
-+----------------+--------------------+---------------------------------+
-| Mar 31st, 2017 | 1.0 | Yardstick for Danube release |
-| | | |
-+----------------+--------------------+---------------------------------+
++-------------------+-----------+---------------------------------+
+| *Date* | *Version* | *Comment* |
+| | | |
++-------------------+-----------+---------------------------------+
+| April 27, 2018 | 6.0.0 | Yardstick for Fraser release |
+| | | |
++-------------------+-----------+---------------------------------+
Important Notes
===============
The software delivered in the OPNFV Yardstick_ Project, comprising the
-*Yardstick framework*, the *Yardstick test cases* and the experimental
-framework *Apex Lake* is a realization of the methodology in ETSI-ISG
-NFV-TST001_.
+*Yardstick framework*, and the *Yardstick test cases* is a realization of
+the methodology in ETSI-ISG NFV-TST001_.
The *Yardstick* framework is *installer*, *infrastructure* and *application*
independent.
+OPNFV Fraser Release
+====================
-OPNFV Danube Release
-======================
-
-This Danube release provides *Yardstick* as a framework for NFVI testing
+This Fraser release provides *Yardstick* as a framework for NFVI testing
and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Documentation generated with Sphinx
* User Guide
-
* Developer Guide
-
* Release notes (this document)
-
* Results
* Automated Yardstick test suite (daily, weekly)
* Yardstick test cases yaml files
-* Yardstick plug-in configration yaml files, plug-in install/remove scripts
+* Yardstick plug-in configuration yaml files, plug-in install/remove scripts
-For Danube release, the *Yardstick framework* is used for the following
+For Fraser release, the *Yardstick framework* is used for the following
testing:
* OPNFV platform testing - generic test cases to measure the categories:
* Compute
-
* Network
-
* Storage
-* OPNFV platform network service benchmarking(NSB)
+* OPNFV platform network service benchmarking (NSB)
* NSB
* Test cases for the following OPNFV Projects:
+ * Container4NFV
* High Availability
-
* IPv6
-
* KVM
-
* Parser
-
* StorPerf
-
* VSperf
- * virtual Traffic Classifier
-
The *Yardstick framework* is developed in the OPNFV community, by the
-Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
-Project.
+Yardstick_ team.
.. note:: The test case description template used for the Yardstick test cases
is based on the document ETSI-ISG NFV-TST001_; the results report template
Release Data
============
-+--------------------------------------+--------------------------------------+
-| **Project** | Yardstick |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/Danube.3.1 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | Danube.3.1 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Danube |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | July 14th, 2017 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Danube release 3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
++--------------------------------+-----------------------+
+| **Project** | Yardstick |
+| | |
++--------------------------------+-----------------------+
+| **Repo/tag** | yardstick/opnfv-6.0.0 |
+| | |
++--------------------------------+-----------------------+
+| **Yardstick Docker image tag** | opnfv-6.0.0 |
+| | |
++--------------------------------+-----------------------+
+| **Release designation** | Fraser |
+| | |
++--------------------------------+-----------------------+
+| **Release date** | April 27, 2018 |
+| | |
++--------------------------------+-----------------------+
+| **Purpose of the delivery** | OPNFV Fraser 6.0.0 |
+| | |
++--------------------------------+-----------------------+
Deliverables
Documents
---------
- - User Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/user/userguide/index.html
+ - User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/index.html
- - Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/yardstick/docs/testing/developer/devguide/index.html
+ - Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/developer/devguide/index.html
Software Deliverables
---------------------
-
- - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: danube.3.1)
-
-
-**Contexts**
-
-+---------------------+-------------------------------------------------------+
-| **Context** | **Description** |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Heat* | Models orchestration using OpenStack Heat |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Node* | Models Baremetal, Controller, Compute |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Standalone* | Models VM running on Non-Managed NFVi |
-| | |
-+---------------------+-------------------------------------------------------+
-
-
-**Runners**
-
-+---------------------+-------------------------------------------------------+
-| **Runner** | **Description** |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Arithmetic* | Steps every run arithmetically according to specified |
-| | input value |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Duration* | Runs for a specified period of time |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Iteration* | Runs for a specified number of iterations |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Sequence* | Selects input value to a scenario from an input file |
-| | and runs all entries sequentially |
-| | |
-+---------------------+-------------------------------------------------------+
-
-
-**Scenarios**
-
-+---------------------+-------------------------------------------------------+
-| **Category** | **Delivered** |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Availability* | Attacker: |
-| | |
-| | * baremetal, process |
-| | |
-| | HA tools: |
-| | |
-| | * check host, openstack, process, service |
-| | * kill process |
-| | * start/stop service |
-| | |
-| | Monitor: |
-| | |
-| | * command, process |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Compute* | * cpuload |
-| | |
-| | * cyclictest |
-| | |
-| | * lmbench |
-| | |
-| | * lmbench_cache |
-| | |
-| | * perf |
-| | |
-| | * unixbench |
-| | |
-| | * ramspeed |
-| | |
-| | * cachestat |
-| | |
-| | * memeoryload |
-| | |
-| | * computecapacity |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Networking* | * iperf3 |
-| | |
-| | * netperf |
-| | |
-| | * netperf_node |
-| | |
-| | * ping |
-| | |
-| | * ping6 |
-| | |
-| | * pktgen |
-| | |
-| | * sfc |
-| | |
-| | * sfc with tacker |
-| | |
-| | * vtc instantion validation |
-| | |
-| | * vtc instantion validation with noisy neighbors |
-| | |
-| | * vtc throughput |
-| | |
-| | * vtc throughput in the presence of noisy neighbors |
-| | |
-| | * networkcapacity |
-| | |
-| | * netutilization |
-| | |
-| | * nstat |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Parser* | Tosca2Heat |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Storage* | fio |
-| | |
-| | storagecapacity |
-| | |
-+---------------------+-------------------------------------------------------+
-| *StorPerf* | storperf |
-| | |
-+---------------------+-------------------------------------------------------+
-| *NSB* | vPE thoughput test case |
-| | |
-+---------------------+-------------------------------------------------------+
-
-
-**Test Results Output**
-
-+-----------------------------+-----------------------------------------------+
-| **Dispatcher** | **Description** |
-| | |
-+-----------------------------+-----------------------------------------------+
-| file | Log to a file. |
-| | |
-+-----------------------------+-----------------------------------------------+
-| http | Post data to html. |
-| | |
-+-----------------------------+-----------------------------------------------+
-| influxdb | Post data to influxDB. |
-| | |
-+-----------------------------+-----------------------------------------------+
-
-
-Delivered Test cases
---------------------
+ - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.0.0)
+
+List of Contexts
+^^^^^^^^^^^^^^^^
+
++--------------+-------------------------------------------+
+| **Context** | **Description** |
+| | |
++--------------+-------------------------------------------+
+| *Heat* | Models orchestration using OpenStack Heat |
+| | |
++--------------+-------------------------------------------+
+| *Node* | Models Baremetal, Controller, Compute |
+| | |
++--------------+-------------------------------------------+
+| *Standalone* | Models VM running on Non-Managed NFVi |
+| | |
++--------------+-------------------------------------------+
+| *Kubernetes* | Models VM running on Non-Managed NFVi |
+| | |
++--------------+-------------------------------------------+
+
+
+List of Runners
+^^^^^^^^^^^^^^^
+
+Note: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search".
+
++---------------+-------------------------------------------------------+
+| **Runner** | **Description** |
+| | |
++---------------+-------------------------------------------------------+
+| *Arithmetic* | Steps every run arithmetically according to specified |
+| | input value |
+| | |
++---------------+-------------------------------------------------------+
+| *Duration* | Runs for a specified period of time |
+| | |
++---------------+-------------------------------------------------------+
+| *Iteration* | Runs for a specified number of iterations |
+| | |
++---------------+-------------------------------------------------------+
+| *Sequence* | Selects input value to a scenario from an input file |
+| | and runs all entries sequentially |
+| | |
++---------------+-------------------------------------------------------+
+| **Dynamictp** | A runner that searches for the max throughput with |
+| | binary search |
+| | |
++---------------+-------------------------------------------------------+
+| **Search** | A runner that runs a specific time before it returns |
+| | |
++---------------+-------------------------------------------------------+
+
+
+List of Scenarios
+^^^^^^^^^^^^^^^^^
+
++----------------+-----------------------------------------------------+
+| **Category** | **Delivered** |
+| | |
++----------------+-----------------------------------------------------+
+| *Availability* | Attacker: |
+| | |
+| | * baremetal, process |
+| | |
+| | HA tools: |
+| | |
+| | * check host, openstack, process, service |
+| | * kill process |
+| | * start/stop service |
+| | |
+| | Monitor: |
+| | |
+| | * command, process |
+| | |
++----------------+-----------------------------------------------------+
+| *Compute* | * cpuload |
+| | * cyclictest |
+| | * lmbench |
+| | * lmbench_cache |
+| | * perf |
+| | * unixbench |
+| | * ramspeed |
+| | * cachestat |
+| | * memeoryload |
+| | * computecapacity |
+| | * SpecCPU2006 |
+| | |
++----------------+-----------------------------------------------------+
+| *Networking* | * iperf3 |
+| | * netperf |
+| | * netperf_node |
+| | * ping |
+| | * ping6 |
+| | * pktgen |
+| | * sfc |
+| | * sfc with tacker |
+| | * networkcapacity |
+| | * netutilization |
+| | * nstat |
+| | * pktgenDPDK |
+| | |
++----------------+-----------------------------------------------------+
+| *Parser* | Tosca2Heat |
+| | |
++----------------+-----------------------------------------------------+
+| *Storage* | * fio |
+| | * bonnie++ |
+| | * storagecapacity |
+| | |
++----------------+-----------------------------------------------------+
+| *StorPerf* | storperf |
+| | |
++----------------+-----------------------------------------------------+
+| *NSB* | vFW thoughput test case |
+| | |
++----------------+-----------------------------------------------------+
+
+
+New Test cases
+--------------
* Generic NFVI test cases
- * OPNFV_YARDSTICK_TCOO1 - NW Performance
-
- * OPNFV_YARDSTICK_TCOO2 - NW Latency
-
- * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
-
- * OPNFV_YARDSTICK_TCOO5 - Storage Performance
-
- * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
-
- * OPNFV_YARDSTICK_TCOO9 - Packet Loss
-
- * OPNFV_YARDSTICK_TCO10 - Memory Latency
-
- * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
-
- * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
-
- * OPNFV_YARDSTICK_TCO14 - Processing Speed
-
- * OPNFV_YARDSTICK_TCO24 - CPU Load
-
- * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
-
- * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
- Test
-
- * OPNFV_YARDSTICK_TCO42 - Network Performance
-
- * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
-
- * OPNFV_YARDSTICK_TCO44 - Memory Utilization
-
- * OPNFV_YARDSTICK_TCO55 - Compute Capacity
-
- * OPNFV_YARDSTICK_TCO61 - Network Utilization
-
- * OPNFV_YARDSTICK_TCO63 - Storage Capacity
-
- * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
-
- * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
- Loss
-
- * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
-
- * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
- Loss
-
- * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
-
- * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
-
- * OPNFV_YARDSTICK_TCO76 - Network frame error rate
-
-* Test Cases for OPNFV HA Project:
-
- * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
-
- * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
-
- * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
- server
-
- * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
-
- * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
- api
+ * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
- * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
- api
+* HA Test cases
- * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
- proxy
-
- * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
- Availability
-
- * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
- Availability
-
- * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
- Availability
-
- * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
- Availability
-
- * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
-
-* Test Case for OPNFV IPv6 Project:
-
- * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
-
-* Test Case for OPNFV KVM Project:
-
- * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
-
-* Test Case for OPNFV Parser Project:
-
- * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
-
-* Test Case for OPNFV StorPerf Project:
-
- * OPNFV_YARDSTICK_TCO74 - Storperf
-
-* Test Cases for Virtual Traffic Classifier:
-
- * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
-
-* Benchmarking Test:
-
- * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
-
-* Benchmarking in presence of noisy neighbors Test:
-
- * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
-
- * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
- presence of noisy neighbors Test
+ * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration
+ * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance
+ * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api
Version Change
Module Version Changes
----------------------
-This is the fourth tracked release of Yardstick. It is based on following
+This is the sixth tracked release of Yardstick. It is based on following
upstream versions:
-- ONOS Ibis
-
-- OpenStack Newton
-
-- OpenDaylight Boron
+- OpenStack Pike
+- OpenDayLight Oxygen
Document Version Changes
------------------------
-This is the fourth tracked version of the Yardstick framework in OPNFV.
+This is the sixth tracked version of the Yardstick framework in OPNFV.
It includes the following documentation updates:
- Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
add "Yardstick user interface" chapter; Update Yardstick installation chapter;
-
- Yardstick Developer Guide
-
- Yardstick Release Notes for Yardstick: this document
Feature additions
-----------------
-- Yardstick RESTful API support
-
-- Introduce Network service benchmarking
-
-- Introduce stress testing with Bottlenecks team
-
-- Yardstick framework improvement:
-
- - Parellel test cases execution support
-
- - yardstick report CLI
-
- - Node context support openstack configuration via Ansible
-
- - Https support
-
-- Python 3 support
+- Plugin-based test cases support Heat context
+- SR-IOV support for the Heat context
+- Support using existing network in Heat context
+- Support running test cases with existing VNFs/without destroying VNF in Heat context
+- Add vFW scale-up template
+- Improvements of unit tests and gating
+- GUI improvement about passing parameters
Scenario Matrix
===============
-For Danube 3.0, Yardstick was tested on the following scenarios:
-
-+-------------------------+---------+---------+---------+---------+
-| Scenario | Apex | Compass | Fuel | Joid |
-+=========================+=========+=========+=========+=========+
-| os-nosdn-nofeature-noha | | | X | X |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-nofeature-ha | X | X | X | X |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-nofeature-ha | | X | X | X |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-nofeature-noha| | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l3-nofeature-ha | X | X | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l3-nofeature-noha| | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-onos-sfc-ha | | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-onos-nofeature-ha | | X | | X |
-+-------------------------+---------+---------+---------+---------+
-| os-onos-nofeature-noha | | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-sfc-ha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-sfc-noha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-bgpvpn-ha | X | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-bgpvpn-noha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm-ha | X | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm-noha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-ovs-ha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-ovs-noha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-ocl-nofeature-ha | | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-lxd-ha | | | | X |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-lxd-noha | | | | X |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-fdio-ha | X | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-fdio-noha | X | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl-gluon-noha | X | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-openo-ha | | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm_ovs_dpdk | | | X | |
-| -noha | | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm_ovs_dpdk-ha| | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm_ovs_dpdk | | | X | |
-| _bar-ha | | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm_ovs_dpdk | | | X | |
-| _bar-noha | | | | |
-+-------------------------+---------+---------+---------+---------+
-| opnfv_os-ovn-nofeature- | X | | | |
-| noha_daily | | | | |
-+-------------------------+---------+---------+---------+---------+
+For Fraser 6.0.0, Yardstick was tested on the following scenarios:
+
++-------------------------+------+---------+----------+------+------+-------+
+| Scenario | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy |
++=========================+======+=========+==========+======+======+=======+
+| os-nosdn-nofeature-noha | X | X | | | X | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-nosdn-nofeature-ha | X | X | X | X | X | X |
++-------------------------+------+---------+----------+------+------+-------+
+| os-nosdn-bar-noha | X | X | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-nosdn-bar-ha | X | | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-odl-bgpvpn-ha | X | | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-nosdn-calipso-noha | X | | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-nosdn-kvm-ha | | X | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-odl_l3-nofeature-ha | | X | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-odl-sfc-ha | | X | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| os-odl-nofeature-ha | | | | X | | X |
++-------------------------+------+---------+----------+------+------+-------+
+| os-nosdn-ovs-ha | | | | X | | |
++-------------------------+------+---------+----------+------+------+-------+
+| k8-nosdn-nofeature-ha | | X | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+| k8-nosdn-stor4nfv-noha | | X | | | | |
++-------------------------+------+---------+----------+------+------+-------+
+
Test results
============
The reporting pages can be found at:
- * apex: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-apex.html
- * compass: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-compass.html
- * fuel: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-fuel.html
- * joid: http://testresults.opnfv.org/reporting/yardstick/release/danube/index-status-joid.html
-
++---------------+----------------------------------------------------------------------------------+
+| apex | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html |
++---------------+----------------------------------------------------------------------------------+
+| compass | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html |
++---------------+----------------------------------------------------------------------------------+
+| fuel\@x86 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html |
++---------------+----------------------------------------------------------------------------------+
+| fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html |
++---------------+----------------------------------------------------------------------------------+
+| joid | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html |
++---------------+----------------------------------------------------------------------------------+
Known Issues/Faults
-------------
-
- - Floating IP not supported in bgpvpn scenario
-
- - VM instance cannot get floating IP in compass-os-odl_l2-nofeature-ha scenario
-
-.. note:: The faults not related to *Yardstick* framework, addressing scenarios
- which were not fully verified, are listed in the OPNFV installer's release
- notes.
+-------------------
Corrected Faults
----------------
-Danube.3.1:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **DESCRIPTION** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-714 | Add yardstick env influxdb/grafana command for |
-| | CentOS |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-655 | Monitor command in tc019 may not show the |
-| | real nova-api service status |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-397 | HA testing framework improvement |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-660 | Improve monitor_process pass criteria |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-657 | HA monitor_multi bug, |
-| | KeyError: 'max_outage_time' |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-647 | TC025 fault_type value is wrong when using |
-| | baremetal pod scripts |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-659 | Terminate openstack service process using kill |
-| | command in HA test cases |
-+----------------------------+------------------------------------------------+
-| JIRA: ARMBAND-275 | Yardstick TC005 fails with |
-| | "Cannot map zero-fill pages" error |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-561 | Bugfix: AttributeError: 'dict' object has no |
-| | attribute 'split' if run sample/ping-hot.yaml |
-+----------------------------+------------------------------------------------+
-| JIRA: ARMBAND-268 | ERROR No JSON object could be decoded from |
-| | LMBENCH in TC010 |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-680 | storperf test case tc074 do not get results |
-| | |
-+----------------------------+------------------------------------------------+
-
-Danube.2.0:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **DESCRIPTION** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-608 | Set work directory in Yardstick container |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-616 | Bugfix: https support should adapt insecure |
-| | situation |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-620 | Yardstick virtualenv support |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-621 | Bugfix: fix query job status in TC074 |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-622 | Bugfix: take test case modification into |
-| | effect in load_images.sh |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-623 | change openrc file path to |
-| | /etc/yardstick/openstack.creds |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-624 | Add opnfv_os-ovn-nofeature-noha_daily test |
-| | suite |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-628 | Bugfix: Make tc019 and tc025 accept |
-| | --task-args options |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-629 | Bugfix: yardstick env prepare cmd do not |
-| | support other installer |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-632 | Bugfix: KeyError when using http dispatcher |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-633 | Bugfix: Environment Compatibility Issues in HA |
-| | Test Cases |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-634 | fix ha issue when run tc050~tc054 in ci |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-635 | Bugfix: Local Openstack Operation in HA test |
-| | frameworks |
-+----------------------------+------------------------------------------------+
-
-Danube.1.0:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **DESCRIPTION** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
-| | 'openstack -h' |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
-| | are dynamic |
-+----------------------------+------------------------------------------------+
-
-
-Danube 3.1 known restrictions/issues
-====================================
+Fraser 6.0.0:
+
++--------------------+--------------------------------------------------------------------------+
+| **JIRA REFERENCE** | **DESCRIPTION** |
++====================+==========================================================================+
+| YARDSTICK-831 | tc053 kill haproxy wrong |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-842 | load image fails when there's cirros image exist |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-857 | tc006 failed due to volume attached to different location "/dev/vdc" |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-874 | Specify supported architecture for Ubuntu backports repository |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-875 | Check if multiverse repository is available in Ubuntu |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-893 | Fix proxy env handling and ansible multinode support |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-899 | Variable local_iface_name is read before it is set |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-900 | Section in "upload_yardstick_image.yml" invalid |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-911 | Remove 'inconsistent-return-statements' from Pylint checks |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-989 | Yardstick real-time influxdb KPI reporting regressions |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-994 | NSB set-up build script for baremetal broken |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-996 | Error in address input format in "_ip_range_action_partial" |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1003 | Prox vnf descriptor cleanup for tg and vnf |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1006 | Ansible destroy script will fail if vm has already been undefined |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1012 | constants: fix pylint warnings for OSError |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1014 | Remove unused args in |
+| | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1016 | Allow vm to access outside world through default gateway |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1019 | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1021 | NSB: All Sample VNF test cases timeout after 1 hour of execution |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1036 | Prox: Addition of storage of extra counters for Grafana |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1038 | Missing file which is described in the operation_conf.yaml |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1047 | Error in string format in HeatTemplateError message |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1056 | yardstick report command print error when run test case |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1059 | Reduce the log level if TRex client is no connected |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1073 | Error when retrieving "options" section in "scenario" |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1080 | Running Test Case in Latest Yardstick Docker Image shows Error |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1082 | tc043,tc055, tc063, tc075, pass wrong node name in the ci scenario yaml |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1102 | Don't hide exception traceback from Task.start() |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1107 | bad exception traceback print due to atexit_handler |
++--------------------+--------------------------------------------------------------------------+
+| YARDSTICK-1120 | HA test case tc050 should start monitor before attack |
++--------------------+--------------------------------------------------------------------------+
+
+Fraser 6.0.0 known restrictions/issues
+======================================
+
+-----------+-----------+----------------------------------------------+
-| Installer | Scenario | Issue |
+| Installer | Scenario | Issue |
+===========+===========+==============================================+
-| any | *-bgpvpn | Floating ips not supported. Some Test cases |
-| | | related to floating ips are excluded. |
-+-----------+-----------+----------------------------------------------+
-| any | odl_l3-* | Some test cases related to using floating IP |
-| | | addresses fail because of a known ODL bug. |
| | | |
+-----------+-----------+----------------------------------------------+
-| compass | odl_l2-* | In some test cases, VM instance will failed |
-| | | raising network interfaces. |
-| | | |
-+-----------+-----------+----------------------------------------------+
-
-
-Open JIRA tickets
-=================
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **DESCRIPTION** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-626 | Fio and Lmbench don't work in Ubuntu-arm64 |
-| | image |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-603 | Timeout waiting for floating ip |
-| | (which actually pingable) |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
-| | |
-+----------------------------+------------------------------------------------+
-
Useful links
============
- wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
- - wiki Yardstick Danube release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Danube+Release+Planning
+ - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser
- Yardstick repo: https://git.opnfv.org/cgit/yardstick
Code Review / yardstick.git / blobdiff