-=======
License
=======
-OPNFV Colorado release note for Yardstick Docs
+OPNFV Euphrates release note for Yardstick Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
-The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
-experimental framework are opensource software, licensed under the terms of the
-Apache License, Version 2.0.
+The *Yardstick framework*, the *Yardstick test cases* are open-source software,
+ licensed under the terms of the Apache License, Version 2.0.
-=========================================
-OPNFV Colorado Release Note for Yardstick
-=========================================
+OPNFV Euphrates Release Note for Yardstick
+==========================================
.. toctree::
:maxdepth: 2
Abstract
-========
+--------
This document describes the release note of Yardstick project.
Version History
-===============
-
-+----------------+--------------------+---------------------------------+
-| *Date* | *Version* | *Comment* |
-| | | |
-+----------------+--------------------+---------------------------------+
-| Dec 5th, 2016 | 3.0 | Yardstick for Colorado release |
-| | | |
-+----------------+--------------------+---------------------------------+
-| Oct 27th, 2016 | 2.0 | Yardstick for Colorado release |
-| | | |
-+----------------+--------------------+---------------------------------+
-| Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release |
-| | | |
-+----------------+--------------------+---------------------------------+
+---------------
++-------------------+-----------+---------------------------------+
+| *Date* | *Version* | *Comment* |
+| | | |
++-------------------+-----------+---------------------------------+
+| December 15, 2017 | 5.1.0 | Yardstick for Euphrates release |
+| | | |
++-------------------+-----------+---------------------------------+
+| October 20, 2017 | 5.0.0 | Yardstick for Euphrates release |
+| | | |
++-------------------+-----------+---------------------------------+
Important Notes
-===============
+---------------
The software delivered in the OPNFV Yardstick_ Project, comprising the
*Yardstick framework*, the *Yardstick test cases* and the experimental
The *Yardstick* framework is *installer*, *infrastructure* and *application*
independent.
+OPNFV Euphrates Release
+-----------------------
-OPNFV Colorado Release
-======================
-
-This Colorado release provides *Yardstick* as a framework for NFVI testing
+This Euphrates release provides *Yardstick* as a framework for NFVI testing
and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Documentation generated with Sphinx
* User Guide
- * Code Documentation
+ * Developer Guide
* Release notes (this document)
* Yardstick test cases yaml files
-* Yardstick pliug-in configration yaml files, plug-in install/remove scripts
+* Yardstick plug-in configuration yaml files, plug-in install/remove scripts
-For Colorado release, the *Yardstick framework* is used for the following
+For Euphrates release, the *Yardstick framework* is used for the following
testing:
* OPNFV platform testing - generic test cases to measure the categories:
* Storage
+* OPNFV platform network service benchmarking(NSB)
+
+ * NSB
+
* Test cases for the following OPNFV Projects:
+ * Container4NFV
+
* High Availability
* IPv6
* virtual Traffic Classifier
The *Yardstick framework* is developed in the OPNFV community, by the
-Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
-Project.
+Yardstick_ team.
.. note:: The test case description template used for the Yardstick test cases
is based on the document ETSI-ISG NFV-TST001_; the results report template
Release Data
-============
-
-+--------------------------------------+--------------------------------------+
-| **Project** | Yardstick |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/colorado.3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | colorado.3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Colorado |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | December 5th, 2016 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Colorado release 3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
+------------
+
++--------------------------------+-----------------------+
+| **Project** | Yardstick |
+| | |
++--------------------------------+-----------------------+
+| **Repo/tag** | yardstick/opnfv-5.1.0 |
+| | |
++--------------------------------+-----------------------+
+| **Yardstick Docker image tag** | opnfv-5.1.0 |
+| | |
++--------------------------------+-----------------------+
+| **Release designation** | Euphrates |
+| | |
++--------------------------------+-----------------------+
+| **Release date** | December 15, 2017 |
+| | |
++--------------------------------+-----------------------+
+| **Purpose of the delivery** | OPNFV Euphrates 5.1.0 |
+| | |
++--------------------------------+-----------------------+
Deliverables
-============
+------------
Documents
----------
+^^^^^^^^^
- - User Guide: http://artifacts.opnfv.org/yardstick/colorado/docs/userguide/index.html
+ - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html
- - Test Results: http://artifacts.opnfv.org/yardstick/colorado/docs/results/overview.html
+ - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html
Software Deliverables
----------------------
-
-**Yardstick framework source code <colorado.3.0>**
-
-+--------------------------------------+--------------------------------------+
-| **Project** | Yardstick |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/colorado.3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | colorado.3.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Colorado |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | December 5th, 2016 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Colorado release |
-| | |
-+--------------------------------------+--------------------------------------+
-
-
-**Contexts**
-
-+---------------------+-------------------------------------------------------+
-| **Context** | **Description** |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Heat* | Models orchestration using OpenStack Heat |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Node* | Models Baremetal, Controller, Compute |
-| | |
-+---------------------+-------------------------------------------------------+
-
-
-**Runners**
-
-+---------------------+-------------------------------------------------------+
-| **Runner** | **Description** |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Arithmetic* | Steps every run arithmetically according to specified |
-| | input value |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Duration* | Runs for a specified period of time |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Iteration* | Runs for a specified number of iterations |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Sequence* | Selects input value to a scenario from an input file |
-| | and runs all entries sequentially |
-| | |
-+---------------------+-------------------------------------------------------+
-
-
-**Scenarios**
-
-+---------------------+-------------------------------------------------------+
-| **Category** | **Delivered** |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Availability* | Attacker: |
-| | |
-| | * baremetal, process |
-| | |
-| | HA tools: |
-| | |
-| | * check host, openstack, process, service |
-| | * kill process |
-| | * start/stop service |
-| | |
-| | Monitor: |
-| | |
-| | * command, process |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Compute* | * cpuload |
-| | |
-| | * cyclictest |
-| | |
-| | * lmbench |
-| | |
-| | * lmbench_cache |
-| | |
-| | * perf |
-| | |
-| | * unixbench |
-| | |
-| | * ramspeed |
-| | |
-| | * cachestat |
-| | |
-| | * memeoryload |
-| | |
-| | * computecapacity |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Networking* | * iperf3 |
-| | |
-| | * netperf |
-| | |
-| | * netperf_node |
-| | |
-| | * ping |
-| | |
-| | * ping6 |
-| | |
-| | * pktgen |
-| | |
-| | * sfc |
-| | |
-| | * sfc with tacker |
-| | |
-| | * vtc instantion validation |
-| | |
-| | * vtc instantion validation with noisy neighbors |
-| | |
-| | * vtc throughput |
-| | |
-| | * vtc throughput in the presence of noisy neighbors |
-| | |
-| | * networkcapacity |
-| | |
-| | * netutilization |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Parser* | Tosca2Heat |
-| | |
-+---------------------+-------------------------------------------------------+
-| *Storage* | fio |
-| | |
-| | storagecapacity |
-| | |
-+---------------------+-------------------------------------------------------+
-| *StorPerf* | storperf |
-| | |
-+---------------------+-------------------------------------------------------+
-
-
-**API to Other Frameworks**
-
-+---------------------+-------------------------------------------------------+
-| **Framework** | **Description** |
-| | |
-+---------------------+-------------------------------------------------------+
-| *ApexLake* | Experimental framework that enables the user to |
-| | validate NFVI from the perspective of a VNF. |
-| | A virtual Traffic Classifier is utilized as VNF. |
-| | Enables experiments with SR-IOV on Compute Node. |
-| | |
-+---------------------+-------------------------------------------------------+
-
-
-**Test Results Output**
-
-+-----------------------------+-----------------------------------------------+
-| **Dispatcher** | **Description** |
-| | |
-+-----------------------------+-----------------------------------------------+
-| file | Log to a file. |
-| | |
-+-----------------------------+-----------------------------------------------+
-| http | Post data to html. |
-| | |
-+-----------------------------+-----------------------------------------------+
-| influxdb | Post data to influxDB. |
-| | |
-+-----------------------------+-----------------------------------------------+
-
-
-Delivered Test cases
---------------------
+^^^^^^^^^^^^^^^^^^^^^
+
+
+ - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.1.0)
+
+
+New Contexts
+############
+
++--------------+-------------------------------------------+
+| **Context** | **Description** |
+| | |
++--------------+-------------------------------------------+
+| *Heat* | Models orchestration using OpenStack Heat |
+| | |
++--------------+-------------------------------------------+
+| *Node* | Models Baremetal, Controller, Compute |
+| | |
++--------------+-------------------------------------------+
+| *Standalone* | Models VM running on Non-Managed NFVi |
+| | |
++--------------+-------------------------------------------+
+| *Kubernetes* | Models VM running on Non-Managed NFVi |
+| | |
++--------------+-------------------------------------------+
+
+
+New Runners
+###########
+
++--------------+-------------------------------------------------------+
+| **Runner** | **Description** |
+| | |
++--------------+-------------------------------------------------------+
+| *Arithmetic* | Steps every run arithmetically according to specified |
+| | input value |
+| | |
++--------------+-------------------------------------------------------+
+| *Duration* | Runs for a specified period of time |
+| | |
++--------------+-------------------------------------------------------+
+| *Iteration* | Runs for a specified number of iterations |
+| | |
++--------------+-------------------------------------------------------+
+| *Sequence* | Selects input value to a scenario from an input file |
+| | and runs all entries sequentially |
+| | |
++--------------+-------------------------------------------------------+
+
+
+New Scenarios
+#############
+
++----------------+-----------------------------------------------------+
+| **Category** | **Delivered** |
+| | |
++----------------+-----------------------------------------------------+
+| *Availability* | Attacker: |
+| | |
+| | * baremetal, process |
+| | |
+| | HA tools: |
+| | |
+| | * check host, openstack, process, service |
+| | * kill process |
+| | * start/stop service |
+| | |
+| | Monitor: |
+| | |
+| | * command, process |
+| | |
++----------------+-----------------------------------------------------+
+| *Compute* | * cpuload |
+| | |
+| | * cyclictest |
+| | |
+| | * lmbench |
+| | |
+| | * lmbench_cache |
+| | |
+| | * perf |
+| | |
+| | * unixbench |
+| | |
+| | * ramspeed |
+| | |
+| | * cachestat |
+| | |
+| | * memeoryload |
+| | |
+| | * computecapacity |
+| | |
+| | * SpecCPU2006 |
+| | |
++----------------+-----------------------------------------------------+
+| *Networking* | * iperf3 |
+| | |
+| | * netperf |
+| | |
+| | * netperf_node |
+| | |
+| | * ping |
+| | |
+| | * ping6 |
+| | |
+| | * pktgen |
+| | |
+| | * sfc |
+| | |
+| | * sfc with tacker |
+| | |
+| | * vtc instantion validation |
+| | |
+| | * vtc instantion validation with noisy neighbors |
+| | |
+| | * vtc throughput |
+| | |
+| | * vtc throughput in the presence of noisy neighbors |
+| | |
+| | * networkcapacity |
+| | |
+| | * netutilization |
+| | |
+| | * nstat |
+| | |
+| | * pktgenDPDK |
+| | |
++----------------+-----------------------------------------------------+
+| *Parser* | Tosca2Heat |
+| | |
++----------------+-----------------------------------------------------+
+| *Storage* | fio |
+| | |
+| | bonnie++ |
+| | |
+| | storagecapacity |
+| | |
++----------------+-----------------------------------------------------+
+| *StorPerf* | storperf |
+| | |
++----------------+-----------------------------------------------------+
+| *NSB* | vPE thoughput test case |
+| | |
++----------------+-----------------------------------------------------+
+
+
+
+New Test cases
+^^^^^^^^^^^^^^
* Generic NFVI test cases
- * OPNFV_YARDSTICK_TCOO1 - NW Performance
-
- * OPNFV_YARDSTICK_TCOO2 - NW Latency
-
- * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
-
- * OPNFV_YARDSTICK_TCOO5 - Storage Performance
-
- * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
-
- * OPNFV_YARDSTICK_TCOO9 - Packet Loss
-
- * OPNFV_YARDSTICK_TCO10 - Memory Latency
-
- * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
-
- * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
-
- * OPNFV_YARDSTICK_TCO14 - Processing Speed
-
- * OPNFV_YARDSTICK_TCO24 - CPU Load
-
- * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
-
- * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
- Test
-
- * OPNFV_YARDSTICK_TCO42 - Network Performance
-
- * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
-
- * OPNFV_YARDSTICK_TCO44 - Memory Utilization
-
- * OPNFV_YARDSTICK_TCO55 - Compute Capacity
-
- * OPNFV_YARDSTICK_TCO61 - Network Utilization
-
- * OPNFV_YARDSTICK_TCO63 - Storage Capacity
-
- * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
-
- * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
- Loss
-
- * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
-
- * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
- Loss
-
- * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
-
- * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
-
-* Test Cases for OPNFV HA Project:
-
- * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
-
- * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
-
- * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
- server
-
- * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
-
- * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
- api
-
- * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
- api
-
- * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
- proxy
+ * OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006
- * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
- Availability
+ * OPNFV_YARDSTICK_TCO79 - Bonnie++
- * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
- Availability
+* Kubernetes Test cases
- * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
- Availability
+ * OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER
- * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
- Availability
+ * OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM
- * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
-* Test Case for OPNFV IPv6 Project:
-
- * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
-
-* Test Case for OPNFV KVM Project:
-
- * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
-
-* Test Case for OPNFV Parser Project:
-
- * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
+Version Change
+--------------
-* Test Case for OPNFV StorPerf Project:
+Module Version Changes
+^^^^^^^^^^^^^^^^^^^^^^
- * OPNFV_YARDSTICK_TCO74 - Storperf
+This is the fifth tracked release of Yardstick. It is based on following
+upstream versions:
-* Test Cases for Virtual Traffic Classifier:
+- OpenStack Ocata
- * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
-Benchmarking Test
+- OpenDayLight Nitrogen
- * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
-Benchmarking in presence of noisy neighbors Test
+- ONOS Junco
- * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
- * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
-presence of noisy neighbors Test
+Document Version Changes
+^^^^^^^^^^^^^^^^^^^^^^^^
+This is the fifth tracked version of the Yardstick framework in OPNFV.
+It includes the following documentation updates:
-Version Change
-==============
+- Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
+ add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
+ add "Yardstick user interface" chapter; Update Yardstick installation chapter;
-Module Version Changes
-----------------------
+- Yardstick Developer Guide
-This is the second tracked release of Yardstick. It is based on following
-upstream versions:
+- Yardstick Release Notes for Yardstick: this document
-- ONOS Goldeneye
-- OpenStack Mitaka
+Feature additions
+^^^^^^^^^^^^^^^^^
-- OpenDaylight Beryllium
+- Yardstick RESTful API support
+- Network service benchmarking
-Document Version Changes
-------------------------
+- Stress testing with Bottlenecks team
-This is the second tracked version of the Yardstick framework in OPNFV.
-It includes the following documentation updates:
+- Yardstick framework improvement:
-- Yardstick User Guide: added yardstick plugin chapter; added Store Other
-Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter.
+ - yardstick report CLI
-- Yardstick Code Documentation: no changes
+ - Node context support OpenStack configuration via Ansible
-- Yardstick Release Notes for Yardstick: this document
+ - Https support
-- Test Results report for Colorado testing with Yardstick: updated listed of
-verified scenarios and limitations
+ - Kubernetes context type
+- Yardstick container local GUI
-Feature additions
------------------
- - Yardstick plugin
- - Yardstick reporting
- - StorPerf Integration
+- Python 3 support
Scenario Matrix
-===============
-
-For Colorado 3.0, Yardstick was tested on the following scenarios:
-
-+-------------------------+---------+---------+---------+---------+
-| Scenario | Apex | Compass | Fuel | Joid |
-+=========================+=========+=========+=========+=========+
-| os-nosdn-nofeature-noha | | | | X |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-nofeature-ha | X | | X | X |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-nofeature-ha | X | X | X | X |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-nofeature-noha| | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l3-nofeature-ha | X | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l3-nofeature-ha | | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-onos-sfc-ha | X | | X | X |
-+-------------------------+---------+---------+---------+---------+
-| os-onos-nofeature-ha | X | | X | X |
-+-------------------------+---------+---------+---------+---------+
-| os-onos-nofeature-noha | | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-sfc-ha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-sfc-noha | X | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-bgpvpn-ha | X | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-bgpvpn-noha | | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm-ha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-kvm-noha | | X | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-ovs-ha | | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-ovs-noha | X | | X | |
-+-------------------------+---------+---------+---------+---------+
-| os-ocl-nofeature-ha | | | | |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-lxd-ha | | | | X |
-+-------------------------+---------+---------+---------+---------+
-| os-nosdn-lxd-noha | | | | X |
-+-------------------------+---------+---------+---------+---------+
-| os-odl_l2-fdio-noha | X | | | |
-+-------------------------+---------+---------+---------+---------+
-
+---------------
+
+For Euphrates 5.0.0, Yardstick was tested on the following scenarios:
+
++--------------------------+------+---------+------+------+
+| Scenario | Apex | Compass | Fuel | Joid |
++==========================+======+=========+======+======+
+| os-nosdn-nofeature-noha | | | X | X |
++--------------------------+------+---------+------+------+
+| os-nosdn-nofeature-ha | X | X | X | X |
++--------------------------+------+---------+------+------+
+| os-odl_l2-nofeature-ha | | X | X | X |
++--------------------------+------+---------+------+------+
+| os-odl_l2-nofeature-noha | | | X | |
++--------------------------+------+---------+------+------+
+| os-odl_l3-nofeature-ha | X | X | X | |
++--------------------------+------+---------+------+------+
+| os-odl_l3-nofeature-noha | | | X | |
++--------------------------+------+---------+------+------+
+| os-onos-sfc-ha | | | | |
++--------------------------+------+---------+------+------+
+| os-onos-nofeature-ha | | X | | X |
++--------------------------+------+---------+------+------+
+| os-onos-nofeature-noha | | | | |
++--------------------------+------+---------+------+------+
+| os-odl_l2-sfc-ha | | | X | |
++--------------------------+------+---------+------+------+
+| os-odl_l2-sfc-noha | | | X | |
++--------------------------+------+---------+------+------+
+| os-odl_l2-bgpvpn-ha | X | | X | |
++--------------------------+------+---------+------+------+
+| os-odl_l2-bgpvpn-noha | | | X | |
++--------------------------+------+---------+------+------+
+| os-nosdn-kvm-ha | X | | X | |
++--------------------------+------+---------+------+------+
+| os-nosdn-kvm-noha | | | X | |
++--------------------------+------+---------+------+------+
+| os-nosdn-ovs-ha | | | X | |
++--------------------------+------+---------+------+------+
+| os-nosdn-ovs-noha | | | X | |
++--------------------------+------+---------+------+------+
+| os-ocl-nofeature-ha | | X | | |
++--------------------------+------+---------+------+------+
+| os-nosdn-lxd-ha | | | | X |
++--------------------------+------+---------+------+------+
+| os-nosdn-lxd-noha | | | | X |
++--------------------------+------+---------+------+------+
+| os-nosdn-fdio-ha | X | | | |
++--------------------------+------+---------+------+------+
+| os-odl_l2-fdio-noha | X | | | |
++--------------------------+------+---------+------+------+
+| os-odl-gluon-noha | X | | | |
++--------------------------+------+---------+------+------+
+| os-nosdn-openo-ha | | X | | |
++--------------------------+------+---------+------+------+
+| os-nosdn-kvm_ovs_dpdk | | | X | |
+| -noha | | | | |
++--------------------------+------+---------+------+------+
+| os-nosdn-kvm_ovs_dpdk-ha | | | X | |
++--------------------------+------+---------+------+------+
+| os-nosdn-kvm_ovs_dpdk | | | X | |
+| _bar-ha | | | | |
++--------------------------+------+---------+------+------+
+| os-nosdn-kvm_ovs_dpdk | | | X | |
+| _bar-noha | | | | |
++--------------------------+------+---------+------+------+
+| opnfv_os-ovn-nofeature- | X | | | |
+| noha_daily | | | | |
++--------------------------+------+---------+------+------+
Test results
-============
+------------
Test results are available in:
The reporting pages can be found at:
- * apex: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html
- * compass: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html
- * fuel: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-fuel.html
- * joid: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-joid.html
-
-You can get additional details through test logs on http://artifacts.opnfv.org/.
-As no search engine is available on the OPNFV artifact web site you must
-retrieve the pod identifier on which the tests have been executed (see
-field pod in any of the results) then click on the selected POD and look
-for the date of the test you are interested in.
-
++---------------+-------------------------------------------------------------------------------------+
+| apex | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html |
++---------------+-------------------------------------------------------------------------------------+
+| compass | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html |
++---------------+-------------------------------------------------------------------------------------+
+| fuel\@x86 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html |
++---------------+-------------------------------------------------------------------------------------+
+| fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html |
++---------------+-------------------------------------------------------------------------------------+
+| joid | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html |
++---------------+-------------------------------------------------------------------------------------+
Known Issues/Faults
-------------
- - Floating IP not supported in bgpvpn scenario
- - Floating IP not supported in apex-os-odl_l3-nofeature-ha scenario
-
-.. note:: The faults not related to *Yardstick* framework, addressing scenarios
- which were not fully verified, are listed in the OPNFV installer's release
- notes.
+^^^^^^^^^^^^^^^^^^^
Corrected Faults
-----------------
-
-Colorado.3.0:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **SLOGAN** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-239 | Define process for working with Yardstick |
-| | Grafana dashboard. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-373 | Add os-odl_l2-fdio-ha scenario support. |
-| | |
-+----------------------------+------------------------------------------------+
-
-
-Colorado.2.0:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **SLOGAN** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-325 | Provide raw format yardstick vm image for |
-| | nova-lxd scenario. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-358 | tc027 ipv6 test case to de-coupling to the |
-| | installers. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-359 | ipv6 testcase disable port-security on |
-| | vRouter. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-363 | ipv6 testcase to support fuel. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-367 | Add d3 graph presentation to yardstick |
-| | reporting. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-371 | Provide raw format yardstick vm image for |
-| | nova-lxd scenario. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-372 | cannot find yardstick-img-dpdk-modify and |
-| | yardstick-img-lxd-modify in environment |
-| | varibales. |
-| | |
-+----------------------------+------------------------------------------------+
-
-
-Colorado 3.0 known restrictions/issues
-==================================
+^^^^^^^^^^^^^^^^
+
+Euphrates 5.1.0:
+
++---------------------+-------------------------------------------------------------------------+
+| **JIRA REFERENCE** | **DESCRIPTION** |
+| | |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-841 | Fix various NSB license issues |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-73 | How To Work with Test Cases |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-500 | VNF testing documentation |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-826 | Allow overriding Heat IP addresses to match traffic generator profile |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-828 | Refactor doc/testing/user/userguide "Yardstick Installation" |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-830 | build_yardstick_image Ansible mount module doesn't work on Ubuntu 14.04 |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-833 | ansible_common transform password into lower case |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-847 | tc006, tc079, tc082 miss grafana dashboard in local deployment |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-849 | kill process do not accurately kill the process like "nova-api" |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-850 | tc023 miss description and tc050-58 wrong description |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-852 | tc078 cpu2006 fails in some situation |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-854 | yardstick docker lack of trex_client |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-867 | testcase tc078 have no data stored or dashboard to show results |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-871 | Remove img_modify_playbook assignation in build_yardstick_image.yml |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-829 | "nsb_setup.sh" doesn't parse the controller IP correctly |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-839 | NSB Prox BM test cases to be fixed for incorporating scale-up |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-840 | NSB Prox test documentation of vPE and LW-AFTR test cases |
++---------------------+-------------------------------------------------------------------------+
+| JIRA: YARDSTICK-848 | NSB "Prox" : Cleanup duplicated traffic profile |
++---------------------+-------------------------------------------------------------------------+
+
+
+
+
+Euphrates 5.0.0:
+
++---------------------+--------------------------------------------+
+| **JIRA REFERENCE** | **DESCRIPTION** |
+| | |
++---------------------+--------------------------------------------+
+| JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
+| | 'openstack -h' |
++---------------------+--------------------------------------------+
+| JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they |
+| | are dynamic |
++---------------------+--------------------------------------------+
+
+
+Euphratess 5.0.0 known restrictions/issues
+------------------------------------------
+-----------+-----------+----------------------------------------------+
-| Installer | Scenario | Issue |
+| Installer | Scenario | Issue |
+===========+===========+==============================================+
-| any | *-bgpvpn | Floating ips not supported. Some Test cases |
+| any | \*-bgpvpn | Floating ips not supported. Some Test cases |
| | | related to floating ips are excluded. |
+-----------+-----------+----------------------------------------------+
-| any | odl_l3-* | Some test cases related to using floating IP |
+| any | odl_l3-\* | Some test cases related to using floating IP |
| | | addresses fail because of a known ODL bug. |
-| | | https://jira.opnfv.org/browse/APEX-112 |
+| | | |
++-----------+-----------+----------------------------------------------+
+| compass | odl_l2-\* | In some test cases, VM instance will failed |
+| | | raising network interfaces. |
+| | | |
+-----------+-----------+----------------------------------------------+
-
-
-Open JIRA tickets
-=================
Useful links
-============
+------------
- wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
- - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
-
- - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
+ - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning
- Yardstick repo: https://git.opnfv.org/cgit/yardstick
- Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
- Yardstick IRC chanel: #opnfv-yardstick
-
-.. _`YARDSTICK-239` : https://jira.opnfv.org/browse/YARDSTICK-239
-
-.. _`YARDSTICK-325` : https://jira.opnfv.org/browse/YARDSTICK-325
-
-.. _`YARDSTICK-358` : https://jira.opnfv.org/browse/YARDSTICK-358
-
-.. _`YARDSTICK-359` : https://jira.opnfv.org/browse/YARDSTICK-359
-
-.. _`YARDSTICK-363` : https://jira.opnfv.org/browse/YARDSTICK-363
-
-.. _`YARDSTICK-367` : https://jira.opnfv.org/browse/YARDSTICK-367
-
-.. _`YARDSTICK-371` : https://jira.opnfv.org/browse/YARDSTICK-371
-
-.. _`YARDSTICK-372` : https://jira.opnfv.org/browse/YARDSTICK-372
-
-.. _`YARDSTICK-373` : https://jira.opnfv.org/browse/YARDSTICK-373