-.. This work is licensed under a Creative Commons Attribution 4.0 International
-.. License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) OPNFV, Ericsson AB and others.
+=======
+License
+=======
+
+OPNFV Colorado release note for Yardstick Docs
+are licensed under a Creative Commons Attribution 4.0 International License.
+You should have received a copy of the license along with this.
+If not, see <http://creativecommons.org/licenses/by/4.0/>.
+The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
+experimental framework are opensource software, licensed under the terms of the
+Apache License, Version 2.0.
-============================================
-OPNFV Brahmaputra Release Note for Yardstick
-============================================
+=========================================
+OPNFV Colorado Release Note for Yardstick
+=========================================
.. toctree::
:maxdepth: 2
.. _Yardstick: https://wiki.opnfv.org/yardstick
-.. _Dashboard: http://130.211.154.108/grafana/dashboard/db/yardstick-main
+.. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
-.. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/
+.. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
Abstract
========
-This document compiles the release notes for the OPNFV Brahmaputra release
-for Yardstick framework as well as Yardstick_ Project deliverables.
-
-License
-=======
-
-The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
-experimental framework are opensource software, licensed under the terms of the
-Apache License, Version 2.0.
+This document describes the release note of Yardstick project.
Version History
===============
-+---------------+--------------------+---------------------------------+
-| *Date* | *Version* | *Comment* |
-| | | |
-+---------------+--------------------+---------------------------------+
-| Feb 25th,2016 | 1.0 | Brahmaputra release |
-| | | |
-+---------------+--------------------+---------------------------------+
++----------------+--------------------+---------------------------------+
+| *Date* | *Version* | *Comment* |
+| | | |
++----------------+--------------------+---------------------------------+
+| Dec 5th, 2016 | 3.0 | Yardstick for Colorado release |
+| | | |
++----------------+--------------------+---------------------------------+
+| Oct 27th, 2016 | 2.0 | Yardstick for Colorado release |
+| | | |
++----------------+--------------------+---------------------------------+
+| Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release |
+| | | |
++----------------+--------------------+---------------------------------+
Important Notes
independent.
-Summary
-=======
+OPNFV Colorado Release
+======================
-This Brahmaputra release provides *Yardstick* as a framework for NFVI testing
+This Colorado release provides *Yardstick* as a framework for NFVI testing
and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Documentation generated with Sphinx
* Automated Yardstick test results visualization
- * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB used as
+ * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
backend
* Yardstick framework source code
* Yardstick test cases yaml files
-For Brahmaputra release, the *Yardstick framework* is used for the following
+* Yardstick pliug-in configration yaml files, plug-in install/remove scripts
+
+For Colorado release, the *Yardstick framework* is used for the following
testing:
* OPNFV platform testing - generic test cases to measure the categories:
* Parser
+ * StorPerf
+
+ * VSperf
+
+ * virtual Traffic Classifier
+
The *Yardstick framework* is developed in the OPNFV community, by the
-Yardstick_ team.
+Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
+Project.
.. note:: The test case description template used for the Yardstick test cases
is based on the document ETSI-ISG NFV-TST001_; the results report template
| **Project** | Yardstick |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/brahmaputra.1.0 |
+| **Repo/tag** | yardstick/colorado.3.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | brahmaputra.1.0 |
+| **Yardstick Docker image tag** | colorado.3.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Brahmaputra |
+| **Release designation** | Colorado |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | Feb 25th, 2016 |
+| **Release date** | December 5th, 2016 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Brahmaputra release |
+| **Purpose of the delivery** | OPNFV Colorado release 3.0 |
| | |
+--------------------------------------+--------------------------------------+
-Version Change
---------------
-
-Module Version Changes
-~~~~~~~~~~~~~~~~~~~~~~
-
-This is the first tracked release of Yardstick. It is based on following
-upstream versions:
-
-- OpenStack Liberty
-
-- OpenDaylight Beryllium
-
-
-Document Version Changes
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-This is the first tracked version of the Yardstick framework in OPNFV.
-It includes the following documentation:
-
-- Yardstick User Guide
-
-- Yardstick Code Documentation
-
-- Yardstick Release Notes for Yardstick
-
-- Test Results report for Brahmaputra testing with Yardstick
-
-
-Reason for Version
-------------------
-
-Feature additions
-~~~~~~~~~~~~~~~~~
-
-This is the first tracked version of OPNFV Yardstick.
-
-
-Corrected Faults
-~~~~~~~~~~~~~~~~
-
-This is the first tracked version of OPNFV Yardstick.
-
-
-Known Faults
-~~~~~~~~~~~~
-
+Deliverables
+============
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **SLOGAN** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-175 | Running test suite, if a test cases running |
-| | failed, the test is stopped. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-176 | Fix plotter bug since Output format has been |
-| | changed. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-216 | ArgsAlreadyParsedError: arguments already |
-| | parsed: cannot register CLI option. |
-| | |
-+----------------------------+------------------------------------------------+
+Documents
+---------
-.. note:: The faults not related to *Yardstick* framework, addressing scenarios
- which were not fully verified, are listed in the OPNFV installer's release
- notes.
+ - User Guide: http://artifacts.opnfv.org/yardstick/colorado/docs/userguide/index.html
+ - Test Results: http://artifacts.opnfv.org/yardstick/colorado/docs/results/overview.html
-Deliverables
-------------
Software Deliverables
-~~~~~~~~~~~~~~~~~~~~~
+---------------------
-**Yardstick framework source code <brahmaputra.1.0>**
+**Yardstick framework source code <colorado.3.0>**
+--------------------------------------+--------------------------------------+
| **Project** | Yardstick |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/brahmaputra.1.0 |
+| **Repo/tag** | yardstick/colorado.3.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | brahmaputra.1.0 |
+| **Yardstick Docker image tag** | colorado.3.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Brahmaputra |
+| **Release designation** | Colorado |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | Feb 25th, 2016 |
+| **Release date** | December 5th, 2016 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Brahmaputra release |
+| **Purpose of the delivery** | OPNFV Colorado release |
| | |
+--------------------------------------+--------------------------------------+
+
**Contexts**
+---------------------+-------------------------------------------------------+
| | |
+---------------------+-------------------------------------------------------+
+
**Scenarios**
+---------------------+-------------------------------------------------------+
| | |
| | * lmbench |
| | |
+| | * lmbench_cache |
+| | |
| | * perf |
| | |
| | * unixbench |
| | |
+| | * ramspeed |
+| | |
+| | * cachestat |
+| | |
+| | * memeoryload |
+| | |
+| | * computecapacity |
+| | |
+---------------------+-------------------------------------------------------+
| *Networking* | * iperf3 |
| | |
| | * netperf |
| | |
+| | * netperf_node |
+| | |
| | * ping |
| | |
| | * ping6 |
| | |
| | * vtc throughput in the presence of noisy neighbors |
| | |
+| | * networkcapacity |
+| | |
+| | * netutilization |
+| | |
+---------------------+-------------------------------------------------------+
| *Parser* | Tosca2Heat |
| | |
+---------------------+-------------------------------------------------------+
| *Storage* | fio |
| | |
+| | storagecapacity |
+| | |
++---------------------+-------------------------------------------------------+
+| *StorPerf* | storperf |
+| | |
+---------------------+-------------------------------------------------------+
+
**API to Other Frameworks**
+---------------------+-------------------------------------------------------+
| | |
+---------------------+-------------------------------------------------------+
+
**Test Results Output**
+-----------------------------+-----------------------------------------------+
| http | Post data to html. |
| | |
+-----------------------------+-----------------------------------------------+
-| influxdb | Post data to influxdB. |
+| influxdb | Post data to influxDB. |
| | |
+-----------------------------+-----------------------------------------------+
Delivered Test cases
-~~~~~~~~~~~~~~~~~~~~
+--------------------
* Generic NFVI test cases
* OPNFV_YARDSTICK_TCOO2 - NW Latency
+ * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
+
* OPNFV_YARDSTICK_TCOO5 - Storage Performance
* OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
* OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
Test
+ * OPNFV_YARDSTICK_TCO42 - Network Performance
+
+ * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
+
+ * OPNFV_YARDSTICK_TCO44 - Memory Utilization
+
+ * OPNFV_YARDSTICK_TCO55 - Compute Capacity
+
+ * OPNFV_YARDSTICK_TCO61 - Network Utilization
+
+ * OPNFV_YARDSTICK_TCO63 - Storage Capacity
+
+ * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
+
+ * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
+ Loss
+
+ * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
+
+ * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
+ Loss
+
+ * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
+
+ * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
* Test Cases for OPNFV HA Project:
* OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
+ * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
+ server
+
+ * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
+
+ * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
+ api
+
+ * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
+ api
+
+ * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
+ proxy
+
+ * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
+ Availability
+
+ * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
+ Availability
+
+ * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
+ Availability
+
+ * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
+ Availability
+
+ * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
+
* Test Case for OPNFV IPv6 Project:
* OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
* Test Case for OPNFV Parser Project:
* OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
+
+* Test Case for OPNFV StorPerf Project:
+
+ * OPNFV_YARDSTICK_TCO74 - Storperf
+
+* Test Cases for Virtual Traffic Classifier:
+
+ * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
+Benchmarking Test
+
+ * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
+Benchmarking in presence of noisy neighbors Test
+
+ * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
+
+ * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
+presence of noisy neighbors Test
+
+
+Version Change
+==============
+
+Module Version Changes
+----------------------
+
+This is the second tracked release of Yardstick. It is based on following
+upstream versions:
+
+- ONOS Goldeneye
+
+- OpenStack Mitaka
+
+- OpenDaylight Beryllium
+
+
+Document Version Changes
+------------------------
+
+This is the second tracked version of the Yardstick framework in OPNFV.
+It includes the following documentation updates:
+
+- Yardstick User Guide: added yardstick plugin chapter; added Store Other
+Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter.
+
+- Yardstick Code Documentation: no changes
+
+- Yardstick Release Notes for Yardstick: this document
+
+- Test Results report for Colorado testing with Yardstick: updated listed of
+verified scenarios and limitations
+
+
+Feature additions
+-----------------
+ - Yardstick plugin
+ - Yardstick reporting
+ - StorPerf Integration
+
+
+Scenario Matrix
+===============
+
+For Colorado 3.0, Yardstick was tested on the following scenarios:
+
++-------------------------+---------+---------+---------+---------+
+| Scenario | Apex | Compass | Fuel | Joid |
++=========================+=========+=========+=========+=========+
+| os-nosdn-nofeature-noha | | | | X |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-nofeature-ha | X | | X | X |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l2-nofeature-ha | X | X | X | X |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l2-nofeature-noha| | X | | |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l3-nofeature-ha | X | | X | |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l3-nofeature-ha | | X | | |
++-------------------------+---------+---------+---------+---------+
+| os-onos-sfc-ha | X | | X | X |
++-------------------------+---------+---------+---------+---------+
+| os-onos-nofeature-ha | X | | X | X |
++-------------------------+---------+---------+---------+---------+
+| os-onos-nofeature-noha | | X | | |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l2-sfc-ha | | | X | |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l2-sfc-noha | X | X | | |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l2-bgpvpn-ha | X | | X | |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l2-bgpvpn-noha | | X | | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-kvm-ha | | | X | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-kvm-noha | | X | | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-ovs-ha | | | X | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-ovs-noha | X | | X | |
++-------------------------+---------+---------+---------+---------+
+| os-ocl-nofeature-ha | | | | |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-lxd-ha | | | | X |
++-------------------------+---------+---------+---------+---------+
+| os-nosdn-lxd-noha | | | | X |
++-------------------------+---------+---------+---------+---------+
+| os-odl_l2-fdio-noha | X | | | |
++-------------------------+---------+---------+---------+---------+
+
+
+Test results
+============
+
+Test results are available in:
+
+ - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
+
+The reporting pages can be found at:
+
+ * apex: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-apex.html
+ * compass: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-compass.html
+ * fuel: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-fuel.html
+ * joid: http://testresults.opnfv.org/reporting/yardstick/release/colorado/index-status-joid.html
+
+You can get additional details through test logs on http://artifacts.opnfv.org/.
+As no search engine is available on the OPNFV artifact web site you must
+retrieve the pod identifier on which the tests have been executed (see
+field pod in any of the results) then click on the selected POD and look
+for the date of the test you are interested in.
+
+
+Known Issues/Faults
+------------
+ - Floating IP not supported in bgpvpn scenario
+ - Floating IP not supported in apex-os-odl_l3-nofeature-ha scenario
+
+.. note:: The faults not related to *Yardstick* framework, addressing scenarios
+ which were not fully verified, are listed in the OPNFV installer's release
+ notes.
+
+
+Corrected Faults
+----------------
+
+Colorado.3.0:
+
++----------------------------+------------------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-239 | Define process for working with Yardstick |
+| | Grafana dashboard. |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-373 | Add os-odl_l2-fdio-ha scenario support. |
+| | |
++----------------------------+------------------------------------------------+
+
+
+Colorado.2.0:
+
++----------------------------+------------------------------------------------+
+| **JIRA REFERENCE** | **SLOGAN** |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-325 | Provide raw format yardstick vm image for |
+| | nova-lxd scenario. |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-358 | tc027 ipv6 test case to de-coupling to the |
+| | installers. |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-359 | ipv6 testcase disable port-security on |
+| | vRouter. |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-363 | ipv6 testcase to support fuel. |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-367 | Add d3 graph presentation to yardstick |
+| | reporting. |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-371 | Provide raw format yardstick vm image for |
+| | nova-lxd scenario. |
+| | |
++----------------------------+------------------------------------------------+
+| JIRA: YARDSTICK-372 | cannot find yardstick-img-dpdk-modify and |
+| | yardstick-img-lxd-modify in environment |
+| | varibales. |
+| | |
++----------------------------+------------------------------------------------+
+
+
+Colorado 3.0 known restrictions/issues
+==================================
++-----------+-----------+----------------------------------------------+
+| Installer | Scenario | Issue |
++===========+===========+==============================================+
+| any | *-bgpvpn | Floating ips not supported. Some Test cases |
+| | | related to floating ips are excluded. |
++-----------+-----------+----------------------------------------------+
+| any | odl_l3-* | Some test cases related to using floating IP |
+| | | addresses fail because of a known ODL bug. |
+| | | https://jira.opnfv.org/browse/APEX-112 |
++-----------+-----------+----------------------------------------------+
+
+
+Open JIRA tickets
+=================
+
+
+Useful links
+============
+
+ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
+
+ - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
+
+ - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
+
+ - Yardstick repo: https://git.opnfv.org/cgit/yardstick
+
+ - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
+
+ - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
+
+ - Yardstick IRC chanel: #opnfv-yardstick
+
+.. _`YARDSTICK-239` : https://jira.opnfv.org/browse/YARDSTICK-239
+
+.. _`YARDSTICK-325` : https://jira.opnfv.org/browse/YARDSTICK-325
+
+.. _`YARDSTICK-358` : https://jira.opnfv.org/browse/YARDSTICK-358
+
+.. _`YARDSTICK-359` : https://jira.opnfv.org/browse/YARDSTICK-359
+
+.. _`YARDSTICK-363` : https://jira.opnfv.org/browse/YARDSTICK-363
+
+.. _`YARDSTICK-367` : https://jira.opnfv.org/browse/YARDSTICK-367
+
+.. _`YARDSTICK-371` : https://jira.opnfv.org/browse/YARDSTICK-371
+
+.. _`YARDSTICK-372` : https://jira.opnfv.org/browse/YARDSTICK-372
+
+.. _`YARDSTICK-373` : https://jira.opnfv.org/browse/YARDSTICK-373
Code Review / yardstick.git / blobdiff