============================================
-OPNFV Brahmaputra Release Note for Yardstick
+OPNFV Colorado Release Note for Yardstick
============================================
.. toctree::
Abstract
========
-This document compiles the release notes for the OPNFV Brahmaputra release
-for Yardstick framework as well as Yardstick_ Project deliverables.
+This document describes the release note of Yardstick project.
+
License
=======
+OPNFV Colorado release note for Yardstick Docs
+are licensed under a Creative Commons Attribution 4.0 International License.
+You should have received a copy of the license along with this.
+If not, see <http://creativecommons.org/licenses/by/4.0/>.
+
The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
experimental framework are opensource software, licensed under the terms of the
Apache License, Version 2.0.
Version History
===============
-+---------------+--------------------+---------------------------------+
-| *Date* | *Version* | *Comment* |
-| | | |
-+---------------+--------------------+---------------------------------+
-| Apr 27th,2016 | 3.0 | Brahmaputra release |
-| | | |
-+---------------+--------------------+---------------------------------+
-| Mar 30th,2016 | 2.0 | Brahmaputra release |
-| | | |
-+---------------+--------------------+---------------------------------+
-| Feb 25th,2016 | 1.0 | Brahmaputra release |
-| | | |
-+---------------+--------------------+---------------------------------+
++----------------+--------------------+---------------------------------+
+| *Date* | *Version* | *Comment* |
+| | | |
++----------------+--------------------+---------------------------------+
+| Aug 22nd, 2016 | 1.0 | Yardstick for Colorado release |
+| | | |
++----------------+--------------------+---------------------------------+
Important Notes
Summary
=======
-This Brahmaputra release provides *Yardstick* as a framework for NFVI testing
+This Colorado release provides *Yardstick* as a framework for NFVI testing
and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
* Documentation generated with Sphinx
* Automated Yardstick test results visualization
- * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB used as
+ * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
backend
* Yardstick framework source code
* Yardstick test cases yaml files
-For Brahmaputra release, the *Yardstick framework* is used for the following
+* Yardstick pliug-in configration yaml files, plug-in install/remove scripts
+
+For Colorado release, the *Yardstick framework* is used for the following
testing:
* OPNFV platform testing - generic test cases to measure the categories:
* Parser
-* Test cases added in Brahmaputra2.0:
+ * StorPerf
+
+ * VSperf
* virtual Traffic Classifier
| **Project** | Yardstick |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/brahmaputra.3.0 |
+| **Repo/tag** | yardstick/colorado.1.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | brahmaputra.3.0 |
+| **Yardstick Docker image tag** | colorado.1.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Brahmaputra |
+| **Release designation** | Colorado base release |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | Apr 27th, 2016 |
+| **Release date** | September 22 2016 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Brahmaputra release |
+| **Purpose of the delivery** | Colorado base release |
| | |
+--------------------------------------+--------------------------------------+
-Version Change
---------------
-
-Module Version Changes
-~~~~~~~~~~~~~~~~~~~~~~
-
-This is the third tracked release of Yardstick. It is based on following
-upstream versions:
-
-- OpenStack Liberty
-
-- OpenDaylight Beryllium
-
-
-Document Version Changes
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-This is the third tracked version of the Yardstick framework in OPNFV.
-It includes the following documentation updates:
-
-- Yardstick User Guide: corrected faulty links
-
-- Yardstick Code Documentation: no changes
-
-- Yardstick Release Notes for Yardstick: this document
-
-- Test Results report for Brahmaputra testing with Yardstick: updated listed of
-verified scenarios and limitations
-
-Documentation updates on the second tracked version:
-
-- Yardstick User Guide: added software architecture chapter
-
-- Yardstick Code Documentation: no changes
-
-- Yardstick Release Notes for Yardstick: this document
-
-- Test Results report for Brahmaputra testing with Yardstick: added test cases
-and results for virtual Traffic Classifier
-
-
-Reason for Version
-------------------
-
-Feature additions
-~~~~~~~~~~~~~~~~~
-
-No new features.
-
-Brahmaputra.2.0:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **SLOGAN** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-227 | Heat HTTPS SSL support. |
-| | |
-+----------------------------+------------------------------------------------+
-
-
-Corrected Faults
-~~~~~~~~~~~~~~~~
-
-No corrected faults.
-
-Brahmaputra.2.0:
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **SLOGAN** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: - | Change copyrights for base scenario, runners, |
-| | dispatchers, cover. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: - | Update setup.py and dependencies |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: - | Add missing dependencies to docker file |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: - | Fix Heat template for noisy neighbors deploy |
-| | |
-+----------------------------+------------------------------------------------+
-
-Known Faults
-~~~~~~~~~~~~
-
-
-+----------------------------+------------------------------------------------+
-| **JIRA REFERENCE** | **SLOGAN** |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-175 | Running test suite, if a test cases running |
-| | failed, the test is stopped. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-176 | Fix plotter bug since Output format has been |
-| | changed. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-216 | ArgsAlreadyParsedError: arguments already |
-| | parsed: cannot register CLI option. |
-| | |
-+----------------------------+------------------------------------------------+
-| JIRA: YARDSTICK-231 | Installation instructions on Wiki not accurate |
-| | |
-+----------------------------+------------------------------------------------+
-
-.. note:: The faults not related to *Yardstick* framework, addressing scenarios
- which were not fully verified, are listed in the OPNFV installer's release
- notes.
-
-
Deliverables
-------------
+============
Software Deliverables
-~~~~~~~~~~~~~~~~~~~~~
+---------------------
-**Yardstick framework source code <brahmaputra.3.0>**
+**Yardstick framework source code <colorado.1.0>**
+--------------------------------------+--------------------------------------+
| **Project** | Yardstick |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | yardstick/brahmaputra.3.0 |
+| **Repo/tag** | yardstick/colorado.1.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Yardstick Docker image tag** | brahmaputra.3.0 |
+| **Yardstick Docker image tag** | colorado.1.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Brahmaputra |
+| **Release designation** | colorado |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | Apr 27th, 2016 |
+| **Release date** | September 22th, 2016 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | OPNFV Brahmaputra release |
+| **Purpose of the delivery** | OPNFV Colorado release |
| | |
+--------------------------------------+--------------------------------------+
+
**Contexts**
+---------------------+-------------------------------------------------------+
| | |
+---------------------+-------------------------------------------------------+
+
**Scenarios**
+---------------------+-------------------------------------------------------+
| | |
| | * lmbench |
| | |
+| | * lmbench_cache |
+| | |
| | * perf |
| | |
| | * unixbench |
| | |
+| | * ramspeed |
+| | |
+| | * cachestat |
+| | |
+| | * memeoryload |
+| | |
+| | * computecapacity |
+| | |
+---------------------+-------------------------------------------------------+
| *Networking* | * iperf3 |
| | |
| | * netperf |
| | |
+| | * netperf_node |
+| | |
| | * ping |
| | |
| | * ping6 |
| | |
| | * vtc throughput in the presence of noisy neighbors |
| | |
+| | * networkcapacity |
+| | |
+| | * netutilization |
+| | |
+---------------------+-------------------------------------------------------+
| *Parser* | Tosca2Heat |
| | |
+---------------------+-------------------------------------------------------+
| *Storage* | fio |
| | |
+| | storagecapacity |
+| | |
+---------------------+-------------------------------------------------------+
+| *StorPerf* | storperf |
+| | |
++---------------------+-------------------------------------------------------+
+
**API to Other Frameworks**
| | |
+---------------------+-------------------------------------------------------+
+
**Test Results Output**
+-----------------------------+-----------------------------------------------+
| http | Post data to html. |
| | |
+-----------------------------+-----------------------------------------------+
-| influxdb | Post data to influxdB. |
+| influxdb | Post data to influxDB. |
| | |
+-----------------------------+-----------------------------------------------+
Delivered Test cases
-~~~~~~~~~~~~~~~~~~~~
+--------------------
* Generic NFVI test cases
* OPNFV_YARDSTICK_TCOO2 - NW Latency
+ * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
+
* OPNFV_YARDSTICK_TCOO5 - Storage Performance
* OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
* OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
Test
+ * OPNFV_YARDSTICK_TCO42 - Network Performance
+
+ * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
+
+ * OPNFV_YARDSTICK_TCO44 - Memory Utilization
+
+ * OPNFV_YARDSTICK_TCO55 - Compute Capacity
+
+ * OPNFV_YARDSTICK_TCO61 - Network Utilization
+
+ * OPNFV_YARDSTICK_TCO63 - Storage Capacity
+
+ * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
+
+ * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
+ Loss
+
+ * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
+
+ * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
+ Loss
+
+ * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
+
+ * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
* Test Cases for OPNFV HA Project:
* OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
+ * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
+ server
+
+ * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
+
+ * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
+ api
+
+ * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
+ api
+
+ * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
+ proxy
+
+ * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
+ Availability
+
+ * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
+ Availability
+
+ * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
+ Availability
+
+ * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
+ Availability
+
+ * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
+
* Test Case for OPNFV IPv6 Project:
* OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
* OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
+* Test Case for OPNFV StorPerf Project:
+
+ * OPNFV_YARDSTICK_TCO74 - Storperf
+
* Test Cases for Virtual Traffic Classifier:
* OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
* OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
presence of noisy neighbors Test
+
+
+Version Change
+==============
+
+Module Version Changes
+----------------------
+
+This is the third tracked release of Yardstick. It is based on following
+upstream versions:
+
+- OpenStack Mitaka
+
+- OpenDaylight Beryllium
+
+
+Document Version Changes
+------------------------
+
+This is the third tracked version of the Yardstick framework in OPNFV.
+It includes the following documentation updates:
+
+- Yardstick User Guide: corrected faulty links
+
+- Yardstick Code Documentation: no changes
+
+- Yardstick Release Notes for Yardstick: this document
+
+- Test Results report for Colorado testing with Yardstick: updated listed of
+verified scenarios and limitations
+
+Documentation updates on the second tracked version:
+
+- Yardstick User Guide: added software architecture chapter
+
+- Yardstick Code Documentation: no changes
+
+- Yardstick Release Notes for Yardstick: this document
+
+- Test Results report for Brahmaputra testing with Yardstick: added test cases
+and results for virtual Traffic Classifier
+
+
+Reason for Version
+==================
+* TODO *
+
+Feature additions
+-----------------
+* TODO *
+
+Corrected Faults
+----------------
+* TODO *
+
+Known Issues/Faults
+------------
+* TODO *
+
+.. note:: The faults not related to *Yardstick* framework, addressing scenarios
+ which were not fully verified, are listed in the OPNFV installer's release
+ notes.
+
+
+Colorado known restrictions/issues
+==================================
++-----------+-----------+----------------------------------------------+
+| Installer | Scenario | Issue |
++===========+===========+==============================================+
+| any | *-bgpvpn | floating ips not supported. Some Test cases |
+| | | related to floating ips are excluded. |
++-----------+-----------+----------------------------------------------+
+* TODO *
+
+
+Test results
+============
+
+Test results are available in:
+
+ - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
+
+
+Open JIRA tickets
+=================
+
++------------------+-----------------------------------------------+
+| JIRA | Description |
++==================+===============================================+
++------------------+-----------------------------------------------+
++------------------+-----------------------------------------------+
++------------------+-----------------------------------------------+
++------------------+-----------------------------------------------+
++------------------+-----------------------------------------------+
+
+
+Useful links
+============
+
+ - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
+
+ - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
+
+ - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
+
+ - Yardstick repo: https://git.opnfv.org/cgit/yardstick
+
+ - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
+
+ - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
+
+ - Yardstick IRC chanel: #opnfv-yardstick
+
.. _install-framework:
+You can install Yardstick framework directly on Ubuntu 14.04 or in an Ubuntu
+14.04 Docker image.
+No matter which way you choose to install Yardstick framework, the following
+installation steps are identical.
+If you choose to use the Ubuntu 14.04 Docker image, You can pull the Ubuntu
+14.04 Docker image from Docker hub:
+
+::
+
+ docker pull ubuntu:14.04
+
Installing Yardstick framework
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Install dependencies:
+
::
sudo apt-get update && sudo apt-get install -y \
python-setuptools
Create a python virtual environment, source it and update setuptools:
+
::
virtualenv ~/yardstick_venv
easy_install -U setuptools
Download source code and install python dependencies:
+
::
git clone https://gerrit.opnfv.org/gerrit/yardstick
:alt: http://www.youtube.com/watch?v=4S4izNolmR0
:target: http://www.youtube.com/watch?v=4S4izNolmR0
-Installing extra tools
-^^^^^^^^^^^^^^^^^^^^^^
-yardstick-plot
-""""""""""""""
-Yardstick has an internal plotting tool ``yardstick-plot``, which can be installed
-using the following command:
-::
-
- sudo apt-get install -y g++ libfreetype6-dev libpng-dev pkg-config
- python setup.py develop easy_install yardstick[plot]
-
.. _guest-image:
Building a guest image
Also you may need install several additional packages to use this tool, by
follwing the commands below:
+
::
apt-get update && apt-get install -y \
This image can be built using the following command while in the directory where
Yardstick is installed (``~/yardstick`` if the framework is installed
by following the commands above):
+
::
sudo ./tools/yardstick-img-modify tools/ubuntu-server-cloudimg-modify.sh
via the OpenStack Dashboard.
Example command:
+
::
glance --os-image-api-version 1 image-create \
Installing Yardstick using Docker
---------------------------------
-Yardstick has two Docker images, first one (**Yardstick-framework**) serves as a
-replacement for installing the Yardstick framework in a virtual environment (for
-example as done in :ref:`install-framework`), while the other image is mostly for
-CI purposes (**Yardstick-CI**).
-
-Yardstick-framework image
-^^^^^^^^^^^^^^^^^^^^^^^^^
-Download the source code:
-
-::
-
- git clone https://gerrit.opnfv.org/gerrit/yardstick
-
-Build the Docker image and tag it as *yardstick-framework*:
-
-::
-
- cd yardstick
- docker build -t yardstick-framework .
-
-Run the Docker instance:
-
-::
-
- docker run --name yardstick_instance -i -t yardstick-framework
+Yardstick iteself has a Docker image, this Docker image (**Yardstick-stable**)
+serves as a replacement for installing the Yardstick framework in a virtual
+environment (for example as done in :ref:`install-framework`).
+It is recommended to use this Docker image to run Yardstick test.
-To build a guest image for Yardstick, see :ref:`guest-image`.
-
-Yardstick-CI image
-^^^^^^^^^^^^^^^^^^
-Pull the Yardstick-CI Docker image from Docker hub:
+Yardstick-stable image
+^^^^^^^^^^^^^^^^^^^^^^
+Pull the Yardstick-stable Docker image from Docker hub:
::
- docker pull opnfv/yardstick:$DOCKER_TAG
-
-Where ``$DOCKER_TAG`` is latest for master branch, as for the release branches,
-this coincides with its release name, such as brahmaputra.1.0.
+ docker pull opnfv/yardstick:stable
Run the Docker image:
and sshkey are required. See the link
https://git.opnfv.org/cgit/releng/tree/jjb/yardstick/yardstick-ci-jobs.yml.
-Note: exec_tests.sh is used for executing test suite here, furthermore, if someone wants to execute the
-test suite manually, it can be used as long as the parameters are configured correct. Another script
-called run_tests.sh is used for unittest in Jenkins verify job, in local manaul environment,
-it is recommended to run before test suite execuation.
+Note: exec_tests.sh is used for executing test suite here, furthermore, if someone
+wants to execute the test suite manually, it can be used as long as the parameters
+are configured correct. Another script called run_tests.sh is used for unittest in
+Jenkins verify job, in local manaul environment, it is recommended to run before
+test suite execuation.
-Basic steps performed by the **Yardstick-CI** container:
+Basic steps performed by the **Yardstick-stable** container:
1. clone yardstick and releng repos
2. setup OS credentials (releng scripts)
3. install yardstick and dependencies
4. build yardstick cloud image and upload it to glance
-5. upload cirros-0.3.3 cloud image to glance
+5. upload cirros-0.3.3 cloud image and ubuntu-14.04 cloud image to glance
6. run yardstick test scenarios
7. cleanup
+If someone only wants to execute a single test case, one can log into the yardstick-stable
+container first using command:
+
+::
+
+ docker run -it openfv/yardstick /bin/bash
+
+Then in the container run yardstick task command to execute single test case.
+Detailed steps about executing Yardstick test case can be found below.
+
OpenStack parameters and credentials
------------------------------------
Each testing tool supported by Yardstick has a sample configuration file.
These configuration files can be found in the **samples** directory.
-Example invocation of ``yardstick-plot`` tool:
-::
-
- yardstick-plot -i /tmp/yardstick.out -o /tmp/plots/
-
Default location for the output is ``/tmp/yardstick.out``.
-More info about the tool can be found by executing:
-::
-
- yardstick-plot -h
-
Deploy InfluxDB and Grafana locally
------------------------------------
.. pull docker images
Pull docker images
+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
::
log on using admin/admin and config database resource to be {YOUR_IP_HERE}:8086
.. image:: images/Grafana_config.png
+ :width: 800px
+ :alt: Grafana data source configration
Config yardstick conf
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
All in all, to create a test suite in yardstick, you just need to create a suite yaml file
and add test cases and constraint or task arguments if necessary.
+
Code Review / yardstick.git / commitdiff