+++ /dev/null
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) Huawei Technologies Co.,Ltd and others.
-
-****************************
-Project General Test Methods
-****************************
-
-For each *test suite*, you can setup *test story* by including several *test cases*
-only with one configuration parameter different, by comparing the test results,
-you can see the influence of the configuration parameter.
-
-Brahmaputra: rubbos is introduced, which is an end2end NFVI perforamnce tool.
-Virtual switch test framework(VSTF) is also introduced,
-which is an test framework used for vswitch performance test.
-
-Colorado: rubbos is refactored by using puppet, which makes it quite flexible
-to configure with different number of load generator(Client), worker(tomcat).
-vstf is refactored by extracting the test case's configuration information.
-
-***********************************
-Test suite & Test case Description
-***********************************
-+--------+-----------------------------+
-|Rubbos | rubbos_basic |
-| +-----------------------------+
-| | rubbos_TC1101 |
-| +-----------------------------+
-| | rubbos_TC1201 |
-| +-----------------------------+
-| | rubbos_TC1301 |
-| +-----------------------------+
-| | rubbos_TC1401 |
-| +-----------------------------+
-| | rubbos_heavy_TC1101 |
-+--------+-----------------------------+
-|vstf | vstf_Ti1 |
-| +-----------------------------+
-| | vstf_Ti2 |
-| +-----------------------------+
-| | vstf_Ti3 |
-| +-----------------------------+
-| | vstf_Tn1 |
-| +-----------------------------+
-| | vstf_Tn2 |
-| +-----------------------------+
-| | vstf_Tu1 |
-| +-----------------------------+
-| | vstf_Tu2 |
-| +-----------------------------+
-| | vstf_Tu3 |
-+--------+-----------------------------+
-
-***********************
-Integration Description
-***********************
-+-------------+----------------------+
-| Release | integrated installer |
-+-------------+----------------------+
-| Brahmaputra | Fuel |
-+-------------+----------------------+
-| Colorado | Compass |
-+-------------+----------------------+
+++ /dev/null
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-.. (c) Huawei Technologies Co.,Ltd and others.
-
-*************************************
-Bottlenecks - installation prodedure
-*************************************
-
-.. toctree::
- :maxdepth: 2
-
- ./rubbos_installation.rst
- ./vstf_installation.rst
- ./bottlenecks_test_result.rst
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-******************************
-Bottlenecks - platformoverview
-******************************
+*******************************
+Bottlenecks - Platform Overview
+*******************************
.. toctree::
:maxdepth: 2
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-**************************
+************************
Bottlenecks Introduction
-**************************
+************************
-**Welcome to bottlenecks' documentation !**
+**Welcome to Bottlenecks' documentation !**
.. _Bottlenecks: https://wiki.opnfv.org/bottlenecks
+.. _Proposal: https://wiki.opnfv.org/download/attachments/2926187/bottlenecks_project_for_test_meeting_20150820.pptx?version=1&modificationDate=1458849192000&api=v2
-Bottlenecks_ is an OPNFV Project.
+Bottlenecks_ is an OPNFV testing Project.
The project's goal is to find system bottlenecks by testing and verifying
OPNFV infrastructure in a staging environment before committing it to a production environment.
Instead of debugging a deployment in production environment,
an automatic method for executing benchmarks which plans to
validate the deployment during staging is adopted.
-This project will provide frameworks to find the bottlenecks of OPNFV infrastructure.
+This project forms a staging framework to find bottlenecks and to do analysis of the OPNFV infrastructure.
The *Bottlenecks* framework is deployed in several OPNFV community labs. It is
installer, infrastructure and application independent.
.. seealso:: Pharos_ for information on OPNFV community labs.
+Proposal_ for more project information.
+
Contact Bottlenecks
===================
-Feedback? `Contact us`_
+Feedback `Contact us`_
.. _Contact us: opnfv-users@lists.opnfv.org
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-*****************************
-Bottlenecks - configureguide
-*****************************
+***************************
+Bottlenecks - Release Notes
+***************************
.. toctree::
:maxdepth: 2
- ./README.rst
+ ./release_notes.rst
+ ./requirements_notes.rst
.. (c) Huawei Technologies Co.,Ltd and others.
-============================================
-OPNFV Colorado Release Notes for Bottlenecks
-============================================
+==================================================
+Bottlenecks Release Notes for OPNFV Danube Release
+==================================================
.. _Bottlenecks: https://wiki.opnfv.org/display/bottlenecks
License
=======
-OPNFV Colorado release notes for Bottlenecks Docs
+OPNFV release notes for Bottlenecks Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
| *Date* | *Version* | *Comment* |
| | | |
+----------------+--------------------+---------------------------------+
-| Sept 22nd, 2016| 1.0 | Bottlenecks for Colorado release|
+| Sept 22nd, 2016| 1.0 | Bottlenecks Colorado release 1.0|
+| | | |
++----------------+--------------------+---------------------------------+
+| Feb 17nd, 2016 | 1.1 | Bottlenecks Danube release 1.0 |
| | | |
+----------------+--------------------+---------------------------------+
Release Data
============
+Bramaputra Release Data
+-----------------------
+
++--------------------------------------+--------------------------------------+
+| **Project** | Bottlenecks |
+| | |
++--------------------------------------+--------------------------------------+
+| **Repo/tag** | Bottlenecks/brahmaputra.1.0 |
+| | |
++--------------------------------------+--------------------------------------+
+| **Bottlenecks Docker image tag** | brahmaputra.1.0 |
+| | |
++--------------------------------------+--------------------------------------+
+| **Release designation** | Brahmaputra base release |
+| | |
++--------------------------------------+--------------------------------------+
+| **Release date** | February 25 2016 |
+| | |
++--------------------------------------+--------------------------------------+
+| **Purpose of the delivery** | Brahmaputra base release |
+| | |
++--------------------------------------+--------------------------------------+
+
+Colorado Release Data
+-----------------------
+
+--------------------------------------+--------------------------------------+
| **Project** | Bottlenecks |
| | |
| | |
+--------------------------------------+--------------------------------------+
+Colorado Release Data
+-----------------------
+* TODO
-Deliverables
-============
+Danube Deliverables
+===================
+
+Software Deliverables
+---------------------
+
+**Bottlenecks source code <danube.1.0>**
+
+* https://gerrit.opnfv.org/gerrit/gitweb?p=bottlenecks.git;a=summary
+
+
+Documentatiion Deliverables
+---------------------------
+
+**Bottlenecks documentation <danube.1.0>**
+
+* Platformation Overview: http://artifacts.opnfv.org/bottlenecks/danube/1.0/docs/platformoverview/index.html
+* Configuration Guide: http://artifacts.opnfv.org/bottlenecks/danube/1.0/configurationguide/index.html
+* Installation Guide: http://artifacts.opnfv.org/bottlenecks/danube/docs/installationprocedure/index.html
+* http://artifacts.opnfv.org/bottlenecks/danube/1.0/releasenotes/index.html
+* User Guide: http://artifacts.opnfv.org/bottlenecks/danube/1.0/docs/userguide/index.html
+
+Colorado Deliverables
+=====================
Software Deliverables
---------------------
**Bottlenecks source code <colorado.1.0>**
-* TODO
+* https://gerrit.opnfv.org/gerrit/gitweb?p=bottlenecks.git;a=summary
+
Documentatiion Deliverables
---------------------------
**Bottlenecks documentation <colorado.1.0>**
-* TODO
+* Platformation Overview: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/platformoverview/index.html
+* Configuration Guide: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/configurationguide/index.html
+* Installation Guide: http://artifacts.opnfv.org/bottlenecks/colorado/docs/installationprocedure/index.html
+* http://artifacts.opnfv.org/bottlenecks/colorado/1.0/releasenotes/index.html
+* User Guide: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/userguide/index.html
Reason for Version
==================
-* TODO
+* In Danube, POSCA testsuite is introduced and two stress tests are implemented.
+* In Colorado, rubbos is refactored by using puppet, which makes it quite flexible to configure with different number of load generator(Client), worker(tomcat).
+* In Colorado, vstf is refactored by extracting the test case's configuration information.
-Colorado known restrictions/issues
-==================================
+Known restrictions/issues
+=========================
* TODO
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Huawei Technologies Co.,Ltd and others.
+
+
+*******************************************************
+Bottlenecks Requirements Notes for OPNFV Danube Release
+*******************************************************
+
+======================
+Installer Requirements
+======================
+
+* TODO
+
+=====================
+Hardware Requirements
+=====================
+
+* TODO
+
+=====================
+Software Requirements
+=====================
+
+Operation System
+----------------
+
+* TODO
+
+Depending Package
+-----------------
+
+- appdirs==1.4.0
+- Babel==2.2.0
+- backport-ipaddress==0.1
+- cliff==2.0.0
+- cmd2==0.6.8
+- coverage==4.1b2
+- debtcollector==1.3.0
+- ecdsa==0.13
+- extras==0.0.3
+- fixtures==1.4.0
+- flake8==2.5.4
+- funcsigs==0.4
+- functools32==3.2.3.post2
+- futures==3.0.5
+- iso8601==0.1.11
+- Jinja2==2.8
+- jsonpatch==1.13
+- jsonpointer==1.10
+- jsonschema==2.5.1
+- keystoneauth1==2.3.0
+- linecache2==1.0.0
+- lxml==3.5.0
+- MarkupSafe==0.23
+- mccabe==0.4.0
+- mock==1.3.0
+- monotonic==1.0
+- msgpack-python==0.4.7
+- netaddr==0.7.18
+- netifaces==0.10.4
+- nose==1.3.7
+- openstacksdk==0.8.1
+- os-client-config==1.16.0
+- oslo.config==3.9.0
+- oslo.i18n==3.4.0
+- oslo.serialization==2.4.0
+- oslo.utils==3.7.0
+- paramiko==1.16.0
+- pbr==1.8.1
+- pep8==1.7.0
+- positional==1.0.1
+- prettytable==0.7.2
+- pycrypto==2.6.1
+- pyflakes==1.0.0
+- pyparsing==2.1.0
+- pyrsistent==0.11.12
+- python-cinderclient==1.6.0
+- python-glanceclient==2.0.0
+- python-heatclient==1.0.0
+- python-keystoneclient==2.3.1
+- python-mimeparse==1.5.1
+- python-neutronclient==4.1.1
+- python-novaclient==3.3.0
+- python-openstackclient==2.2.0
+- python-subunit==1.2.0
+- python-swiftclient==3.0.0
+- pytz==2015.7
+- PyYAML==3.11
+- requests==2.9.1
+- requestsexceptions==1.1.3
+- scp==0.10.2
+- simplejson==3.8.2
+- six==1.10.0
+- stevedore==1.12.0
+- testrepository==0.0.20
+- testtools==2.0.0
+- traceback2==1.4.0
+- unicodecsv==0.14.1
+- unittest2==1.1.0
+- warlock==1.2.0
+- wrapt==1.10.6
+- pyroute2==0.4.10
+- elasticsearch==5.0.1
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-*************************
-Bottlenecks Test Report
-*************************
-
+**************
+Dashbard guide
+**************
Scope
=====
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-=========================
-Bottlenecks Release Notes
-=========================
+***************************
+Bottlenecks - Testing Guide
+***************************
.. toctree::
:maxdepth: 2
- release-notes
-
+ ./overview.rst
+ ./posca_guide.rst
+ ./rubbos_guide.rst
+ ./vstf_guide.rst
+ ./dashboard_guide.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Huawei Technologies Co.,Ltd and others.
+
+*********************
+Project Testing Guide
+*********************
+
+For each *test suite*, you can either setup *test story* or *test case* to run
+certain test. *test story* could include several *test cases* as a set in one
+configuration file. You could then call the *test story* or *test case* by using
+Bottlencks CLI or Python build process.
+Details will be shown in the following section.
+
+Brief Introdcution of the Test suites in Project Releases
+=============================================================
+
+Brahmaputra: rubbos is introduced, which is an end2end NFVI perforamnce tool.
+Virtual switch test framework(VSTF) is also introduced,
+which is an test framework used for vswitch performance test.
+
+Colorado: rubbos is refactored by using puppet, which makes it quite flexible
+to configure with different number of load generator(Client), worker(tomcat).
+vstf is refactored by extracting the test case's configuration information.
+
+Danube: posca testsuite is introduced to implementing stress (factor), scenario and
+tuning test in parametric manner. Two testcases are developed and integrated into
+community CI pipeline.
+
+Integration Description
+=======================
++-------------+----------------------+----------------------+
+| Release | integrated installer | Supported Testsuite |
++-------------+----------------------+----------------------+
+| Brahmaputra | Fuel | Rubbos, VSTF |
++-------------+----------------------+----------------------+
+| Colorado | Compass | Rubbos, VSTF |
++-------------+----------------------+----------------------+
+| Danube | Compass | POSCA |
++-------------+----------------------+----------------------+
+
+Test suite & Test case Description
+==================================
++--------+-------------------------------+
+|Rubbos | rubbos_basic |
+| +-------------------------------+
+| | rubbos_TC1101 |
+| +-------------------------------+
+| | rubbos_TC1201 |
+| +-------------------------------+
+| | rubbos_TC1301 |
+| +-------------------------------+
+| | rubbos_TC1401 |
+| +-------------------------------+
+| | rubbos_heavy_TC1101 |
++--------+-------------------------------+
+|vstf | vstf_Ti1 |
+| +-------------------------------+
+| | vstf_Ti2 |
+| +-------------------------------+
+| | vstf_Ti3 |
+| +-------------------------------+
+| | vstf_Tn1 |
+| +-------------------------------+
+| | vstf_Tn2 |
+| +-------------------------------+
+| | vstf_Tu1 |
+| +-------------------------------+
+| | vstf_Tu2 |
+| +-------------------------------+
+| | vstf_Tu3 |
++--------+-------------------------------+
+|posca | posca_stress_ping |
+| +-------------------------------+
+| | posca_stress_traffic |
+| | |
+| | (posca_factor_sys_bandwidth) |
++--------+-------------------------------+
+
+Installation of Testsuites
+==========================
+
+* TODO
+
+Setting Up Configrations
+========================
+
+* TODO
+
+Run Tests Brief
+===============
+
+* TODO
+* More will be shown in the platform overview
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) Huawei Technologies Co.,Ltd and others.
+
+*********************
+POSCA Testsuite Guide
+*********************
+
+
+POSCA Introduction
+====================
+The POSCA (Parametric Bottlenecks Testing Catalogue) testsuite
+classifies the bottlenecks test cases and results into 5 categories.
+Then the results will be analyzed and bottlenecks will be searched
+among these categories.
+
+The POSCA testsuite aims to locate the bottlenecks in parmetric
+manner and to decouple the bottlenecks regarding the deployment
+requirements.
+The POSCA testsuite provides an user friendly way to profile and
+understand the E2E system behavior and deployment requirements.
+
+Goals of the POSCA testsuite:
+ a) Automatically locate the bottlenecks in a iterative manner.
+ b) Automatically generate the testing report for bottlenecks in
+different categories.
+ c) Implementing Automated Staging.
+
+Scopes of the POSCA testsuite:
+ a) Modeling, Testing and Test Result analysis.
+ b) Parameters choosing and Algorithms.
+
+Test stories of POSCA testsuite:
+ a) Factor test (Stress test): base test cases that Feature test and Optimization will be
+dependant on.
+ b) Feature test: test cases for features/scenarios.
+ c) Optimization test: test to tune the system parameter.
+
+Detailed workflow is illutrated below.
+* TODO Add image here
+
+Preinstall Packages
+====================
+* TODO Description of dependent packages
+
+Run POSCA Locally
+=================
+* TO Description of POSCA testing steps
+
+Run POSCA through Community CI
+==============================
+* TODO Description of POSCA integrated into CI system
+
+Test Result Description
+=======================
+* TODO hwo to access the test result
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-**************************************
-Bottlenecks Rubbos Installation Guide
-**************************************
+**********************
+Rubbos Testsuite Guide
+**********************
Rubbos Introduction
.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-************************************
-Bottlenecks VSTF Installation Guide
-************************************
+********************
+VSTF Testsuite Guide
+********************
VSTF Introduction
====================
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+**************************
+Bottlenecks - User Guide
+**************************
+
+.. This index.rst file is used to structure and organise the content in these directories.
+.. The documentation toolchain will create web and pdf versions of the content of this file.
+
+.. toctree::
+ :maxdepth: 2
+
+ ./rubbos/rubbos_basic.rst
+ ./rubbos/rubbos_tc1101.rst
+ ./rubbos/rubbos_tc1201.rst
+ ./rubbos/rubbos_tc1301.rst
+ ./rubbos/rubbos_tc1401.rst
+ ./rubbos/rubbos_heavy_tc1101.rst
+ ./vstf/vstf_Ti1.rst
+ ./vstf/vstf_Ti2.rst
+ ./vstf/vstf_Ti3.rst
+ ./vstf/vstf_Tn1.rst
+ ./vstf/vstf_Tn2.rst
+ ./vstf/vstf_Tu1.rst
+ ./vstf/vstf_Tu2.rst
+ ./vstf/vstf_Tu3.rst
+ ./posca/posca_stress_ping.rst
+ ./posca/posca_stress_traffic.rst
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Tech and others.
+
+***************************************************
+POSCA Stress (Factor) Test of Perfomance Life-Cycle
+***************************************************
+
+Test Case
+========
+
++-----------------------------------------------------------------------------+
+|Bottlenecks POSCA Stress Test Ping |
+| |
++--------------+--------------------------------------------------------------+
+|test case name| posca_stress_ping |
+| | |
++--------------+--------------------------------------------------------------+
+|description | Stress test regarding perfomance life-cycle while using ping |
+| | to validate the VM pairs constructions |
++--------------+--------------------------------------------------------------+
+|configuration | config file: |
+| | /testsuite/posca/testcase_cfg/posca_stress_ping |
+| | |
+| | stack number: 5, 10, 20, 50 ... |
+| | |
++--------------+--------------------------------------------------------------+
+|test result | PKT loss rate, success rate, test time, latency |
+| | |
++--------------+--------------------------------------------------------------+
+
+Configuration
+============
+load_manager:
+ scenarios:
+ tool: ping
+ test_times: 100
+ package_size:
+ num_stack: 2, 5, 10
+ package_loss: 10%
+
+ contexts:
+ stack_create: yardstick
+ flavor:
+ yardstick_test_ip:
+ yardstick_test_dir: "samples"
+ yardstick_testcase: "ping_bottlenecks"
+
+dashboard:
+ dashboard: "y"
+ dashboard_ip:
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International
+.. License.
+.. http://creativecommons.org/licenses/by/4.0
+.. (c) OPNFV, Huawei Tech and others.
+
+**********************************************
+POSCA Stress (Factor) Test of System bandwidth
+**********************************************
+
+
+Test Case
+========
++-----------------------------------------------------------------------------+
+|Bottlenecks POSCA Stress Test Traffic |
+| |
++--------------+--------------------------------------------------------------+
+|test case name| posca_stress_ping |
+| | |
++--------------+--------------------------------------------------------------+
+|description | Stress test regarding baseline of the system for a single |
+| | user, i.e., a VM pair while increasing the package size |
++--------------+--------------------------------------------------------------+
+|configuration | config file: |
+| | /testsuite/posca/testcase_cfg/posca_factor_system_bandwith |
+| | |
+| | stack number: 1 |
+| | |
++--------------+--------------------------------------------------------------+
+|test result | PKT loss rate, latency, throupht, cpu usage |
+| | |
++--------------+--------------------------------------------------------------+
+
+Configration
+===========
+test_config:
+ tool: netperf
+ protocol: tcp
+ test_time: 20
+ tx_pkt_sizes: 64, 256, 1024, 4096, 8192, 16384, 32768, 65536
+ rx_pkt_sizes: 64, 256, 1024, 4096, 8192, 16384, 32768, 65536
+ cpu_load: 0.9
+ latency: 100000
+runner_config:
+ dashboard: "y"
+ dashboard_ip:
+ stack_create: yardstick
+ yardstick_test_ip:
+ yardstick_test_dir: "samples"
+ yardstick_testcase: "netperf_bottlenecks"
+++ /dev/null
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
-
-**************************
-Bottlenecks - user guide
-**************************
-
-.. This index.rst file is used to structure and organise the content in these directories.
-.. The documentation toolchain will create web and pdf versions of the content of this file.
-
-.. toctree::
- :maxdepth: 2
-
- ./rubbos/opnfv_bottlenecks_rubbos_basic.rst
- ./rubbos/opnfv_bottlenecks_rubbos_tc1101.rst
- ./rubbos/opnfv_bottlenecks_rubbos_tc1201.rst
- ./rubbos/opnfv_bottlenecks_rubbos_tc1301.rst
- ./rubbos/opnfv_bottlenecks_rubbos_tc1401.rst
- ./rubbos/opnfv_bottlenecks_rubbos_heavy_tc1101.rst
- ./vstf/opnfv_bottlenecks_vstf_Ti1.rst
- ./vstf/opnfv_bottlenecks_vstf_Ti2.rst
- ./vstf/opnfv_bottlenecks_vstf_Ti3.rst
- ./vstf/opnfv_bottlenecks_vstf_Tn1.rst
- ./vstf/opnfv_bottlenecks_vstf_Tn2.rst
- ./vstf/opnfv_bottlenecks_vstf_Tu1.rst
- ./vstf/opnfv_bottlenecks_vstf_Tu2.rst
- ./vstf/opnfv_bottlenecks_vstf_Tu3.rst
Code Review / bottlenecks.git / commitdiff