Version History
===============
-+----------------+--------------------+---------------------------------+
-| *Date* | *Version* | *Comment* |
-| | | |
-+----------------+--------------------+---------------------------------+
-| Sept 22nd, 2016| 1.0 | Bottlenecks Colorado release 1.0|
-| | | |
-+----------------+--------------------+---------------------------------+
-| Feb 17nd, 2016 | 1.1 | Bottlenecks Danube release 1.0 |
-| | | |
-+----------------+--------------------+---------------------------------+
-
++----------------+--------------------+-----------------------------------+
+| *Date* | *Version* | *Comment* |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Sept 22nd, 2016| 1.0 | Bottlenecks Colorado release 1.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Feb 17nd, 2017 | 1.1 | Bottlenecks Danube release 1.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Mar 24nd, 2017 | 1.2 | Bottlenecks Danube release 1.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Mar 24nd, 2017 | 1.3 | Bottlenecks Danube release 1.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Apr 25th, 2017 | 1.4 | Bottlenecks Danube release 2.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Jun 19th, 2017 | 1.5 | Bottlenecks Danube release 3.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Sept 15th, 2017| 1.6 | Bottlenecks Euphrates release 5.0 |
+| | | |
++----------------+--------------------+-----------------------------------+
+| Nov 15th, 2017 | 1.6.1 | Bottlenecks Euphrates release 5.1 |
+| | | |
++----------------+--------------------+-----------------------------------+
Summary
=======
* Documentation generated with Sphinx
- * User guide
-
- * Installation Procedure
-
- * Release Notes (this document)
-
- * Platform Overview
+ * Release
- * Configuration Guide
+ * Release Notes
-* Bottlenecks test suite
+ * Testing
- * Jenkins Jobs for OPNFV community labs
+ * User Guide
-* Bottlenecks framework
-
-* Bottlenecks test cases
+ * Developer Guide
The *Bottlenecks software* is developed in the OPNFV community, by the
Bottlenecks_ team.
Release Data
============
-Bramaputra Release Data
+Euphrates Release Data
+-----------------------
+
++--------------------------------------+--------------------------------+
+| **Project** | Bottlenecks |
+| | |
++--------------------------------------+--------------------------------+
+| **Repo/tag** | * Bottlenecks/5.1.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Bottlenecks Docker image tag** | * 5.1.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Release designation** | * 5.1.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Release date** | * Nov 15th 2017 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Purpose of the delivery** | Euphrates stable release |
+| | |
++--------------------------------------+--------------------------------+
+
++--------------------------------------+--------------------------------+
+| **Project** | Bottlenecks |
+| | |
++--------------------------------------+--------------------------------+
+| **Repo/tag** | * Bottlenecks/5.0.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Bottlenecks Docker image tag** | * 5.0.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Release designation** | * 5.0.0 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Release date** | * Sept 15th 2017 |
+| | |
+| | |
++--------------------------------------+--------------------------------+
+| **Purpose of the delivery** | Euphrates stable release |
+| | |
++--------------------------------------+--------------------------------+
+
+Danube Release Data
-----------------------
-+--------------------------------------+--------------------------------------+
-| **Project** | Bottlenecks |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Repo/tag** | Bottlenecks/brahmaputra.1.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Bottlenecks Docker image tag** | brahmaputra.1.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Brahmaputra base release |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | February 25 2016 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | Brahmaputra base release |
-| | |
-+--------------------------------------+--------------------------------------+
++--------------------------------------+--------------------------------+
+| **Project** | Bottlenecks |
+| | |
++--------------------------------------+--------------------------------+
+| **Repo/tag** | * Bottlenecks/danube.1.0 |
+| | * Bottlenecks/danube.2.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Bottlenecks Docker image tag** | * danube.1.0 |
+| | * danube.2.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Release designation** | * Danube 1.0 |
+| | * Danube 2.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Release date** | * March 31st 2017 |
+| | * May 1st 2017 |
+| | |
++--------------------------------------+--------------------------------+
+| **Purpose of the delivery** | Danube stable release |
+| | |
++--------------------------------------+--------------------------------+
Colorado Release Data
-----------------------
-+--------------------------------------+--------------------------------------+
-| **Project** | Bottlenecks |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Repo/tag** | Bottlenecks/colorado.1.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Bottlenecks Docker image tag** | colorado.1.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Colorado base release |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | September 22 2016 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | Colorado base release |
-| | |
-+--------------------------------------+--------------------------------------+
++--------------------------------------+--------------------------------+
+| **Project** | Bottlenecks |
+| | |
++--------------------------------------+--------------------------------+
+| **Repo/tag** | Bottlenecks/colorado.1.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Bottlenecks Docker image tag** | * colorado.1.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Release designation** | * Colorado 1.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Release date** | * September 22 2016 |
+| | |
++--------------------------------------+--------------------------------+
+| **Purpose of the delivery** | Colorado stable release |
+| | |
++--------------------------------------+--------------------------------+
-Colorado Release Data
+Bramaputra Release Data
-----------------------
-* TODO
+
++--------------------------------------+--------------------------------+
+| **Project** | Bottlenecks |
+| | |
++--------------------------------------+--------------------------------+
+| **Repo/tag** | Bottlenecks/brahmaputra.1.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Bottlenecks Docker image tag** | * brahmaputra.1.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Release designation** | * Brahmaputra 1.0 |
+| | |
++--------------------------------------+--------------------------------+
+| **Release date** | * February 25 2016 |
+| | |
++--------------------------------------+--------------------------------+
+| **Purpose of the delivery** | Brahmaputra stable release |
+| | |
++--------------------------------------+--------------------------------+
+
Danube Deliverables
===================
Software Deliverables
---------------------
-**Bottlenecks source code <danube.1.0>**
+**Bottlenecks source code**
* https://gerrit.opnfv.org/gerrit/gitweb?p=bottlenecks.git;a=summary
Documentatiion Deliverables
---------------------------
-**Bottlenecks documentation <danube.1.0>**
+**Bottlenecks documentation <euphrates>**
-* Platformation Overview: http://artifacts.opnfv.org/bottlenecks/danube/1.0/docs/platformoverview/index.html
-* Configuration Guide: http://artifacts.opnfv.org/bottlenecks/danube/1.0/configurationguide/index.html
-* Installation Guide: http://artifacts.opnfv.org/bottlenecks/danube/docs/installationprocedure/index.html
-* http://artifacts.opnfv.org/bottlenecks/danube/1.0/releasenotes/index.html
-* User Guide: http://artifacts.opnfv.org/bottlenecks/danube/1.0/docs/userguide/index.html
+* Release Notes: http://docs.opnfv.org/en/stable-euphrates/submodules/bottlenecks/docs/release/release-notes/release_notes.html
+* User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/bottlenecks/docs/testing/user/userguide/index.html
+* Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/bottlenecks/docs/testing/developer/devguide/index.html
-Colorado Deliverables
-=====================
+**Bottlenecks documentation <danube>**
-Software Deliverables
----------------------
-
-**Bottlenecks source code <colorado.1.0>**
-
-* https://gerrit.opnfv.org/gerrit/gitweb?p=bottlenecks.git;a=summary
-
-
-Documentatiion Deliverables
----------------------------
+* Release Notes: http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/release/release-notes/release_notes.html
+* User Guide: http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/index.html
+* Developer Guide: http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/developer/devguide/index.html
**Bottlenecks documentation <colorado.1.0>**
-* Platformation Overview: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/platformoverview/index.html
+* Platform Overview: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/platformoverview/index.html
* Configuration Guide: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/configurationguide/index.html
* Installation Guide: http://artifacts.opnfv.org/bottlenecks/colorado/docs/installationprocedure/index.html
* http://artifacts.opnfv.org/bottlenecks/colorado/1.0/releasenotes/index.html
* User Guide: http://artifacts.opnfv.org/bottlenecks/colorado/1.0/docs/userguide/index.html
-Reason for Version
-==================
+Reason for Euphrates
+====================
-* In Danube, POSCA testsuite is introduced and two stress tests are implemented.
-* In Colorado, rubbos is refactored by using puppet, which makes it quite flexible to configure with different number of load generator(Client), worker(tomcat).
-* In Colorado, vstf is refactored by extracting the test case's configuration information.
+* Introduction of a simple monitoring module, i.e., Prometheus+Collectd+Node+Grafana to monitor the system behavior when executing stress tests.
+* Support VNF scale up/out tests to verify NFVI capability to adapt the resource consuming.
+* Extend Life-cycle test to data-plane to validate the system capability to handle concurrent networks usage.
+* Testing framework is revised to support installer-agnostic testing.
+
+These enhancements and test cases help the end users to gain more comprehensive understanding of the SUT.
+Graphic reports of the system behavior additional to test cases are provided to indicate the confidence level of SUT.
+Installer-agnostic testing framework allow end user to do stress testing adaptively over either Open Source or commercial deployments.
Known restrictions/issues
=========================
-* TODO
+* Bottlenecks uses Yardstick to do stress tests by iteratively calling yardstick running test cases and in the meantime increasing load.
+
+ * Sometimes, Yardstick just waits for the test environment preparing.
+ This cause troubles since it seams to get Yardstick into a loop to keep waiting.
+ For OPNFV CI, this loop will be automatically stopped because of the default timeout setting. However, for local testing, user should stop this loop manually.
+
+ * Sometimes, Yardstick will return empty test results with test flag indicating test is excuted succefully.
+ It maybe due to the environment issue or poor internet connection causing testing tools are not installed successfully.
+
+* Sometimes, a process will go to sleep state. In this case when a running tool go to sleep state, we try to call it twice. Normally, it will response. This applies to the traffic generator, i.e., netperf.
Test results
- jenkins logs on CI: https://build.opnfv.org/ci/view/bottlenecks/
+The test results are reported to MongoDB. An example is given below.
+
+::
+
+ {
+ "project_name": "bottlenecks",
+ "scenario": "os-odl_l2-nofeature-ha",
+ "stop_date": null,
+ "trust_indicator": null,
+ "case_name": "posca_stress_ping",
+ "build_tag": "bottlenecks-compass-posca_stress_ping-baremetal-daily-master",
+ "version": "master",
+ "pod_name": "huawei-pod2",
+ "criteria": "PASS",
+ "installer": "compass",
+ "_id": "58cf4d3e32c829000a1150a8",
+ "start_date": "2017-3-9 4:33:04",
+ "details": {}
+ }
+
+For more information, please refer to: https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing
+ - Results reported in MongoDB could find at http://testresults.opnfv.org/test/api/v1/results?project=bottlenecks
+ - Test Cases are defined in http://testresults.opnfv.org/test/api/v1/results?project=bottlenecks
Open JIRA tickets
=================
-+------------------+-----------------------------------------------+
-| JIRA | Description |
-+==================+===============================================+
-+------------------+-----------------------------------------------+
-+------------------+-----------------------------------------------+
-+------------------+-----------------------------------------------+
-+------------------+-----------------------------------------------+
-+------------------+-----------------------------------------------+
++------------------+----------------------------------------------------+
+| JIRA | Description |
++==================+====================================================+
+| BOTTLENECK-147 | Investigting why calling remote docker client |
++------------------+----------------------------------------------------+
+| BOTTLENECK-207 | Yardstick loop when executing test cases |
++------------------+----------------------------------------------------+
+| BOTTLENECK-208 | Yardstick empty results when executing test cases |
++------------------+----------------------------------------------------+
+| BOTTLENECK-209 | Sleep process when executing test cases |
++------------------+----------------------------------------------------+
Useful links
- Bottlenecks CI dashboard: https://build.opnfv.org/ci/view/bottlenecks
- Bottlenecks IRC chanel: #opnfv-bottlenecks
-
-
Code Review / bottlenecks.git / blobdiff