.. http://creativecommons.org/licenses/by/4.0
.. (c) Huawei Technologies Co.,Ltd and others.
-*********************
-Project Testing Guide
-*********************
+**********
+User Guide
+**********
For each *testsuite*, you can either setup *teststory* or *testcase* to run
certain test. *teststory* comprises several *testcases* as a set in one
configuration file. You could call *teststory* or *testcase* by using
-Bottlencks user interfaces.
+Bottlenecks user interfaces.
Details will be shown in the following section.
Brief Introdcution of the Test suites in Project Releases
=============================================================
-Brahmaputra: rubbos is introduced, which is an end2end NFVI perforamnce tool.
-Virtual switch test framework(VSTF) is also introduced,
-which is an test framework used for vswitch performance test.
+Brahmaputra:
-Colorado: rubbos is refactored by using puppet, which makes it quite flexible
-to configure with different number of load generator (Client), worker (tomcat).
-vstf is refactored by extracting the test case's configuration information.
+* rubbos is introduced, which is an end2end NFVI perforamnce tool.
+* Virtual switch test framework (VSTF) is also introduced, which is an test framework used for vswitch performance test.
-Danube: posca testsuite is introduced to implementing stress (factor), scenario and
-tuning test in parametric manner. Two testcases are developed and integrated into
-community CI pipeline. Rubbos and VSTF are not supported any more.
+Colorado:
+
+* rubbos is refactored by using puppet, and ease the integration with several load generators(Client) and worker(Tomcat).
+* VSTF is refactored by extracting the test case's configuration information.
+
+Danube:
+
+* posca testsuite is introduced to implement stress (factor), feature and tuning test in parametric manner.
+* Two testcases are developed and integrated into community CI pipeline.
+* Rubbos and VSTF are not supported any more.
+
+Euphrates:
+
+* Introduction of a simple monitoring module, i.e., Prometheus+Collectd+Node+Grafana to monitor the system behavior when executing stress tests.
+* Support VNF scale up/out tests to verify NFVI capability to adapt the resource consuming.
+* Extend Life-cycle test to data-plane to validate the system capability to handle concurrent networks usage.
+* Testing framework is revised to support installer-agnostic testing.
+
+Gambia:
+
+* Introduction of kubetenetes stress test for scaling capability of PoD management
+* Introduction of test scheduler supporting testing services ochestration while providing monitoring services
+
+Hunter:
+
+* Introduction of AI based algorithms to do anomaly detection and correlation analysis of Bottlenecks testing results
+# Enhancing test scheduler to supporting testing results graphing
+
+These enhancements and test cases help the end users to gain more comprehensive understanding of the SUT.
+Graphic reports of the system behavior additional to test cases are provided to indicate the confidence level of SUT.
+Installer-agnostic testing framework allow end user to do stress testing adaptively over either Open Source or commercial deployments.
Integration Description
=======================
+-------------+----------------------+----------------------+
| Danube | Compass | POSCA |
+-------------+----------------------+----------------------+
+| Euphrates | Any | POSCA |
++-------------+----------------------+----------------------+
+| Fraser | Any | POSCA |
++-------------+----------------------+----------------------+
+| Gambia | Any | POSCA, kubestone |
++-------------+----------------------+----------------------+
+| Hunter | Any | POSCA, kubestone |
++-------------+----------------------+----------------------+
Test suite & Test case Description
==================================
-+--------+-------------------------------+
-|POSCA | posca_factor_ping |
-| +-------------------------------+
-| | posca_factor_system_bandwidth |
-+--------+-------------------------------+
-|Rubbos | rubbos_basic |
-| +-------------------------------+
-| | rubbos_TC1101 |
-| +-------------------------------+
-| | rubbos_TC1201 |
-| +-------------------------------+
-| | rubbos_TC1301 |
-| +-------------------------------+
-| | rubbos_TC1401 |
-| +-------------------------------+
-| | rubbos_heavy_TC1101 |
-+--------+-------------------------------+
-|vstf | vstf_Ti1 |
-| +-------------------------------+
-| | vstf_Ti2 |
-| +-------------------------------+
-| | vstf_Ti3 |
-| +-------------------------------+
-| | vstf_Tn1 |
-| +-------------------------------+
-| | vstf_Tn2 |
-| +-------------------------------+
-| | vstf_Tu1 |
-| +-------------------------------+
-| | vstf_Tu2 |
-| +-------------------------------+
-| | vstf_Tu3 |
-+--------+-------------------------------+
++----------+----+-------------------------------------------+
+|POSCA | 1 | posca_factor_ping |
+| +----+-------------------------------------------+
+| | 2 | posca_factor_system_bandwidth |
+| +----+-------------------------------------------+
+| | 3 | posca_facotor_soak_througputs |
+| +----+-------------------------------------------+
+| | 4 | posca_feature_vnf_scale_up |
+| +----+-------------------------------------------+
+| | 5 | posca_feature_vnf_scale_out |
+| +----+-------------------------------------------+
+| | 6 | posca_factor_storperf |
+| +----+-------------------------------------------+
+| | 7 | posca_factor_multistack_storage_parallel |
+| +----+-------------------------------------------+
+| | 8 | posca_factor_multistack_storage |
+| +----+-------------------------------------------+
+| | 9 | posca_feature_moon_resources |
+| +----+-------------------------------------------+
+| | 10 | posca_feature_moon_tenants |
++----------+----+-------------------------------------------+
+|Kubestone | 1 | deployment_capacity |
++----------+----+-------------------------------------------+
+
+As for the abandoned test suite in the previous Bottlenecks releases, please
+refer to http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/deprecated.html.
Code Review / bottlenecks.git / blobdiff