docs/testing/user/userguide/overview.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3 .. (c) Huawei Technologies Co.,Ltd and others.
   4
   5 **********
   6 User Guide
   7 **********
   8
   9 For each *testsuite*, you can either setup *teststory* or *testcase* to run
  10 certain test. *teststory* comprises several *testcases* as a set in one
  11 configuration file. You could call *teststory* or *testcase* by using
  12 Bottlenecks user interfaces.
  13 Details will be shown in the following section.
  14
  15 Brief Introdcution of the Test suites in Project Releases
  16 =============================================================
  17
  18 Brahmaputra:
  19
  20 * rubbos is introduced, which is an end2end NFVI perforamnce tool.
  21 * Virtual switch test framework (VSTF) is also introduced, which is an test framework used for vswitch performance test.
  22
  23 Colorado:
  24
  25 * rubbos is refactored by using puppet, and ease the integration with several load generators(Client) and worker(Tomcat).
  26 * VSTF is refactored by extracting the test case's configuration information.
  27
  28 Danube:
  29
  30 * posca testsuite is introduced to implement stress (factor), feature and tuning test in parametric manner.
  31 * Two testcases are developed and integrated into community CI pipeline.
  32 * Rubbos and VSTF are not supported any more.
  33
  34 Euphrates:
  35
  36 * Introduction of a simple monitoring module, i.e., Prometheus+Collectd+Node+Grafana to monitor the system behavior when executing stress tests.
  37 * Support VNF scale up/out tests to verify NFVI capability to adapt the resource consuming.
  38 * Extend Life-cycle test to data-plane to validate the system capability to handle concurrent networks usage.
  39 * Testing framework is revised to support installer-agnostic testing.
  40
  41 Gambia:
  42
  43 * Introduction of kubetenetes stress test for scaling capability of PoD management
  44 * Introduction of test scheduler supporting testing services ochestration while providing monitoring services
  45
  46 Hunter:
  47
  48 * Introduction of AI based algorithms to do anomaly detection and correlation analysis of Bottlenecks testing results
  49 # Enhancing test scheduler to supporting testing results graphing
  50
  51 These enhancements and test cases help the end users to gain more comprehensive understanding of the SUT.
  52 Graphic reports of the system behavior additional to test cases are provided to indicate the confidence level of SUT.
  53 Installer-agnostic testing framework allow end user to do stress testing adaptively over either Open Source or commercial deployments.
  54
  55 Integration Description
  56 =======================
  57 +-------------+----------------------+----------------------+
  58 | Release     | Integrated Installer | Supported Testsuite  |
  59 +-------------+----------------------+----------------------+
  60 | Brahmaputra |    Fuel              | Rubbos, VSTF         |
  61 +-------------+----------------------+----------------------+
  62 | Colorado    |    Compass           | Rubbos, VSTF         |
  63 +-------------+----------------------+----------------------+
  64 | Danube      |    Compass           | POSCA                |
  65 +-------------+----------------------+----------------------+
  66 | Euphrates   |    Any               | POSCA                |
  67 +-------------+----------------------+----------------------+
  68 | Fraser      |    Any               | POSCA                |
  69 +-------------+----------------------+----------------------+
  70 | Gambia      |    Any               | POSCA, kubestone     |
  71 +-------------+----------------------+----------------------+
  72 | Hunter      |    Any               | POSCA, kubestone     |
  73 +-------------+----------------------+----------------------+
  74
  75 Test suite & Test case Description
  76 ==================================
  77 +----------+----+-------------------------------------------+
  78 |POSCA     | 1  | posca_factor_ping                         |
  79 |          +----+-------------------------------------------+
  80 |          | 2  | posca_factor_system_bandwidth             |
  81 |          +----+-------------------------------------------+
  82 |          | 3  | posca_facotor_soak_througputs             |
  83 |          +----+-------------------------------------------+
  84 |          | 4  | posca_feature_vnf_scale_up                |
  85 |          +----+-------------------------------------------+
  86 |          | 5  | posca_feature_vnf_scale_out               |
  87 |          +----+-------------------------------------------+
  88 |          | 6  | posca_factor_storperf                     |
  89 |          +----+-------------------------------------------+
  90 |          | 7  | posca_factor_multistack_storage_parallel  |
  91 |          +----+-------------------------------------------+
  92 |          | 8  | posca_factor_multistack_storage           |
  93 |          +----+-------------------------------------------+
  94 |          | 9  | posca_feature_moon_resources              |
  95 |          +----+-------------------------------------------+
  96 |          | 10 | posca_feature_moon_tenants                |
  97 +----------+----+-------------------------------------------+
  98 |Kubestone | 1  | deployment_capacity                       |
  99 +----------+----+-------------------------------------------+
 100
 101 As for the abandoned test suite in the previous Bottlenecks releases, please
 102 refer to http://docs.opnfv.org/en/stable-danube/submodules/bottlenecks/docs/testing/user/userguide/deprecated.html.