1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
3 .. (c) Huawei Technologies Co.,Ltd and others.
12 The POSCA (Parametric Bottlenecks Testing Catalogue) test suite
13 classifies the bottlenecks test cases and results into 5 categories.
14 Then the results will be analyzed and bottlenecks will be searched
15 among these categories.
17 The POSCA testsuite aims to locate the bottlenecks in parametric
18 manner and to decouple the bottlenecks regarding the deployment
20 The POSCA testsuite provides an user friendly way to profile and
21 understand the E2E system behavior and deployment requirements.
23 Goals of the POSCA testsuite:
24 a) Automatically locate the bottlenecks in a iterative manner.
25 b) Automatically generate the testing report for bottlenecks in different categories.
26 c) Implementing Automated Staging.
28 Scopes of the POSCA testsuite:
29 a) Modeling, Testing and Test Result analysis.
30 b) Parameters choosing and Algorithms.
32 Test stories of POSCA testsuite:
33 a) Factor test (Stress test): base test cases that Feature test and Optimization will be dependant on.
34 b) Feature test: test cases for features/scenarios.
35 c) Optimization test: test to tune the system parameter.
37 Detailed workflow is illutrated below.
39 * https://wiki.opnfv.org/display/bottlenecks
45 * Docker: https://docs.docker.com/engine/installation/
46 * For Ubuntu, please refer to https://docs.docker.com/engine/installation/linux/ubuntu/
48 * Docker-Compose: https://docs.docker.com/compose/
52 if [ -d usr/local/bin/docker-compose ]; then
53 rm -rf usr/local/bin/docker-compose
55 curl -L https://github.com/docker/compose/releases/download/1.11.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose
56 chmod +x /usr/local/bin/docker-compose
62 The test environment preparation, the installation of the testing tools,
63 the execution of the tests and the reporting/analyisis of POSCA test suite
65 A few steps are needed to run it locally.
67 In Euphrates, Bottlenecks has modified its framework to support installer-agnostic
68 testing which means that test cases could be executed over different deployments.
71 Downloading Bottlenecks Software
72 --------------------------------
78 git clone https://gerrit.opnfv.org/gerrit/bottlenecks
82 Preparing Python Virtual Evnironment
83 ------------------------------------
90 Preparing configuration/description files
91 -----------------------------------------
93 Put OpenStack RC file (admin_rc.sh), os_carcert and pod.yaml (pod descrition file) in /tmp directory.
94 Edit admin_rc.sh and add the following line
98 export OS_CACERT=/tmp/os_cacert
100 If you are using compass, fuel, apex or joid to deploy your openstack
101 environment, you could use the following command to get the required files.
105 bash /utils/env_prepare/config_prepare.sh -i <installer> [--debug]
107 Note that if we execute the command above, then admin_rc.sh and pod.yml gets created automatically in /tmp folder along with the line `export OS_CACERT=/tmp/os_cacert` added in admin_rc.sh file.
110 Executing Specified Testcase
111 ---------------------------
113 1. Bottlenecks provides a CLI interface to run the tests, which is one of the most convenient way since it is more close to our natural languge. An GUI interface with rest API will also be provided in later update.
117 bottlenecks testcase|teststory run <testname>
119 For the *testcase* command, testname should be as the same name of the test case configuration file located in testsuites/posca/testcase_cfg.
120 For stress tests in Danube/Euphrates, *testcase* should be replaced by either *posca_factor_ping* or *posca_factor_system_bandwidth*.
121 For the *teststory* command, a user can specify the test cases to be executed by defining it in a teststory configuration file located in testsuites/posca/testsuite_story. There is also an example there named *posca_factor_test*.
123 2. There are also other 2 ways to run test cases and test stories.
125 The first one is to use shell script.
129 bash run_tests.sh [-h|--help] -s <testsuite>|-c <testcase>
132 The second is to use python interpreter.
137 opts="--privileged=true -id"
138 docker_volume="-v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp"
139 docker run $opts --name bottlenecks-load-master $docker_volume opnfv/bottlenecks:latest /bin/bash
141 POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca"
142 docker exec bottlenecks-load-master python ${POSCA_SCRIPT}/../run_posca.py testcase|teststory <testname> ${REPORT}
148 Bottlenecks uses ELK to illustrate the testing results.
149 Asumming IP of the SUT (System Under Test) is denoted as ipaddr,
150 then the address of Kibana is http://[ipaddr]:5601. One can visit this address to see the illustrations.
151 Address for elasticsearch is http://[ipaddr]:9200. One can use any Rest Tool to visit the testing data stored in elasticsearch.
153 Cleaning Up Environment
154 -----------------------
161 If you want to clean the dockers that established during the test, you can excute the additional commands below.
165 bash run_tests.sh --cleanup
167 Note that you can also add cleanup parameter when you run a test case. Then environment will be automatically cleaned up when
170 Run POSCA through Community CI
171 ==============================
172 POSCA test cases are runned by OPNFV CI now. See https://build.opnfv.org for details of the building jobs.
173 Each building job is set up to execute a single test case. The test results/logs will be printed on the web page and
174 reported automatically to community MongoDB. There are two ways to report the results.
176 1. Report testing result by shell script
180 bash run_tests.sh [-h|--help] -s <testsuite>|-c <testcase> --report
182 2. Report testing result by python interpreter
187 opts="--privileged=true -id"
188 docker_volume="-v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp"
189 docker run $opts --name bottlenecks-load-master $docker_volume opnfv/bottlenecks:latest /bin/bash
192 POSCA_SCRIPT="/home/opnfv/bottlenecks/testsuites/posca"
193 docker exec bottlenecks_load-master python ${POSCA_SCRIPT}/../run_posca.py testcase|teststory <testcase> ${REPORT}
195 Test Result Description
196 =======================
197 * Please refer to release notes and also https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing