1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 Executing the functest suites (Ubuntu)
5 ======================================
10 This section assumes the following:
11 * The Functest Docker container is running
12 * The docker prompt is shown
13 * The Functest environment is ready (Functest CLI command 'functest env prepare'
16 If any of the above steps are missing please refer to the Functest Config Guide
17 as they are a prerequisite and all the commands explained in this section **must** be
18 performed **inside the container**.
20 The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
21 for the execution of Test Tiers or Test Cases::
23 root@22e436918db0:~/repos/functest/ci# functest tier --help
24 Usage: functest tier [OPTIONS] COMMAND [ARGS]...
27 -h, --help Show this message and exit.
30 get-tests Prints the tests in a tier.
31 list Lists the available tiers.
32 run Executes all the tests within a tier.
33 show Shows information about a tier.
34 root@22e436918db0:~/repos/functest/ci# functest testcase --help
36 Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
39 -h, --help Show this message and exit.
42 list Lists the available testcases.
43 run Executes a test case.
44 show Shows information about a test case.
46 More details on the existing Tiers and Test Cases can be seen with the 'list'
49 root@22e436918db0:~/repos/functest/ci# functest tier list
51 ['connection_check', 'api_check', 'snaps_health_check',]
53 ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'odl', 'rally_sanity', 'refstack_defcore', 'snaps_smoke']
55 ['doctor', 'domino', 'promise', security_scan']
57 ['tempest_full_parallel', 'rally_full']
59 ['cloudify_ims', 'orchestra_ims', 'vyos_vrouter']
63 root@22e436918db0:~/repos/functest/ci# functest testcase list
78 Note the list of test cases depend on the installer and the scenario.
80 More specific details on specific Tiers or Test Cases can be seen wih the
83 root@22e436918db0:~/repos/functest/ci# functest tier show smoke
84 +======================================================================+
86 +======================================================================+
88 | CI Loop: (daily)|(weekly) |
90 | Set of basic Functional tests to validate the OpenStack |
95 | - tempest_smoke_serial |
98 +----------------------------------------------------------------------+
102 root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial
103 +======================================================================+
104 | Testcase: tempest_smoke_serial |
105 +======================================================================+
107 | This test case runs the smoke subset of the OpenStack Tempest |
108 | suite. The list of test cases is generated by Tempest |
109 | automatically and depends on the parameters of the OpenStack |
115 +----------------------------------------------------------------------+
118 To execute a Test Tier or Test Case, the 'run' command is used::
120 root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
122 2017-08-16 12:35:51,799 - functest.ci.run_tests - INFO - ############################################
123 2017-08-16 12:35:51,799 - functest.ci.run_tests - INFO - Running tier 'healthcheck'
124 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ############################################
125 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO -
126 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ============================================
127 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - Running test case 'connection_check'...
128 2017-08-16 12:35:51,800 - functest.ci.run_tests - INFO - ============================================
129 2017-08-16 12:36:00,278 - functest.core.testcase - INFO - The results were successfully pushed to DB
130 2017-08-16 12:36:00,279 - functest.ci.run_tests - INFO - Test result:
131 +--------------------------+------------------+------------------+----------------+
132 | TEST CASE | PROJECT | DURATION | RESULT |
133 +--------------------------+------------------+------------------+----------------+
134 | connection_check | functest | 00:06 | PASS |
135 +--------------------------+------------------+------------------+----------------+
136 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO -
137 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - ============================================
138 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - Running test case 'api_check'...
139 2017-08-16 12:36:00,281 - functest.ci.run_tests - INFO - ============================================
140 2017-08-16 12:41:04,088 - functest.core.testcase - INFO - The results were successfully pushed to DB
141 2017-08-16 12:41:04,088 - functest.ci.run_tests - INFO - Test result:
142 +-------------------+------------------+------------------+----------------+
143 | TEST CASE | PROJECT | DURATION | RESULT |
144 +-------------------+------------------+------------------+----------------+
145 | api_check | functest | 05:03 | PASS |
146 +-------------------+------------------+------------------+----------------+
147 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO -
148 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - ============================================
149 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - Running test case 'snaps_health_check'...
150 2017-08-16 12:41:04,092 - functest.ci.run_tests - INFO - ============================================
151 2017-08-16 12:41:39,817 - functest.core.testcase - INFO - The results were successfully pushed to DB
152 2017-08-16 12:41:39,818 - functest.ci.run_tests - INFO - Test result:
153 +----------------------------+------------------+------------------+----------------+
154 | TEST CASE | PROJECT | DURATION | RESULT |
155 +----------------------------+------------------+------------------+----------------+
156 | snaps_health_check | functest | 00:35 | PASS |
157 +----------------------------+------------------+------------------+----------------+
161 root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
162 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - ============================================
163 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - Running test case 'vping_ssh'...
164 2017-08-16 12:41:39,821 - functest.ci.run_tests - INFO - ============================================
165 2017-08-16 12:42:49,861 - functest.core.testcase - INFO - The results were successfully pushed to DB
166 2017-08-16 12:42:49,861 - functest.ci.run_tests - INFO - Test result:
167 +-------------------+------------------+------------------+----------------+
168 | TEST CASE | PROJECT | DURATION | RESULT |
169 +-------------------+------------------+------------------+----------------+
170 | vping_ssh | functest | 00:47 | PASS |
171 +-------------------+------------------+------------------+----------------+
176 To list the test cases which are part of a specific Test Tier, the 'get-tests'
177 command is used with 'functest tier'::
179 root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck
180 Test cases in tier 'healthcheck':
181 ['connection_check', 'api_check', 'snaps_health_check']
184 Please note that for some scenarios some test cases might not be launched.
185 For example, the last example displayed only the 'odl' testcase for the given
186 environment. In this particular system the deployment does not support the 'ocl' SDN
187 Controller Test Case; for example.
189 **Important** If you use the command 'functest tier run <tier_name>', then the
190 Functest CLI utility will call **all valid Test Cases**, which belong to the
191 specified Test Tier, as relevant to scenarios deployed to the SUT environment.
192 Thus, the Functest CLI utility calculates automatically which tests can be
193 executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
194 which is passed in to the Functest docker container.
196 Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
199 * Run a single Test Case, specified by a valid choice of <testcase_name>
200 * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
202 Functest includes a cleaning mechanism in order to remove all the OpenStack
203 resources except those present before running any test. The script
204 *$REPOS_DIR/functest/functest/utils/openstack_snapshot.py* is called once when setting up
205 the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
206 all the OpenStack resources (images, networks, volumes, security groups, tenants,
207 users) so that an eventual cleanup does not remove any of these defaults.
209 The script **openstack_clean.py** which is located in
210 *$REPOS_DIR/functest/functest/utils/* is in charge of cleaning the OpenStack resources
211 that are not specified in the defaults file generated previously which is stored in
212 */home/opnfv/functest/conf/openstack_snapshot.yaml* in the Functest docker container.
214 It is important to mention that if there are new OpenStack resources created
215 manually after the snapshot done before running the tests, they will be removed,
216 unless you use the special method of invoking the test case with specific
217 suppression of clean up. (See the `Troubleshooting`_ section).
219 The reason to include this cleanup meachanism in Functest is because some
220 test suites create a lot of resources (users, tenants, networks, volumes etc.)
221 that are not always properly cleaned, so this function has been set to keep the
222 system as clean as it was before a full Functest execution.
224 Although the Functest CLI provides an easy way to run any test, it is possible to
225 do a direct call to the desired test script. For example:
227 python $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py
233 As mentioned previously, the Functest Docker container preparation as well as
234 invocation of Test Cases can be called within the container from the Jenkins CI
235 system. There are 3 jobs that automate the whole process. The first job runs all
236 the tests referenced in the daily loop (i.e. that must been run daily), the second
237 job runs the tests referenced in the weekly loop (usually long duration tests run
238 once a week maximum) and the third job allows testing test suite by test suite specifying
239 the test suite name. The user may also use either of these Jenkins jobs to execute
240 the desired test suites.
242 One of the most challenging task in the Danube release consists
243 in dealing with lots of scenarios and installers. Thus, when the tests are
244 automatically started from CI, a basic algorithm has been created in order to
245 detect whether a given test is runnable or not on the given scenario.
246 Some Functest test suites cannot be systematically run (e.g. ODL suite can not
247 be run on an ONOS scenario). The daily/weekly notion has been introduces in
248 Colorado in order to save CI time and avoid running systematically
249 long duration tests. It was not used in Colorado due to CI resource shortage.
250 The mechanism remains however as part of the CI evolution.
252 CI provides some useful information passed to the container as environment
255 * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
256 * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
257 * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
259 * controller = (odl|ocl|nosdn|onos)
260 * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|ovs_dpdk_bar)
263 The constraints per test case are defined in the Functest configuration file
264 */usr/local/lib/python2.7/dist-packages/functest/ci/testcases.yaml*::
270 ci_loop: '(daily)|(weekly)'
272 Set of basic Functional tests to validate the OpenStack deployment.
276 criteria: 'status == "PASS"'
279 This test case verifies: 1) SSH to an instance using floating
280 IPs over the public network. 2) Connectivity between 2 instances
281 over a private network.
284 scenario: '^((?!bgpvpn|odl_l3).)*$'
286 module: 'functest.opnfv_tests.openstack.vping.vping_ssh'
290 We may distinguish 2 levels in the test case description:
294 At the tier level, we define the following parameters:
296 * ci_loop: indicate if in automated mode, the test case must be run in dail and/or weekly jobs
297 * description: a high level view of the test case
299 For a given test case we defined:
300 * the name of the test case
301 * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
302 * blocking: if set to true, if the test is failed, the execution of the following tests is canceled
303 * the description of the test case
304 * the dependencies: a combination of 2 regex on the scenario and the installer name
305 * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests
307 For further details on abstraction classes, see developper guide.
309 Additional parameters have been added in the desription in the Database.
310 The target is to use the configuration stored in the Database and consider the
311 local file as backup if the Database is not reachable.
312 The additional fields related to a test case are:
313 * trust: we introduced this notion to put in place a mechanism of scenario promotion.
314 * Version: it indicates since which version you can run this test
315 * domains: the main domain covered by the test suite
316 * tags: a list of tags related to the test suite
318 The order of execution is the one defined in the file if all test cases are selected.
320 In CI daily job the tests are executed in the following order:
322 1) healthcheck (blocking)
323 2) smoke: both vPings are blocking
324 3) Feature project tests cases
326 In CI weekly job we add 2 tiers:
329 5) Components (Rally and Tempest long duration suites)
331 As explained before, at the end of an automated execution, the OpenStack resources
332 might be eventually removed.
333 Please note that a system snapshot is taken before any test case execution.
335 This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.
338 Executing Functest suites (Alpine)
339 ==================================
341 As mentioned in the configuration guide `[1]`_, Alpine docker containers have
342 been introduced in Euphrates.
343 Tier containers have been created.
344 Assuming that you pulled the container and your environement is ready, you can
345 simply run the tiers by typing (e.g. with functest-healthcheck)::
347 sudo docker run --env-file env \
348 -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/openstack.creds \
349 -v $(pwd)/images:/home/opnfv/functest/images \
350 opnfv/functest-healthcheck
354 +----------------------------+------------------+---------------------+------------------+----------------+
355 | TEST CASE | PROJECT | TIER | DURATION | RESULT |
356 +----------------------------+------------------+---------------------+------------------+----------------+
357 | connection_check | functest | healthcheck | 00:02 | PASS |
358 | api_check | functest | healthcheck | 03:19 | PASS |
359 | snaps_health_check | functest | healthcheck | 00:46 | PASS |
360 +----------------------------+------------------+---------------------+------------------+----------------+
362 You can run functest-healcheck, functest-smoke, functest-features,
363 functest-components and functest-vnf.
365 Please note that you may also use the CLI for manual tests using Alpine
369 Functest internal API
370 =====================
372 An internal API has been introduced in Euphrates. The goal is to trigger
373 Functest operations through an API in addition of the CLI.
374 This could be considered as a first step towards a pseudo micro services
375 approach where the different test projects could expose and consume APIs to the
378 In Euphrates the main method of the APIs are:
381 * Prepare Environment
387 * List all testcases within given tier
389 The API can be invoked as follows:
390 http://<functest_url>:5000/api/v1/functest/envs
394 .. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/#