1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 Executing the functest suites
5 =============================
10 This section assumes the following:
11 * The Functest Docker container is running
12 * The docker prompt is shown
13 * The Functest environment is ready (Functest CLI command 'functest env prepare'
16 If any of the above steps are missing please refer to the Functest Config Guide
17 as they are a prerequisite and all the commands explained in this section **must** be
18 performed **inside the container**.
20 Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a
21 new Functest CLI. One difference, is that tests run through the Functest CLI
22 will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this
23 document, where this difference is discussed.
25 The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
26 for the execution of Test Tiers or Test Cases::
28 root@22e436918db0:~/repos/functest/ci# functest tier --help
29 Usage: functest tier [OPTIONS] COMMAND [ARGS]...
32 -h, --help Show this message and exit.
35 get-tests Prints the tests in a tier.
36 list Lists the available tiers.
37 run Executes all the tests within a tier.
38 show Shows information about a tier.
39 root@22e436918db0:~/repos/functest/ci# functest testcase --help
41 Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
44 -h, --help Show this message and exit.
47 list Lists the available testcases.
48 run Executes a test case.
49 show Shows information about a test case.
51 More details on the existing Tiers and Test Cases can be seen with the 'list'
54 root@22e436918db0:~/repos/functest/ci# functest tier list
58 ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
62 ['doctor', 'security_scan']
64 ['tempest_full_parallel', 'rally_full']
70 root@22e436918db0:~/repos/functest/ci# functest testcase list
83 More specific details on specific Tiers or Test Cases can be seen wih the
86 root@22e436918db0:~/repos/functest/ci# functest tier show smoke
87 +======================================================================+
89 +======================================================================+
91 | CI Loop: (daily)|(weekly) |
93 | Set of basic Functional tests to validate the OpenStack |
98 | - tempest_smoke_serial |
101 +----------------------------------------------------------------------+
105 root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial
106 +======================================================================+
107 | Testcase: tempest_smoke_serial |
108 +======================================================================+
110 | This test case runs the smoke subset of the OpenStack Tempest |
111 | suite. The list of test cases is generated by Tempest |
112 | automatically and depends on the parameters of the OpenStack |
118 +----------------------------------------------------------------------+
121 To execute a Test Tier or Test Case, the 'run' command is used::
123 root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
124 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck'
125 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file...
126 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################
127 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck'
128 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################
129 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
130 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'...
131 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
132 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API...
133 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK!
134 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API...
135 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK!
136 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API...
137 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK!
138 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API...
139 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca
140 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks...
141 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets...
142 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers...
143 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK!
144 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API...
145 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK!
146 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP...
149 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK!
150 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed!
151 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
152 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources...
153 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
154 Version 1 is deprecated, use alternative version 2 instead.
155 WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead.
156 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances...
157 2016-06-30 11:48:24,439 - clean_openstack - INFO - -------------------------------------------
158 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images...
159 2016-06-30 11:48:35,853 - clean_openstack - INFO - -------------------------------------------
160 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes...
161 2016-06-30 11:48:37,344 - clean_openstack - INFO - -------------------------------------------
162 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs...
163 2016-06-30 11:48:37,467 - clean_openstack - INFO - -------------------------------------------
164 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects
165 2016-06-30 11:48:53,633 - clean_openstack - INFO - -------------------------------------------
166 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups...
167 2016-06-30 11:48:53,689 - clean_openstack - INFO - -------------------------------------------
168 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users...
169 2016-06-30 11:48:54,444 - clean_openstack - INFO - -------------------------------------------
170 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants...
171 2016-06-30 11:48:54,711 - clean_openstack - INFO - -------------------------------------------
175 root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
176 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
177 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
178 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
179 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
180 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
181 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'...
182 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
183 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
184 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
185 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
186 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
187 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
188 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
189 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
190 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
191 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
192 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
193 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
194 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
195 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
196 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
197 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
198 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
199 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
200 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
201 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
202 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
203 Version 1 is deprecated, use alternative version 2 instead.
208 To list the test cases which are part of a specific Test Tier, the 'get-tests'
209 command is used with 'functest tier'::
211 root@22e436918db0:~/repos/functest/ci# functest tier get-tests sdn_suites
212 Test cases in tier 'sdn_suites':
216 Please note that for some scenarios some test cases might not be launched.
217 For example, the last example displayed only the 'odl' testcase for the given
218 environment. In this particular system the deployment does not support the 'onos' SDN
219 Controller Test Case; for example.
221 **Important** If you use the command 'functest tier run <tier_name>', then the
222 Functest CLI utility will call **all valid Test Cases**, which belong to the
223 specified Test Tier, as relevant to scenarios deployed to the SUT environment.
224 Thus, the Functest CLI utility calculates automatically which tests can be
225 executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
226 which is passed in to the Functest docker container.
228 Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
231 * Run a single Test Case, specified by a valid choice of <testcase_name>
232 * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
236 root@22e436918db0:~/repos/functest/ci# functest testcase run all
237 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t all'
238 2016-06-30 12:03:28,628 - run_tests - INFO - Sourcing the OpenStack RC file...
239 2016-06-30 12:03:28,634 - run_tests - INFO - Tiers to be executed:
243 ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
247 ['doctor', 'security_scan']
249 ['tempest_full_parallel', 'rally_full']
252 2016-06-30 12:03:28,634 - run_tests - INFO - ############################################
253 2016-06-30 12:03:28,635 - run_tests - INFO - Running tier 'healthcheck'
254 2016-06-30 12:03:28,635 - run_tests - INFO - ############################################
255 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
256 2016-06-30 12:03:28,635 - run_tests - INFO - Running test case 'healthcheck'...
257 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
258 2016-06-30 12:03:28,651 - healtcheck - INFO - Testing Keystone API...
259 2016-06-30 12:03:36,676 - healtcheck - INFO - ...Keystone OK!
260 2016-06-30 12:03:36,679 - healtcheck - INFO - Testing Glance API...
265 Functest includes a cleaning mechanism in order to remove all the OpenStack
266 resources except those present before running any test. The script
267 *$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up
268 the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
269 all the OpenStack resources (images, networks, volumes, security groups, tenants,
270 users) so that an eventual cleanup does not remove any of these defaults.
272 The script **clean_openstack.py** which is located in
273 *$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is
274 in charge of cleaning the OpenStack resources that are not specified in the
275 defaults file generated previously which is stored in
276 */home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container.
278 It is important to mention that if there are new OpenStack resources created
279 manually after preparing the Functest environment, they will be removed, unless
280 you use the special method of invoking the test case with specific suppression
281 of clean up. (See the `Troubleshooting`_ section).
283 The reason to include this cleanup meachanism in Functest is because some
284 test suites such as Tempest or Rally create a lot of resources (users,
285 tenants, networks, volumes etc.) that are not always properly cleaned, so this
286 function has been set to keep the system as clean as it was before a
287 full Functest execution.
289 Although the Functest CLI provides an easy way to run any test, it is possible to
290 do a direct call to the desired test script. For example:
292 python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d
298 As mentioned previously, the Functest Docker container preparation as well as
299 invocation of Test Cases can be called within the container from the Jenkins CI
300 system. There are 3 jobs that automate the whole process. The first job runs all
301 the tests referenced in the daily loop (i.e. that must been run daily), the second
302 job runs the tests referenced in the weekly loop (usually long duration tests run
303 once a week maximum) and the third job allows testing test suite by test suite specifying
304 the test suite name. The user may also use either of these Jenkins jobs to execute
305 the desired test suites.
307 One of the most challenging task in the Colorado release consists
308 in dealing with lots of scenarios and installers. Thus, when the tests are
309 automatically started from CI, a basic algorithm has been created in order to
310 detect whether a given test is runnable or not on the given scenario.
311 Some Functest test suites cannot be systematically run (e.g. ODL suite can not
312 be run on an ONOS scenario). Moreover since Colorado, we also introduce the
313 notion of daily/weekly in order to save CI time and avoid running systematically
316 CI provides some useful information passed to the container as environment
319 * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
320 * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
321 * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
323 * controller = (odl|onos|ocl|nosdn)
324 * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|moon|multisites)
327 The constraints per test case are defined in the Functest configuration file
328 */home/opnfv/repos/functest/functest/ci/testcases.yaml*::
334 ci_loop: '(daily)|(weekly)'
336 First tier to be executed to verify the basic
337 operations in the VIM.
341 criteria: 'status == "PASS"'
344 This test case verifies the basic OpenStack services like
345 Keystone, Glance, Cinder, Neutron and Nova.
354 ci_loop: '(daily)|(weekly)'
356 Set of basic Functional tests to validate the OpenStack deployment.
360 criteria: 'status == "PASS"'
363 This test case verifies: 1) SSH to an instance using floating
364 IPs over the public network. 2) Connectivity between 2 instances
365 over a private network.
368 scenario: '^((?!bgpvpn|odl_l3).)*$'
371 We may distinguish 2 levels in the test case description:
375 At the tier level, we define the following parameters:
377 * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs
378 * description: a high level view of the test case
380 For a given test case we defined:
381 * the name of the test case
382 * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
383 * blocking: if set to true, if the test is failed, the execution of the following tests is canceled
384 * the description of the test case
385 * the dependencies: a combination of 2 regex on the scenario and the installer name
387 The order of execution is the one defined in the file if all test cases are selected.
389 In CI daily job the tests are executed in the following order:
391 1) healthcheck (blocking)
392 2) smoke: both vPings are blocking
393 3) SDN controller suites (blocking)
394 4) Feature project tests cases
396 In CI weekly job we add 2 tiers:
401 As explained before, at the end of an automated execution, the OpenStack resources
402 might be eventually removed.
403 Please note that a system snapshot is taken before any test case execution.
405 This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.