1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 Executing the functest suites
5 =============================
10 This section assumes the following:
11 * The Functest Docker container is running
12 * The docker prompt is shown
13 * The Functest environment is ready (Functest CLI command 'functest env prepare'
16 If any of the above steps are missing please refer to the Functest Config Guide
17 as they are a prerequisite and all the commands explained in this section **must** be
18 performed **inside the container**.
20 The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
21 for the execution of Test Tiers or Test Cases::
23 root@22e436918db0:~/repos/functest/ci# functest tier --help
24 Usage: functest tier [OPTIONS] COMMAND [ARGS]...
27 -h, --help Show this message and exit.
30 get-tests Prints the tests in a tier.
31 list Lists the available tiers.
32 run Executes all the tests within a tier.
33 show Shows information about a tier.
34 root@22e436918db0:~/repos/functest/ci# functest testcase --help
36 Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
39 -h, --help Show this message and exit.
42 list Lists the available testcases.
43 run Executes a test case.
44 show Shows information about a test case.
46 More details on the existing Tiers and Test Cases can be seen with the 'list'
49 root@22e436918db0:~/repos/functest/ci# functest tier list
51 ['healthcheck', 'connection_check', 'api_check',]
53 ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity', 'snaps_smoke', 'odl']
55 ['doctor', 'security_scan']
57 ['tempest_full_parallel', 'rally_full']
63 root@22e436918db0:~/repos/functest/ci# functest testcase list
79 More specific details on specific Tiers or Test Cases can be seen wih the
82 root@22e436918db0:~/repos/functest/ci# functest tier show smoke
83 +======================================================================+
85 +======================================================================+
87 | CI Loop: (daily)|(weekly) |
89 | Set of basic Functional tests to validate the OpenStack |
94 | - tempest_smoke_serial |
97 +----------------------------------------------------------------------+
101 root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial
102 +======================================================================+
103 | Testcase: tempest_smoke_serial |
104 +======================================================================+
106 | This test case runs the smoke subset of the OpenStack Tempest |
107 | suite. The list of test cases is generated by Tempest |
108 | automatically and depends on the parameters of the OpenStack |
114 +----------------------------------------------------------------------+
117 To execute a Test Tier or Test Case, the 'run' command is used::
119 root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
120 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck'
121 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file...
122 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################
123 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck'
124 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################
125 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
126 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'...
127 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
128 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API...
129 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK!
130 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API...
131 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK!
132 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API...
133 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK!
134 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API...
135 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca
136 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks...
137 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets...
138 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers...
139 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK!
140 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API...
141 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK!
142 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP...
145 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK!
146 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed!
147 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
148 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources...
149 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
150 Version 1 is deprecated, use alternative version 2 instead.
151 WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead.
152 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances...
153 2016-06-30 11:48:24,439 - clean_openstack - INFO - -------------------------------------------
154 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images...
155 2016-06-30 11:48:35,853 - clean_openstack - INFO - -------------------------------------------
156 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes...
157 2016-06-30 11:48:37,344 - clean_openstack - INFO - -------------------------------------------
158 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs...
159 2016-06-30 11:48:37,467 - clean_openstack - INFO - -------------------------------------------
160 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects
161 2016-06-30 11:48:53,633 - clean_openstack - INFO - -------------------------------------------
162 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups...
163 2016-06-30 11:48:53,689 - clean_openstack - INFO - -------------------------------------------
164 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users...
165 2016-06-30 11:48:54,444 - clean_openstack - INFO - -------------------------------------------
166 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants...
167 2016-06-30 11:48:54,711 - clean_openstack - INFO - -------------------------------------------
171 root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
172 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
173 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
174 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
175 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
176 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
177 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'...
178 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
179 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
180 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
181 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
182 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
183 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
184 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
185 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
186 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
187 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
188 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
189 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
190 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
191 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
192 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
193 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
194 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
195 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
196 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
197 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
198 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
199 Version 1 is deprecated, use alternative version 2 instead.
204 To list the test cases which are part of a specific Test Tier, the 'get-tests'
205 command is used with 'functest tier'::
207 root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck
208 Test cases in tier 'healthcheck':
212 Please note that for some scenarios some test cases might not be launched.
213 For example, the last example displayed only the 'odl' testcase for the given
214 environment. In this particular system the deployment does not support the 'onos' SDN
215 Controller Test Case; for example.
217 **Important** If you use the command 'functest tier run <tier_name>', then the
218 Functest CLI utility will call **all valid Test Cases**, which belong to the
219 specified Test Tier, as relevant to scenarios deployed to the SUT environment.
220 Thus, the Functest CLI utility calculates automatically which tests can be
221 executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
222 which is passed in to the Functest docker container.
224 Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
227 * Run a single Test Case, specified by a valid choice of <testcase_name>
228 * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
230 Functest includes a cleaning mechanism in order to remove all the OpenStack
231 resources except those present before running any test. The script
232 *$REPOS_DIR/functest/functest/utils/generate_defaults.py* is called once when setting up
233 the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
234 all the OpenStack resources (images, networks, volumes, security groups, tenants,
235 users) so that an eventual cleanup does not remove any of these defaults.
237 The script **clean_openstack.py** which is located in
238 *$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is
239 in charge of cleaning the OpenStack resources that are not specified in the
240 defaults file generated previously which is stored in
241 */home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container.
243 It is important to mention that if there are new OpenStack resources created
244 manually after preparing the Functest environment, they will be removed, unless
245 you use the special method of invoking the test case with specific suppression
246 of clean up. (See the `Troubleshooting`_ section).
248 The reason to include this cleanup meachanism in Functest is because some
249 test suites such as Tempest or Rally create a lot of resources (users,
250 tenants, networks, volumes etc.) that are not always properly cleaned, so this
251 function has been set to keep the system as clean as it was before a
252 full Functest execution.
254 Although the Functest CLI provides an easy way to run any test, it is possible to
255 do a direct call to the desired test script. For example:
257 python $REPOS_DIR/functest/functest/opnfv_tests/OpenStack/vPing/vPing_ssh.py -d
263 As mentioned previously, the Functest Docker container preparation as well as
264 invocation of Test Cases can be called within the container from the Jenkins CI
265 system. There are 3 jobs that automate the whole process. The first job runs all
266 the tests referenced in the daily loop (i.e. that must been run daily), the second
267 job runs the tests referenced in the weekly loop (usually long duration tests run
268 once a week maximum) and the third job allows testing test suite by test suite specifying
269 the test suite name. The user may also use either of these Jenkins jobs to execute
270 the desired test suites.
272 One of the most challenging task in the Danube release consists
273 in dealing with lots of scenarios and installers. Thus, when the tests are
274 automatically started from CI, a basic algorithm has been created in order to
275 detect whether a given test is runnable or not on the given scenario.
276 Some Functest test suites cannot be systematically run (e.g. ODL suite can not
277 be run on an ONOS scenario). The daily/weekly notion has been introduces in
278 Colorado in order to save CI time and avoid running systematically
279 long duration tests. It was not used in Colorado due to CI resource shortage.
280 The mechanism remains however as part of the CI evolution.
282 CI provides some useful information passed to the container as environment
285 * Installer (apex|compass|daisy|fuel|joid), stored in INSTALLER_TYPE
286 * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
287 * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
289 * controller = (odl|onos|ocl|nosdn)
290 * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|moon|multisites)
293 The constraints per test case are defined in the Functest configuration file
294 */home/opnfv/repos/functest/functest/ci/testcases.yaml*::
300 ci_loop: '(daily)|(weekly)'
302 First tier to be executed to verify the basic
303 operations in the VIM.
307 criteria: 'status == "PASS"'
310 This test case verifies the basic OpenStack services like
311 Keystone, Glance, Cinder, Neutron and Nova.
320 ci_loop: '(daily)|(weekly)'
322 Set of basic Functional tests to validate the OpenStack deployment.
326 criteria: 'status == "PASS"'
329 This test case verifies: 1) SSH to an instance using floating
330 IPs over the public network. 2) Connectivity between 2 instances
331 over a private network.
334 scenario: '^((?!bgpvpn|odl_l3).)*$'
336 module: 'functest.opnfv_tests.openstack.vping.vping_ssh'
340 We may distinguish 2 levels in the test case description:
344 At the tier level, we define the following parameters:
346 * ci_loop: indicate if in automated mode, the test case must be run in daily and/or weekly jobs
347 * description: a high level view of the test case
349 For a given test case we defined:
350 * the name of the test case
351 * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
352 * blocking: if set to true, if the test is failed, the execution of the following tests is canceled
353 * the description of the test case
354 * the dependencies: a combination of 2 regex on the scenario and the installer name
355 * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests
357 For further details on abstraction classes, see developper guide.
359 Additional parameters have been added in the desription in the Database.
360 The target is to use the configuration stored in the Database and consider the
361 local file as backup if the Database is not reachable.
362 The additional fields related to a test case are:
363 * trust: we introduced this notion to put in place a mechanism of scenario promotion.
364 * Version: it indicates since which version you can run this test
365 * domains: the main domain covered by the test suite
366 * tags: a list of tags related to the test suite
368 The order of execution is the one defined in the file if all test cases are selected.
370 In CI daily job the tests are executed in the following order:
372 1) healthcheck (blocking)
373 2) smoke: both vPings are blocking
374 3) Feature project tests cases
376 In CI weekly job we add 2 tiers:
379 5) Components (Rally and Tempest long duration suites)
381 As explained before, at the end of an automated execution, the OpenStack resources
382 might be eventually removed.
383 Please note that a system snapshot is taken before any test case execution.
385 This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.