1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 Executing the functest suites
5 =============================
10 This section assumes the following:
11 * The Functest Docker container is running
12 * The docker prompt is shown
13 * The Functest environment is ready (Functest CLI command 'functest env prepare'
16 If any of the above steps are missing please refer to the Functest Config Guide
17 as they are a prerequisite and all the commands explained in this section **must** be
18 performed **inside the container**.
20 Note: In Colorado release, the scripts **run_tests.sh** is now replaced with a
21 new Functest CLI. One difference, is that tests run through the Functest CLI
22 will always clean-up OpenStack resources. See the `Troubleshooting`_ section of this
23 document, where this difference is discussed.
25 The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
26 for the execution of Test Tiers or Test Cases::
28 root@22e436918db0:~/repos/functest/ci# functest tier --help
29 Usage: functest tier [OPTIONS] COMMAND [ARGS]...
32 -h, --help Show this message and exit.
35 get-tests Prints the tests in a tier.
36 list Lists the available tiers.
37 run Executes all the tests within a tier.
38 show Shows information about a tier.
39 root@22e436918db0:~/repos/functest/ci# functest testcase --help
43 Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
46 -h, --help Show this message and exit.
49 list Lists the available testcases.
50 run Executes a test case.
51 show Shows information about a test case.
53 More details on the existing Tiers and Test Cases can be seen with the 'list'
56 root@22e436918db0:~/repos/functest/ci# functest tier list
60 ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
64 ['doctor', 'security_scan']
66 ['tempest_full_parallel', 'rally_full']
72 root@22e436918db0:~/repos/functest/ci# functest testcase list
85 More specific details on specific Tiers or Test Cases can be seen wih the
88 root@22e436918db0:~/repos/functest/ci# functest tier show smoke
89 +======================================================================+
91 +======================================================================+
93 | CI Loop: (daily)|(weekly) |
95 | Set of basic Functional tests to validate the OpenStack |
100 | - tempest_smoke_serial |
103 +----------------------------------------------------------------------+
107 root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial
108 +======================================================================+
109 | Testcase: tempest_smoke_serial |
110 +======================================================================+
112 | This test case runs the smoke subset of the OpenStack Tempest |
113 | suite. The list of test cases is generated by Tempest |
114 | automatically and depends on the parameters of the OpenStack |
120 +----------------------------------------------------------------------+
123 To execute a Test Tier or Test Case, the 'run' command is used::
125 root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
126 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t healthcheck'
127 2016-06-30 11:44:56,933 - run_tests - INFO - Sourcing the OpenStack RC file...
128 2016-06-30 11:44:56,937 - run_tests - INFO - ############################################
129 2016-06-30 11:44:56,938 - run_tests - INFO - Running tier 'healthcheck'
130 2016-06-30 11:44:56,938 - run_tests - INFO - ############################################
131 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
132 2016-06-30 11:44:56,938 - run_tests - INFO - Running test case 'healthcheck'...
133 2016-06-30 11:44:56,938 - run_tests - INFO - ============================================
134 2016-06-30 11:44:56,953 - healtcheck - INFO - Testing Keystone API...
135 2016-06-30 11:45:05,351 - healtcheck - INFO - ...Keystone OK!
136 2016-06-30 11:45:05,354 - healtcheck - INFO - Testing Glance API...
137 2016-06-30 11:45:29,746 - healtcheck - INFO - ... Glance OK!
138 2016-06-30 11:45:29,749 - healtcheck - INFO - Testing Cinder API...
139 2016-06-30 11:45:37,502 - healtcheck - INFO - ...Cinder OK!
140 2016-06-30 11:45:37,505 - healtcheck - INFO - Testing Neutron API...
141 2016-06-30 11:45:39,664 - healtcheck - INFO - External network found. ccd98ad6-d34a-4768-b03c-e28ecfcd51ca
142 2016-06-30 11:45:39,667 - healtcheck - INFO - 1. Create Networks...
143 2016-06-30 11:45:44,227 - healtcheck - INFO - 2. Create subnets...
144 2016-06-30 11:45:46,805 - healtcheck - INFO - 4. Create Routers...
145 2016-06-30 11:45:54,261 - healtcheck - INFO - ...Neutron OK!
146 2016-06-30 11:45:54,264 - healtcheck - INFO - Testing Nova API...
147 2016-06-30 11:47:12,272 - healtcheck - INFO - ...Nova OK!
148 2016-06-30 11:47:12,274 - healtcheck - INFO - Checking if instances get an IP from DHCP...
151 2016-06-30 11:48:17,832 - healtcheck - INFO - ...DHCP OK!
152 2016-06-30 11:48:17,835 - healtcheck - INFO - Health check passed!
153 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
154 2016-06-30 11:48:17,837 - clean_openstack - INFO - Cleaning OpenStack resources...
155 2016-06-30 11:48:17,837 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
156 Version 1 is deprecated, use alternative version 2 instead.
157 WARNING:cinderclient.api_versions:Version 1 is deprecated, use alternative version 2 instead.
158 2016-06-30 11:48:18,272 - clean_openstack - INFO - Removing Nova instances...
159 2016-06-30 11:48:24,439 - clean_openstack - INFO - -------------------------------------------
160 2016-06-30 11:48:24,440 - clean_openstack - INFO - Removing Glance images...
161 2016-06-30 11:48:35,853 - clean_openstack - INFO - -------------------------------------------
162 2016-06-30 11:48:35,854 - clean_openstack - INFO - Removing Cinder volumes...
163 2016-06-30 11:48:37,344 - clean_openstack - INFO - -------------------------------------------
164 2016-06-30 11:48:37,344 - clean_openstack - INFO - Removing floating IPs...
165 2016-06-30 11:48:37,467 - clean_openstack - INFO - -------------------------------------------
166 2016-06-30 11:48:37,467 - clean_openstack - INFO - Removing Neutron objects
167 2016-06-30 11:48:53,633 - clean_openstack - INFO - -------------------------------------------
168 2016-06-30 11:48:53,633 - clean_openstack - INFO - Removing Security groups...
169 2016-06-30 11:48:53,689 - clean_openstack - INFO - -------------------------------------------
170 2016-06-30 11:48:53,689 - clean_openstack - INFO - Removing Users...
171 2016-06-30 11:48:54,444 - clean_openstack - INFO - -------------------------------------------
172 2016-06-30 11:48:54,444 - clean_openstack - INFO - Removing Tenants...
173 2016-06-30 11:48:54,711 - clean_openstack - INFO - -------------------------------------------
177 root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
178 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
179 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
180 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
181 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
182 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
183 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img'...
184 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
185 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
186 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
187 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
188 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
189 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
190 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
191 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
192 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
193 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
194 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
195 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
196 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
197 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
198 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
199 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
200 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
201 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
202 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
203 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
204 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
205 Version 1 is deprecated, use alternative version 2 instead.
210 To list the test cases which are part of a specific Test Tier, the 'get-tests'
211 command is used with 'functest tier'::
213 root@22e436918db0:~/repos/functest/ci# functest tier get-tests sdn_suites
214 Test cases in tier 'sdn_suites':
218 Please note that for some scenarios some test cases might not be launched.
219 For example, the last example displayed only the 'odl' testcase for the given
220 environment. In this system the deployment does not support the 'onos' SDN
221 Controller Test Case; for example.
223 **Important** If you use the command 'functest tier run <tier_name>', then the
224 Functest CLI utility will call **all valid Test Cases**, which belong to the
225 specified Test Tier, as relevant to scenarios deployed to the SUT environment.
226 Thus, the Functest CLI utility calculates automatically which tests can be
227 executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
228 which is passed in to the Functest docker container.
230 Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
233 * Run a single Test Case, specified by a valid choice of <testcase_name>
234 * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
238 root@22e436918db0:~/repos/functest/ci# functest testcase run all
239 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t all'
240 2016-06-30 12:03:28,628 - run_tests - INFO - Sourcing the OpenStack RC file...
241 2016-06-30 12:03:28,634 - run_tests - INFO - Tiers to be executed:
245 ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity']
249 ['doctor', 'security_scan']
251 ['tempest_full_parallel', 'rally_full']
254 2016-06-30 12:03:28,634 - run_tests - INFO - ############################################
255 2016-06-30 12:03:28,635 - run_tests - INFO - Running tier 'healthcheck'
256 2016-06-30 12:03:28,635 - run_tests - INFO - ############################################
257 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
258 2016-06-30 12:03:28,635 - run_tests - INFO - Running test case 'healthcheck'...
259 2016-06-30 12:03:28,635 - run_tests - INFO - ============================================
260 2016-06-30 12:03:28,651 - healtcheck - INFO - Testing Keystone API...
261 2016-06-30 12:03:36,676 - healtcheck - INFO - ...Keystone OK!
262 2016-06-30 12:03:36,679 - healtcheck - INFO - Testing Glance API...
267 Functest includes cleaning mechanism in order to remove all the OpenStack
268 resources except what was present before running any test. The script
269 *$repos_dir/functest/utils/generate_defaults.py* is called once when setting up
270 the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
271 all the OpenStack resources (images, networks, volumes, security groups, tenants,
272 users) so that an eventual cleanup does not remove any of this defaults.
274 The script **clean_openstack.py** which is located in
275 *$repos_dir/functest/utils/* is normally called after a test execution. It is
276 in charge of cleaning the OpenStack resources that are not specified in the
277 defaults file generated previously which is stored in
278 */home/opnfv/functest/conf/os_defaults.yaml* in the Functest docker container.
280 It is important to mention that if there are new OpenStack resources created
281 manually after preparing the Functest environment, they will be removed, unless
282 you use the special method of invoking the test case with specific suppression
283 of clean up. (See the `Troubleshooting`_ section).
285 The reason to include this cleanup meachanism in Functest is because some
286 test suites such as Tempest or Rally create a lot of resources (users,
287 tenants, networks, volumes etc.) that are not always properly cleaned, so this
288 function has been set to keep the system as clean as it was before a
289 full Functest execution.
291 Although the Functest CLI provides an easy way to run any test, it is possible to
292 do a direct call to the desired test script. For example:
294 python $repos_dir/functest/testcases/OpenStack/vPing/vPing_ssh.py -d
300 **TODO** Jose, the next section has not yet been revised!
302 As mentioned previously, the Functest Docker container preparation as well as
303 invocation of Test Cases can be called within the container from the Jenkins CI
304 system. There are 2 jobs that automate the whole process. The first job runs all
305 the tests and the second job allows testing test suite by test suite specifying
306 the test suite name. The user may also use either of these Jenkins jobs to execute
307 the desired test suites.
309 One of the most challenging task in the Brahmaputra release consists
310 in dealing with lots of scenarios and installers. Thus, when the tests are
311 automatically started from CI, a basic algorithm has been created in order to
312 detect whether a given test is runnable or not on the given scenario.
313 Some Functest test suites cannot be systematically run (e.g. ODL suite can not
314 be run on an ONOS scenario).
316 CI provides some useful information passed to the container as environment
319 * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
320 * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
321 * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
323 * controller = (odl|onos|ocl|nosdn)
324 * feature = (ovs(dpdk)|kvm|sfc|bgpvpn)
327 The constraints per test case are defined in the Functest configuration file
328 */home/opnfv/functest/config/config_functest.yaml*::
333 scenario: '(ocl)|(odl)|(nosdn)'
336 scenario: '(ocl)|(odl)|(nosdn)'
345 At the end of the Functest environment creation, a file
346 */home/opnfv/functest/conf/testcase-list.txt* is created with the list of
347 all the runnable tests.
348 Functest considers the static constraints as regular expressions and compare them
349 with the given scenario name.
350 For instance, ODL suite can be run only on an scenario including 'odl' in its name.
352 The order of execution is also described in the Functest configuration file::
366 #11: openstack-neutron-bgpvpn-api-extension-tests
370 The tests are executed in the following order:
374 3) SDN controller suites
375 4) Feature project tests cases (Promise, Doctor, BGPVPN...)
379 As explained before, at the end of an automated execution, the OpenStack resources
380 might be eventually removed.