1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 Executing the functest suites
5 =============================
10 This section assumes the following:
11 * The Functest Docker container is running
12 * The docker prompt is shown
13 * The Functest environment is ready (Functest CLI command 'functest env prepare'
16 If any of the above steps are missing please refer to the Functest Config Guide
17 as they are a prerequisite and all the commands explained in this section **must** be
18 performed **inside the container**.
20 The Functest CLI offers two commands (functest tier ...) and (functest testcase ... )
21 for the execution of Test Tiers or Test Cases::
23 root@22e436918db0:~/repos/functest/ci# functest tier --help
24 Usage: functest tier [OPTIONS] COMMAND [ARGS]...
27 -h, --help Show this message and exit.
30 get-tests Prints the tests in a tier.
31 list Lists the available tiers.
32 run Executes all the tests within a tier.
33 show Shows information about a tier.
34 root@22e436918db0:~/repos/functest/ci# functest testcase --help
36 Usage: functest testcase [OPTIONS] COMMAND [ARGS]...
39 -h, --help Show this message and exit.
42 list Lists the available testcases.
43 run Executes a test case.
44 show Shows information about a test case.
46 More details on the existing Tiers and Test Cases can be seen with the 'list'
49 root@22e436918db0:~/repos/functest/ci# functest tier list
51 ['connection_check', 'api_check', 'snaps_health_check',]
53 ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'odl', 'rally_sanity', 'refstack_defcore', 'snaps_smoke']
55 ['doctor', 'domino', 'promise', security_scan']
57 ['tempest_full_parallel', 'rally_full']
59 ['cloudify_ims', 'orchestra_ims', 'vyos_vrouter']
63 root@22e436918db0:~/repos/functest/ci# functest testcase list
78 Note the list of test cases depend on the installer and the scenario.
80 More specific details on specific Tiers or Test Cases can be seen wih the
83 root@22e436918db0:~/repos/functest/ci# functest tier show smoke
84 +======================================================================+
86 +======================================================================+
88 | CI Loop: (daily)|(weekly) |
90 | Set of basic Functional tests to validate the OpenStack |
95 | - tempest_smoke_serial |
98 +----------------------------------------------------------------------+
102 root@22e436918db0:~/repos/functest/ci# functest testcase show tempest_smoke_serial
103 +======================================================================+
104 | Testcase: tempest_smoke_serial |
105 +======================================================================+
107 | This test case runs the smoke subset of the OpenStack Tempest |
108 | suite. The list of test cases is generated by Tempest |
109 | automatically and depends on the parameters of the OpenStack |
115 +----------------------------------------------------------------------+
118 To execute a Test Tier or Test Case, the 'run' command is used::
120 root@22e436918db0:~/repos/functest/ci# functest tier run healthcheck
121 2017-03-21 13:34:21,400 - run_tests - INFO - ############################################
122 2017-03-21 13:34:21,400 - run_tests - INFO - Running tier 'healthcheck'
123 2017-03-21 13:34:21,400 - run_tests - INFO - ############################################
124 2017-03-21 13:34:21,401 - run_tests - INFO -
126 2017-03-21 13:34:21,401 - run_tests - INFO - ============================================
127 2017-03-21 13:34:21,401 - run_tests - INFO - Running test case 'connection_check'...
128 2017-03-21 13:34:21,401 - run_tests - INFO - ============================================
129 test_glance_connect_fail (snaps.openstack.utils.tests.glance_utils_tests.GlanceSmokeTests) ... ok
130 test_glance_connect_success (snaps.openstack.utils.tests.glance_utils_tests.GlanceSmokeTests) ... ok
131 test_keystone_connect_fail (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneSmokeTests) ... ok
132 test_keystone_connect_success (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneSmokeTests) ... ok
133 test_neutron_connect_fail (snaps.openstack.utils.tests.neutron_utils_tests.NeutronSmokeTests) ... ok
134 test_neutron_connect_success (snaps.openstack.utils.tests.neutron_utils_tests.NeutronSmokeTests) ... ok
135 test_retrieve_ext_network_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronSmokeTests) ... ok
136 test_nova_connect_fail (snaps.openstack.utils.tests.nova_utils_tests.NovaSmokeTests) ... ok
137 test_nova_connect_success (snaps.openstack.utils.tests.nova_utils_tests.NovaSmokeTests) ... ok
139 ----------------------------------------------------------------------
140 Ran 9 tests in 3.768s
143 2017-03-21 13:34:26,570 - functest.core.testcase_base - INFO - connection_check OK
144 2017-03-21 13:34:26,918 - functest.core.testcase_base - INFO - The results were successfully pushed to DB
145 2017-03-21 13:34:26,918 - run_tests - INFO - Test execution time: 00:05
146 2017-03-21 13:34:26,918 - run_tests - INFO -
148 2017-03-21 13:34:26,918 - run_tests - INFO - ============================================
149 2017-03-21 13:34:26,918 - run_tests - INFO - Running test case 'api_check'...
150 2017-03-21 13:34:26,919 - run_tests - INFO - ============================================
151 test_create_project_minimal (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneUtilsTests) ... ok
152 test_create_user_minimal (snaps.openstack.utils.tests.keystone_utils_tests.KeystoneUtilsTests) ... ok
153 test_create_delete_user (snaps.openstack.tests.create_user_tests.CreateUserSuccessTests) ... ok
154 test_create_user (snaps.openstack.tests.create_user_tests.CreateUserSuccessTests) ... ok
155 test_create_user_2x (snaps.openstack.tests.create_user_tests.CreateUserSuccessTests) ...
156 2017-03-21 13:34:32,684 - create_user - INFO - Found user with name - CreateUserSuccessTests-7e741e11-c9fd-489-name ok
157 test_create_delete_project (snaps.openstack.tests.create_project_tests.CreateProjectSuccessTests) ... ok
158 test_create_project (snaps.openstack.tests.create_project_tests.CreateProjectSuccessTests) ... ok
159 test_create_project_2x (snaps.openstack.tests.create_project_tests.CreateProjectSuccessTests) ...
160 2017-03-21 13:34:35,922 - create_image - INFO - Found project with name - CreateProjectSuccessTests-b38e08ce-2862-48a-name ok
161 test_create_project_sec_grp_one_user (snaps.openstack.tests.create_project_tests.CreateProjectUserTests) ...
162 2017-03-21 13:34:37,907 - OpenStackSecurityGroup - INFO - Creating security group CreateProjectUserTests-ab8801f6-dad8-4f9-name...
163 2017-03-21 13:34:37,907 - neutron_utils - INFO - Retrieving security group with name - CreateProjectUserTests-ab8801f6-dad8-4f9-name
164 2017-03-21 13:34:38,376 - neutron_utils - INFO - Creating security group with name - CreateProjectUserTests-ab8801f6-dad8-4f9-name
165 2017-03-21 13:34:38,716 - neutron_utils - INFO - Retrieving security group rules associate with the security group - CreateProjectUserTests-ab8801f6-dad8-4f9-name
166 2017-03-21 13:34:38,762 - neutron_utils - INFO - Retrieving security group with ID - 821419cb-c54c-41b4-a61b-fb30e5dd2ec5
167 2017-03-21 13:34:38,886 - neutron_utils - INFO - Retrieving security group with ID - 821419cb-c54c-41b4-a61b-fb30e5dd2ec5
168 2017-03-21 13:34:39,000 - neutron_utils - INFO - Retrieving security group with name - CreateProjectUserTests-ab8801f6-dad8-4f9-name
169 2017-03-21 13:34:39,307 - neutron_utils - INFO - Deleting security group rule with ID - d85fafc0-9649-45c9-a00e-452f3d5c09a6
170 2017-03-21 13:34:39,531 - neutron_utils - INFO - Deleting security group rule with ID - 69d79c09-bc3b-4975-9353-5f43aca51237
171 2017-03-21 13:34:39,762 - neutron_utils - INFO - Deleting security group with name - CreateProjectUserTests-ab8801f6-dad8-4f9-name ok
172 test_create_project_sec_grp_two_users (snaps.openstack.tests.create_project_tests.CreateProjectUserTests) ...
173 2017-03-21 13:34:43,511 - OpenStackSecurityGroup - INFO - Creating security group CreateProjectUserTests-4d9261a6-e008-44b-name...
174 2017-03-21 13:34:43,511 - neutron_utils - INFO - Retrieving security group with name - CreateProjectUserTests-4d9261a6-e008-44b-name
175 2017-03-21 13:34:44,090 - neutron_utils - INFO - Creating security group with name - CreateProjectUserTests-4d9261a6-e008-44b-name
176 2017-03-21 13:34:44,784 - neutron_utils - INFO - Retrieving security group rules associate with the security group - CreateProjectUserTests-4d9261a6-e008-44b-name
177 2017-03-21 13:34:44,864 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f2e-a14d-b01abf74c832
178 2017-03-21 13:34:45,233 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f2e-a14d-b01abf74c832
179 2017-03-21 13:34:45,332 - neutron_utils - INFO - Retrieving security group with name - CreateProjectUserTests-4d9261a6-e008-44b-name
180 2017-03-21 13:34:45,779 - OpenStackSecurityGroup - INFO - Creating security group CreateProjectUserTests-4d9261a6-e008-44b-name...
181 2017-03-21 13:34:45,779 - neutron_utils - INFO - Retrieving security group with name - CreateProjectUserTests-4d9261a6-e008-44b-name
182 2017-03-21 13:34:46,112 - neutron_utils - INFO - Retrieving security group rules associate with the security group - CreateProjectUserTests-4d9261a6-e008-44b-name
183 2017-03-21 13:34:46,184 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f2e-a14d-b01abf74c832
184 2017-03-21 13:34:46,296 - neutron_utils - INFO - Retrieving security group with ID - 780193e4-9bd2-4f2e-a14d-b01abf74c832
185 2017-03-21 13:34:46,387 - neutron_utils - INFO - Deleting security group rule with ID - 2320a573-ec56-47c5-a1ba-ec514d30114b
186 2017-03-21 13:34:46,636 - neutron_utils - INFO - Deleting security group rule with ID - 6186282b-db37-4e47-becc-a3886079c069
187 2017-03-21 13:34:46,780 - neutron_utils - INFO - Deleting security group with name - CreateProjectUserTests-4d9261a6-e008-44b-name
188 2017-03-21 13:34:47,006 - neutron_utils - INFO - Deleting security group rule with ID - 2320a573-ec56-47c5-a1ba-ec514d30114b
189 2017-03-21 13:34:47,072 - OpenStackSecurityGroup - WARNING - Rule not found, cannot delete - Security group rule 2320a573-ec56-47c5-a1ba-ec514d30114b does not exist
190 Neutron server returns request_ids: ['req-d74eb2e2-b26f-4236-87dc-7255866141d9']
191 2017-03-21 13:34:47,072 - neutron_utils - INFO - Deleting security group rule with ID - 6186282b-db37-4e47-becc-a3886079c069
192 2017-03-21 13:34:47,118 - OpenStackSecurityGroup - WARNING - Rule not found, cannot delete - Security group rule 6186282b-db37-4e47-becc-a3886079c069 does not exist
193 Neutron server returns request_ids: ['req-8c0a5a24-be90-4844-a9ed-2a85cc6f59a5']
194 2017-03-21 13:34:47,118 - neutron_utils - INFO - Deleting security group with name - CreateProjectUserTests-4d9261a6-e008-44b-name
195 2017-03-21 13:34:47,172 - OpenStackSecurityGroup - WARNING - Security Group not found, cannot delete - Security group 780193e4-9bd2-4f2e-a14d-b01abf74c832 does not exist
196 Neutron server returns request_ids: ['req-c6e1a6b5-43e0-4d46-bb68-c2e1672d4d21'] ok
197 test_create_image_minimal_file (snaps.openstack.utils.tests.glance_utils_tests.GlanceUtilsTests) ... ok
198 test_create_image_minimal_url (snaps.openstack.utils.tests.glance_utils_tests.GlanceUtilsTests) ... ok
199 test_create_network (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsNetworkTests) ...
200 2017-03-21 13:35:22,275 - neutron_utils - INFO - Creating network with name NeutronUtilsNetworkTests-c06c20e0-d78f-4fa4-8401-099a7a6cab2e-pub-net
201 2017-03-21 13:35:23,965 - neutron_utils - INFO - Deleting network with name NeutronUtilsNetworkTests-c06c20e0-d78f-4fa4-8401-099a7a6cab2e-pub-net ok
202 test_create_network_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsNetworkTests) ... ok
203 test_create_network_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsNetworkTests) ... ok
204 test_create_subnet (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests) ...
205 2017-03-21 13:35:25,495 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-4f440a5f-54e3-4455-ab9b-39dfe06f6d21-pub-net
206 2017-03-21 13:35:26,841 - neutron_utils - INFO - Creating subnet with name NeutronUtilsSubnetTests-4f440a5f-54e3-4455-ab9b-39dfe06f6d21-pub-subnet
207 2017-03-21 13:35:28,311 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsSubnetTests-4f440a5f-54e3-4455-ab9b-39dfe06f6d21-pub-subnet
208 2017-03-21 13:35:29,585 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-4f440a5f-54e3-4455-ab9b-39dfe06f6d21-pub-net ok
209 test_create_subnet_empty_cidr (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests) ...
210 2017-03-21 13:35:31,013 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-41fc0db4-71ee-47e6-bec9-316273e5bcc0-pub-net
211 2017-03-21 13:35:31,652 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-41fc0db4-71ee-47e6-bec9-316273e5bcc0-pub-net ok
212 test_create_subnet_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests) ...
213 2017-03-21 13:35:32,379 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-1030e0cb-1714-4d18-8619-a03bac0d0257-pub-net
214 2017-03-21 13:35:33,516 - neutron_utils - INFO - Creating subnet with name NeutronUtilsSubnetTests-1030e0cb-1714-4d18-8619-a03bac0d0257-pub-subnet
215 2017-03-21 13:35:34,160 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-1030e0cb-1714-4d18-8619-a03bac0d0257-pub-net ok
216 test_create_subnet_null_cidr (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests) ...
217 2017-03-21 13:35:35,784 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-1d7522fd-3fb5-4b1c-8741-97d7c47a5f7d-pub-net
218 2017-03-21 13:35:36,367 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-1d7522fd-3fb5-4b1c-8741-97d7c47a5f7d-pub-net ok
219 test_create_subnet_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSubnetTests) ...
220 2017-03-21 13:35:37,055 - neutron_utils - INFO - Creating network with name NeutronUtilsSubnetTests-0a8ac1b2-e5d4-4522-a079-7e17945e482e-pub-net
221 2017-03-21 13:35:37,691 - neutron_utils - INFO - Deleting network with name NeutronUtilsSubnetTests-0a8ac1b2-e5d4-4522-a079-7e17945e482e-pub-net ok
222 test_add_interface_router (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
223 2017-03-21 13:35:38,994 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-net
224 2017-03-21 13:35:40,311 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-subnet
225 2017-03-21 13:35:41,713 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-router
226 2017-03-21 13:35:44,131 - neutron_utils - INFO - Adding interface to router with name NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-router
227 2017-03-21 13:35:45,725 - neutron_utils - INFO - Removing router interface from router named NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-router
228 2017-03-21 13:35:47,464 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-router
229 2017-03-21 13:35:48,670 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-subnet
230 2017-03-21 13:35:50,921 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-433818c9-4472-49a8-9241-791ad0a71d3f-pub-net ok
231 test_add_interface_router_null_router (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
232 2017-03-21 13:35:52,230 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-1fc2de16-2d3e-497b-b947-022b1bf9d90c-pub-net
233 2017-03-21 13:35:53,662 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-1fc2de16-2d3e-497b-b947-022b1bf9d90c-pub-subnet
234 2017-03-21 13:35:55,203 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-1fc2de16-2d3e-497b-b947-022b1bf9d90c-pub-subnet
235 2017-03-21 13:35:55,694 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-1fc2de16-2d3e-497b-b947-022b1bf9d90c-pub-net ok
236 test_add_interface_router_null_subnet (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
237 2017-03-21 13:35:57,392 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-2e4fb9f3-312b-4954-8015-435464fdc8b0-pub-net
238 2017-03-21 13:35:58,215 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests-2e4fb9f3-312b-4954-8015-435464fdc8b0-pub-router
239 2017-03-21 13:36:00,369 - neutron_utils - INFO - Adding interface to router with name NeutronUtilsRouterTests-2e4fb9f3-312b-4954-8015-435464fdc8b0-pub-router
240 2017-03-21 13:36:00,369 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTests-2e4fb9f3-312b-4954-8015-435464fdc8b0-pub-router
241 2017-03-21 13:36:02,742 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-2e4fb9f3-312b-4954-8015-435464fdc8b0-pub-net ok
242 test_create_port (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
243 2017-03-21 13:36:05,010 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-dde05ce1-a2f8-4c5e-a028-e1ca0e11a05b-pub-net
244 2017-03-21 13:36:05,996 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-dde05ce1-a2f8-4c5e-a028-e1ca0e11a05b-pub-subnet
245 2017-03-21 13:36:09,103 - neutron_utils - INFO - Creating port for network with name - NeutronUtilsRouterTests-dde05ce1-a2f8-4c5e-a028-e1ca0e11a05b-pub-net
246 2017-03-21 13:36:10,312 - neutron_utils - INFO - Deleting port with name NeutronUtilsRouterTests-dde05ce1-a2f8-4c5e-a028-e1ca0e11a05b-port
247 2017-03-21 13:36:11,045 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-dde05ce1-a2f8-4c5e-a028-e1ca0e11a05b-pub-subnet
248 2017-03-21 13:36:14,265 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-dde05ce1-a2f8-4c5e-a028-e1ca0e11a05b-pub-net ok
249 test_create_port_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
250 2017-03-21 13:36:16,250 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-b986a259-e873-431c-bde4-b2771ace4549-pub-net
251 2017-03-21 13:36:16,950 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-b986a259-e873-431c-bde4-b2771ace4549-pub-subnet
252 2017-03-21 13:36:17,798 - neutron_utils - INFO - Creating port for network with name - NeutronUtilsRouterTests-b986a259-e873-431c-bde4-b2771ace4549-pub-net
253 2017-03-21 13:36:18,544 - neutron_utils - INFO - Deleting port with name NeutronUtilsRouterTests-b986a259-e873-431c-bde4-b2771ace4549-port
254 2017-03-21 13:36:19,582 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-b986a259-e873-431c-bde4-b2771ace4549-pub-subnet
255 2017-03-21 13:36:21,606 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-b986a259-e873-431c-bde4-b2771ace4549-pub-net ok
256 test_create_port_invalid_ip (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
257 2017-03-21 13:36:23,779 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-7ab3a329-9dd8-4e6f-9d52-aafb47ea5122-pub-net
258 2017-03-21 13:36:25,201 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-7ab3a329-9dd8-4e6f-9d52-aafb47ea5122-pub-subnet
259 2017-03-21 13:36:25,599 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-7ab3a329-9dd8-4e6f-9d52-aafb47ea5122-pub-subnet
260 2017-03-21 13:36:26,220 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-7ab3a329-9dd8-4e6f-9d52-aafb47ea5122-pub-net ok
261 test_create_port_invalid_ip_to_subnet (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
262 2017-03-21 13:36:27,112 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-c016821d-cd4f-4e0f-8f8c-d5cef3392e64-pub-net
263 2017-03-21 13:36:28,720 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-c016821d-cd4f-4e0f-8f8c-d5cef3392e64-pub-subnet
264 2017-03-21 13:36:29,457 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-c016821d-cd4f-4e0f-8f8c-d5cef3392e64-pub-subnet
265 2017-03-21 13:36:29,909 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-c016821d-cd4f-4e0f-8f8c-d5cef3392e64-pub-net ok
266 test_create_port_null_ip (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
267 2017-03-21 13:36:31,037 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-9a86227f-6041-4b04-86a7-1701fb86baa3-pub-net
268 2017-03-21 13:36:31,695 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-9a86227f-6041-4b04-86a7-1701fb86baa3-pub-subnet
269 2017-03-21 13:36:32,305 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-9a86227f-6041-4b04-86a7-1701fb86baa3-pub-subnet
270 2017-03-21 13:36:33,553 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-9a86227f-6041-4b04-86a7-1701fb86baa3-pub-net ok
271 test_create_port_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
272 2017-03-21 13:36:34,593 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-42efa897-4f65-4d9b-b19d-fbc61f97c966-pub-net
273 2017-03-21 13:36:35,217 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-42efa897-4f65-4d9b-b19d-fbc61f97c966-pub-subnet
274 2017-03-21 13:36:36,648 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-42efa897-4f65-4d9b-b19d-fbc61f97c966-pub-subnet
275 2017-03-21 13:36:37,251 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-42efa897-4f65-4d9b-b19d-fbc61f97c966-pub-net ok
276 test_create_port_null_network_object (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
277 2017-03-21 13:36:37,885 - neutron_utils - INFO - Creating network with name NeutronUtilsRouterTests-617f4110-45c1-4900-bad1-a6204f34dd64-pub-net
278 2017-03-21 13:36:38,468 - neutron_utils - INFO - Creating subnet with name NeutronUtilsRouterTests-617f4110-45c1-4900-bad1-a6204f34dd64-pub-subnet
279 2017-03-21 13:36:40,005 - neutron_utils - INFO - Deleting subnet with name NeutronUtilsRouterTests-617f4110-45c1-4900-bad1-a6204f34dd64-pub-subnet
280 2017-03-21 13:36:41,637 - neutron_utils - INFO - Deleting network with name NeutronUtilsRouterTests-617f4110-45c1-4900-bad1-a6204f34dd64-pub-net ok
281 test_create_router_empty_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ... ok
282 test_create_router_null_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ... ok
283 test_create_router_simple (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
284 2017-03-21 13:36:43,424 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests-b6a2dafc-38d4-4c46-bb41-2ba9e1c0084e-pub-router
285 2017-03-21 13:36:45,013 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTests-b6a2dafc-38d4-4c46-bb41-2ba9e1c0084e-pub-router ok
286 test_create_router_with_public_interface (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsRouterTests) ...
287 2017-03-21 13:36:47,829 - neutron_utils - INFO - Creating router with name - NeutronUtilsRouterTests-d268dda2-7a30-4d3d-a008-e5aa4592637d-pub-router
288 2017-03-21 13:36:49,448 - neutron_utils - INFO - Deleting router with name - NeutronUtilsRouterTests-d268dda2-7a30-4d3d-a008-e5aa4592637d-pub-router ok
289 test_create_delete_simple_sec_grp (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecurityGroupTests) ...
290 2017-03-21 13:36:51,067 - neutron_utils - INFO - Creating security group with name - NeutronUtilsSecurityGroupTests-1543e861-ea38-4fbe-9723-c27552e3eb7aname
291 2017-03-21 13:36:51,493 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSecurityGroupTests-1543e861-ea38-4fbe-9723-c27552e3eb7aname
292 2017-03-21 13:36:51,568 - neutron_utils - INFO - Deleting security group with name - NeutronUtilsSecurityGroupTests-1543e861-ea38-4fbe-9723-c27552e3eb7aname
293 2017-03-21 13:36:51,772 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSecurityGroupTests-1543e861-ea38-4fbe-9723-c27552e3eb7aname ok
294 test_create_sec_grp_no_name (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecurityGroupTests) ... ok
295 test_create_sec_grp_no_rules (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecurityGroupTests) ...
296 2017-03-21 13:36:52,253 - neutron_utils - INFO - Creating security group with name - NeutronUtilsSecurityGroupTests-57c60864-f46c-4391-ba99-6acc4dd123ddname
297 2017-03-21 13:36:52,634 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSecurityGroupTests-57c60864-f46c-4391-ba99-6acc4dd123ddname
298 2017-03-21 13:36:52,718 - neutron_utils - INFO - Deleting security group with name - NeutronUtilsSecurityGroupTests-57c60864-f46c-4391-ba99-6acc4dd123ddname ok
299 test_create_sec_grp_one_rule (snaps.openstack.utils.tests.neutron_utils_tests.NeutronUtilsSecurityGroupTests) ...
300 2017-03-21 13:36:53,082 - neutron_utils - INFO - Creating security group with name - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name
301 2017-03-21 13:36:53,483 - neutron_utils - INFO - Retrieving security group rules associate with the security group - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name
302 2017-03-21 13:36:53,548 - neutron_utils - INFO - Creating security group to security group - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name
303 2017-03-21 13:36:53,548 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name
304 2017-03-21 13:36:53,871 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name
305 2017-03-21 13:36:53,944 - neutron_utils - INFO - Retrieving security group rules associate with the security group - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name
306 2017-03-21 13:36:53,991 - neutron_utils - INFO - Retrieving security group with name - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name
307 2017-03-21 13:36:54,069 - neutron_utils - INFO - Deleting security group rule with ID - 7f76046c-d043-46e0-9d12-4b983525810b
308 2017-03-21 13:36:54,185 - neutron_utils - INFO - Deleting security group rule with ID - f18a9ed1-466f-4373-a6b2-82bd317bc838
309 2017-03-21 13:36:54,338 - neutron_utils - INFO - Deleting security group rule with ID - fe34a3d0-948e-47c1-abad-c3ec8d33b2fb
310 2017-03-21 13:36:54,444 - neutron_utils - INFO - Deleting security group with name - NeutronUtilsSecurityGroupTests-a3ac62bb-a7e8-4fc2-ba4c-e656f1f3c9a1name ok
311 test_create_delete_keypair (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
312 2017-03-21 13:36:54,637 - nova_utils - INFO - Creating keypair with name - NovaUtilsKeypairTests-5ce69b6f-d8d0-4b66-bd25-30a22cf3bda0 ok
313 test_create_key_from_file (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
314 2017-03-21 13:36:58,989 - nova_utils - INFO - Saved public key to - tmp/NovaUtilsKeypairTests-df3e848d-a467-4cc4-99d5-022eb67eee94.pub
315 2017-03-21 13:36:58,990 - nova_utils - INFO - Saved private key to - tmp/NovaUtilsKeypairTests-df3e848d-a467-4cc4-99d5-022eb67eee94
316 2017-03-21 13:36:58,990 - nova_utils - INFO - Saving keypair to - tmp/NovaUtilsKeypairTests-df3e848d-a467-4cc4-99d5-022eb67eee94.pub
317 2017-03-21 13:36:58,990 - nova_utils - INFO - Creating keypair with name - NovaUtilsKeypairTests-df3e848d-a467-4cc4-99d5-022eb67eee94 ok
318 test_create_keypair (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
319 2017-03-21 13:36:59,807 - nova_utils - INFO - Creating keypair with name - NovaUtilsKeypairTests-fc7f7ffd-80f6-43df-bd41-a3c014ba8c3d ok
320 test_floating_ips (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsKeypairTests) ...
321 2017-03-21 13:37:02,765 - nova_utils - INFO - Creating floating ip to external network - admin_floating_net ok
322 test_create_delete_flavor (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsFlavorTests) ... ok
323 test_create_flavor (snaps.openstack.utils.tests.nova_utils_tests.NovaUtilsFlavorTests) ... ok
324 test_create_clean_flavor (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ... ok
325 test_create_delete_flavor (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ... ok
326 test_create_flavor (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ... ok
327 test_create_flavor_existing (snaps.openstack.tests.create_flavor_tests.CreateFlavorTests) ...
328 2017-03-21 13:37:18,545 - create_image - INFO - Found flavor with name - CreateFlavorTests-3befc152-4319-4f9c-82d4-75f8941d9533name ok
330 ----------------------------------------------------------------------
331 Ran 48 tests in 171.000s
334 2017-03-21 13:37:18,620 - functest.core.testcase_base - INFO - api_check OK
335 2017-03-21 13:37:18,977 - functest.core.testcase_base - INFO - The results were successfully pushed to DB
336 2017-03-21 13:37:18,977 - run_tests - INFO - Test execution time: 02:52
337 2017-03-21 13:37:18,981 - run_tests - INFO -
339 2017-03-21 13:37:18,981 - run_tests - INFO - ============================================
340 2017-03-21 13:37:18,981 - run_tests - INFO - Running test case 'snaps_health_check'...
341 2017-03-21 13:37:18,981 - run_tests - INFO - ============================================
342 2017-03-21 13:37:19,098 - file_utils - INFO - Attempting to read OS environment file - /home/opnfv/functest/conf/openstack.creds
343 2017-03-21 13:37:19,099 - openstack_tests - INFO - OS Credentials = OSCreds - username=admin, password=admin, auth_url=http://192.168.10.7:5000/v3, project_name=admin, identity_api_version=3, image_api_version=1, network_api_version=2, compute_api_version=2, user_domain_id=default, proxy_settings=None
344 2017-03-21 13:37:19,434 - file_utils - INFO - Attempting to read OS environment file - /home/opnfv/functest/conf/openstack.creds
345 2017-03-21 13:37:19,435 - openstack_tests - INFO - OS Credentials = OSCreds - username=admin, password=admin, auth_url=http://192.168.10.7:5000/v3, project_name=admin, identity_api_version=3, image_api_version=1, network_api_version=2, compute_api_version=2, user_domain_id=default, proxy_settings=None
346 test_check_vm_ip_dhcp (snaps.openstack.tests.create_instance_tests.SimpleHealthCheck) ...
347 2017-03-21 13:37:26,082 - create_image - INFO - Creating image
348 2017-03-21 13:37:28,793 - create_image - INFO - Image is active with name - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-image
349 2017-03-21 13:37:28,793 - create_image - INFO - Image is now active with name - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-image
350 2017-03-21 13:37:28,794 - OpenStackNetwork - INFO - Creating neutron network SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-priv-net...
351 2017-03-21 13:37:29,308 - neutron_utils - INFO - Creating network with name SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-priv-net
352 2017-03-21 13:37:30,771 - neutron_utils - INFO - Creating subnet with name SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-priv-subnet
353 2017-03-21 13:37:36,974 - neutron_utils - INFO - Creating port for network with name - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-priv-net
354 2017-03-21 13:37:38,188 - create_instance - INFO - Creating VM with name - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-inst
355 2017-03-21 13:37:41,538 - create_instance - INFO - Created instance with name - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-inst
356 2017-03-21 13:37:59,577 - create_instance - INFO - VM is - ACTIVE
357 2017-03-21 13:37:59,577 - create_instance_tests - INFO - Looking for expression Lease of.*obtained in the console log
358 2017-03-21 13:37:59,830 - create_instance_tests - INFO - DHCP lease obtained logged in console
359 2017-03-21 13:37:59,830 - create_instance_tests - INFO - With correct IP address
360 2017-03-21 13:37:59,830 - create_instance - INFO - Deleting Port - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5dport-1
361 2017-03-21 13:37:59,830 - neutron_utils - INFO - Deleting port with name SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5dport-1
362 2017-03-21 13:38:00,705 - create_instance - INFO - Deleting VM instance - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-inst
363 2017-03-21 13:38:01,412 - create_instance - INFO - Checking deletion status
364 2017-03-21 13:38:04,938 - create_instance - INFO - VM has been properly deleted VM with name - SimpleHealthCheck-23244728-5a5a-4545-9b16-50257a595e5d-inst
367 ----------------------------------------------------------------------
368 Ran 1 test in 46.982s
371 2017-03-21 13:38:06,417 - functest.core.testcase_base - INFO - snaps_health_check OK
372 2017-03-21 13:38:06,778 - functest.core.testcase_base - INFO - The results were successfully pushed to DB
373 2017-03-21 13:38:06,779 - run_tests - INFO - Test execution time: 00:47
374 2017-03-21 13:38:06,779 - run_tests - INFO -
377 root@22e436918db0:~/repos/functest/ci# functest testcase run vping_ssh
378 Executing command: 'python /home/opnfv/repos/functest/ci/run_tests.py -t vping_ssh'
379 2016-06-30 11:50:31,861 - run_tests - INFO - Sourcing the OpenStack RC file...
380 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
381 2016-06-30 11:50:31,865 - run_tests - INFO - Running test case 'vping_ssh'...
382 2016-06-30 11:50:31,865 - run_tests - INFO - ============================================
383 2016-06-30 11:50:32,977 - vping_ssh - INFO - Creating image 'functest-vping' from '/home/opnfv/functest/data/cirros-0.3.5-x86_64-disk.img'...
384 2016-06-30 11:50:45,470 - vping_ssh - INFO - Creating neutron network vping-net...
385 2016-06-30 11:50:47,645 - vping_ssh - INFO - Creating security group 'vPing-sg'...
386 2016-06-30 11:50:48,843 - vping_ssh - INFO - Using existing Flavor 'm1.small'...
387 2016-06-30 11:50:48,927 - vping_ssh - INFO - vPing Start Time:'2016-06-30 11:50:48'
388 2016-06-30 11:50:48,927 - vping_ssh - INFO - Creating instance 'opnfv-vping-1'...
389 2016-06-30 11:51:34,664 - vping_ssh - INFO - Instance 'opnfv-vping-1' is ACTIVE.
390 2016-06-30 11:51:34,818 - vping_ssh - INFO - Adding 'opnfv-vping-1' to security group 'vPing-sg'...
391 2016-06-30 11:51:35,209 - vping_ssh - INFO - Creating instance 'opnfv-vping-2'...
392 2016-06-30 11:52:01,439 - vping_ssh - INFO - Instance 'opnfv-vping-2' is ACTIVE.
393 2016-06-30 11:52:01,439 - vping_ssh - INFO - Adding 'opnfv-vping-2' to security group 'vPing-sg'...
394 2016-06-30 11:52:01,754 - vping_ssh - INFO - Creating floating IP for VM 'opnfv-vping-2'...
395 2016-06-30 11:52:01,969 - vping_ssh - INFO - Floating IP created: '10.17.94.140'
396 2016-06-30 11:52:01,969 - vping_ssh - INFO - Associating floating ip: '10.17.94.140' to VM 'opnfv-vping-2'
397 2016-06-30 11:52:02,792 - vping_ssh - INFO - Trying to establish SSH connection to 10.17.94.140...
398 2016-06-30 11:52:19,915 - vping_ssh - INFO - Waiting for ping...
399 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing detected!
400 2016-06-30 11:52:21,108 - vping_ssh - INFO - vPing duration:'92.2' s.
401 2016-06-30 11:52:21,109 - vping_ssh - INFO - vPing OK
402 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
403 2016-06-30 11:52:21,153 - clean_openstack - INFO - Cleaning OpenStack resources...
404 2016-06-30 11:52:21,153 - clean_openstack - INFO - +++++++++++++++++++++++++++++++
405 Version 1 is deprecated, use alternative version 2 instead.
410 To list the test cases which are part of a specific Test Tier, the 'get-tests'
411 command is used with 'functest tier'::
413 root@22e436918db0:~/repos/functest/ci# functest tier get-tests healthcheck
414 Test cases in tier 'healthcheck':
415 ['connection_check', 'api_check', 'snaps_health_check']
418 Please note that for some scenarios some test cases might not be launched.
419 For example, the last example displayed only the 'odl' testcase for the given
420 environment. In this particular system the deployment does not support the 'ocl' SDN
421 Controller Test Case; for example.
423 **Important** If you use the command 'functest tier run <tier_name>', then the
424 Functest CLI utility will call **all valid Test Cases**, which belong to the
425 specified Test Tier, as relevant to scenarios deployed to the SUT environment.
426 Thus, the Functest CLI utility calculates automatically which tests can be
427 executed and which cannot, given the environment variable **DEPLOY_SCENARIO**,
428 which is passed in to the Functest docker container.
430 Currently, the Functest CLI command 'functest testcase run <testcase_name>', supports
433 * Run a single Test Case, specified by a valid choice of <testcase_name>
434 * Run ALL test Test Cases (for all Tiers) by specifying <testcase_name> = 'all'
436 Functest includes a cleaning mechanism in order to remove all the OpenStack
437 resources except those present before running any test. The script
438 *$REPOS_DIR/functest/functest/utils/openstack_snapshot.py* is called once when setting up
439 the Functest environment (i.e. CLI command 'functest env prepare') to snapshot
440 all the OpenStack resources (images, networks, volumes, security groups, tenants,
441 users) so that an eventual cleanup does not remove any of these defaults.
443 It is also called before running a test except if it is disabled by configuration
444 in the testcases.yaml file (clean_flag=false). This flag has been added as some
445 upstream tests already include their own cleaning mechanism (e.g. Rally).
447 The script **openstack_clean.py** which is located in
448 *$REPOS_DIR/functest/functest/utils/* is normally called after a test execution. It is
449 in charge of cleaning the OpenStack resources that are not specified in the
450 defaults file generated previously which is stored in
451 */home/opnfv/functest/conf/openstack_snapshot.yaml* in the Functest docker container.
453 It is important to mention that if there are new OpenStack resources created
454 manually after the snapshot done before running the tests, they will be removed,
455 unless you use the special method of invoking the test case with specific
456 suppression of clean up. (See the `Troubleshooting`_ section).
458 The reason to include this cleanup meachanism in Functest is because some
459 test suites create a lot of resources (users, tenants, networks, volumes etc.)
460 that are not always properly cleaned, so this function has been set to keep the
461 system as clean as it was before a full Functest execution.
463 Although the Functest CLI provides an easy way to run any test, it is possible to
464 do a direct call to the desired test script. For example:
466 python $REPOS_DIR/functest/functest/opnfv_tests/openstack/vping/vping_ssh.py
472 As mentioned previously, the Functest Docker container preparation as well as
473 invocation of Test Cases can be called within the container from the Jenkins CI
474 system. There are 3 jobs that automate the whole process. The first job runs all
475 the tests referenced in the daily loop (i.e. that must been run daily), the second
476 job runs the tests referenced in the weekly loop (usually long duration tests run
477 once a week maximum) and the third job allows testing test suite by test suite specifying
478 the test suite name. The user may also use either of these Jenkins jobs to execute
479 the desired test suites.
481 One of the most challenging task in the Danube release consists
482 in dealing with lots of scenarios and installers. Thus, when the tests are
483 automatically started from CI, a basic algorithm has been created in order to
484 detect whether a given test is runnable or not on the given scenario.
485 Some Functest test suites cannot be systematically run (e.g. ODL suite can not
486 be run on an ONOS scenario). The daily/weekly notion has been introduces in
487 Colorado in order to save CI time and avoid running systematically
488 long duration tests. It was not used in Colorado due to CI resource shortage.
489 The mechanism remains however as part of the CI evolution.
491 CI provides some useful information passed to the container as environment
494 * Installer (apex|compass|fuel|joid), stored in INSTALLER_TYPE
495 * Installer IP of the engine or VM running the actual deployment, stored in INSTALLER_IP
496 * The scenario [controller]-[feature]-[mode], stored in DEPLOY_SCENARIO with
498 * controller = (odl|ocl|nosdn|onos)
499 * feature = (ovs(dpdk)|kvm|sfc|bgpvpn|multisites|netready|ovs_dpdk_bar)
502 The constraints per test case are defined in the Functest configuration file
503 */home/opnfv/repos/functest/functest/ci/testcases.yaml*::
509 ci_loop: '(daily)|(weekly)'
511 Set of basic Functional tests to validate the OpenStack deployment.
515 criteria: 'status == "PASS"'
518 This test case verifies: 1) SSH to an instance using floating
519 IPs over the public network. 2) Connectivity between 2 instances
520 over a private network.
523 scenario: '^((?!bgpvpn|odl_l3).)*$'
525 module: 'functest.opnfv_tests.openstack.vping.vping_ssh'
529 We may distinguish 2 levels in the test case description:
533 At the tier level, we define the following parameters:
535 * ci_loop: indicate if in automated mode, the test case must be run in dail and/or weekly jobs
536 * description: a high level view of the test case
538 For a given test case we defined:
539 * the name of the test case
540 * the criteria (experimental): a criteria used to declare the test case as PASS or FAIL
541 * blocking: if set to true, if the test is failed, the execution of the following tests is canceled
542 * clean_flag: shall the functect internal mechanism be invoked after the test
543 * the description of the test case
544 * the dependencies: a combination of 2 regex on the scenario and the installer name
545 * run: In Danube we introduced the notion of abstract class in order to harmonize the way to run internal, feature or vnf tests
547 For further details on abstraction classes, see developper guide.
549 Additional parameters have been added in the desription in the Database.
550 The target is to use the configuration stored in the Database and consider the
551 local file as backup if the Database is not reachable.
552 The additional fields related to a test case are:
553 * trust: we introduced this notion to put in place a mechanism of scenario promotion.
554 * Version: it indicates since which version you can run this test
555 * domains: the main domain covered by the test suite
556 * tags: a list of tags related to the test suite
558 The order of execution is the one defined in the file if all test cases are selected.
560 In CI daily job the tests are executed in the following order:
562 1) healthcheck (blocking)
563 2) smoke: both vPings are blocking
564 3) Feature project tests cases
566 In CI weekly job we add 2 tiers:
569 5) Components (Rally and Tempest long duration suites)
571 As explained before, at the end of an automated execution, the OpenStack resources
572 might be eventually removed.
573 Please note that a system snapshot is taken before any test case execution.
575 This testcase.yaml file is used for CI, for the CLI and for the automatic reporting.