70e51b93e16283b330626cfa2035e05f8d290530
[snaps.git] / docs / how-to-use / IntegrationTests.rst
1 SNAPS OpenStack Integration Testing
2 ===================================
3
4 These tests are ones designed to be run within their own dynamically created project along with a newly generated user
5 account and generally require other OpenStack object creators.
6
7 The Test Classes
8 ================
9
10 create_security_group_tests.py - CreateSecurityGroupTests
11 ---------------------------------------------------------
12
13 +---------------------------------------+---------------+-----------------------------------------------------------+
14 | Test Name                             | API Versions  | Description                                               |
15 +=======================================+===============+===========================================================+
16 | test_create_group_without_rules       | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can create a     |
17 |                                       | Neutron 2     | security group without any rules                          |
18 +---------------------------------------+---------------+-----------------------------------------------------------+
19 | test_create_group_admin_user_to_new   | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can be created   |
20 | _project                              | Neutron 2     | by the admin user and associated with a new project       |
21 +---------------------------------------+---------------+-----------------------------------------------------------+
22 | test_create_group_new_user_to_admin   | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can be created   |
23 | _project                              | Neutron 2     | by the new user and associated with the admin project     |
24 +---------------------------------------+---------------+-----------------------------------------------------------+
25 | test_create_delete_group              | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class clean() method   |
26 |                                       | Neutron 2     | will not raise an exception should the group be deleted by|
27 |                                       |               | some other process                                        |
28 +---------------------------------------+---------------+-----------------------------------------------------------+
29 | test_create_group_with_one_simple_rule| Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can create a     |
30 |                                       | Neutron 2     | security group with a single rule                         |
31 +---------------------------------------+---------------+-----------------------------------------------------------+
32 | test_create_group_with_several_rules  | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can create a     |
33 |                                       | Neutron 2     | security group with several rules                         |
34 +---------------------------------------+---------------+-----------------------------------------------------------+
35 | test_add_rule                         | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup#add_rule() method      |
36 |                                       | Neutron 2     | properly creates and associates the new rule              |
37 +---------------------------------------+---------------+-----------------------------------------------------------+
38 | test_remove_rule_by_id                | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup#remove_rule() method   |
39 |                                       | Neutron 2     | properly deletes and disassociates the old rule via its ID|
40 +---------------------------------------+---------------+-----------------------------------------------------------+
41 | test_remove_rule_by_setting           | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup#remove_rule() method   |
42 |                                       | Neutron 2     | properly deletes and disassociates the old rule via its   |
43 |                                       |               | setting object                                            |
44 +---------------------------------------+---------------+-----------------------------------------------------------+
45
46 create_image_tests.py - CreateImageSuccessTests
47 -----------------------------------------------
48
49 +---------------------------------------+---------------+-----------------------------------------------------------+
50 | Test Name                             | Glance API    | Description                                               |
51 +=======================================+===============+===========================================================+
52 | test_create_image_clean_url           | 1 & 2         | Ensures the OpenStackImage class can create an image from |
53 |                                       |               | a download URL location                                   |
54 +---------------------------------------+---------------+-----------------------------------------------------------+
55 | test_create_image_clean_url_properties| 1 & 2         | Ensures the OpenStackImage class can create an image from |
56 |                                       |               | a download URL location with custom properties            |
57 +---------------------------------------+---------------+-----------------------------------------------------------+
58 | test_create_image_clean_file          | 1 & 2         | Ensures the OpenStackImage class can create an image from |
59 |                                       |               | a locally sourced image file                              |
60 +---------------------------------------+---------------+-----------------------------------------------------------+
61 | test_create_delete_image              | 1 & 2         | Ensures the OpenStackImage.clean() method deletes an image|
62 |                                       |               | and does not raise an exception on subsequent calls to the|
63 |                                       |               | clean() method                                            |
64 +---------------------------------------+---------------+-----------------------------------------------------------+
65 | test_create_same_image                | 1 & 2         | Ensures the OpenStackImage.create() method does not create|
66 |                                       |               | another image when one already exists with the same name  |
67 +---------------------------------------+---------------+-----------------------------------------------------------+
68 | test_create_same_image_new_settings   | 1 & 2         | Tests the creation of an OpenStack image when the image   |
69 |                                       |               | already exists and the configuration only contains the    |
70 |                                       |               | the name.                                                 |
71 +---------------------------------------+---------------+-----------------------------------------------------------+
72
73 create_image_tests.py - CreateImageNegativeTests
74 ------------------------------------------------
75
76 +---------------------------------------+---------------+-----------------------------------------------------------+
77 | Test Name                             | Glance API    | Description                                               |
78 +=======================================+===============+===========================================================+
79 | test_bad_image_name                   | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
80 |                                       |               | being raised when the ImageSettings.name attribute has    |
81 |                                       |               | not been set                                              |
82 +---------------------------------------+---------------+-----------------------------------------------------------+
83 | test_bad_image_url                    | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
84 |                                       |               | being raised when the download URL is invalid             |
85 +---------------------------------------+---------------+-----------------------------------------------------------+
86 | test_bad_image_type                   | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
87 |                                       |               | being raised when the image format is 'foo'               |
88 +---------------------------------------+---------------+-----------------------------------------------------------+
89 | test_bad_image_file                   | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
90 |                                       |               | being raised when the image file does not exist           |
91 +---------------------------------------+---------------+-----------------------------------------------------------+
92
93 create_image_tests.py - CreateMultiPartImageTests
94 -------------------------------------------------
95
96 +----------------------------------------+---------------+-----------------------------------------------------------+
97 | Test Name                              | Glance API    | Description                                               |
98 +========================================+===============+===========================================================+
99 | test_create_three_part_image_from_url  | 1 & 2         | Ensures that a 3-part image can be created when each part |
100 |                                        |               | is being sourced from URLs                                |
101 +----------------------------------------+---------------+-----------------------------------------------------------+
102 | test_create_three_part_image_from_file | 1 & 2         | Ensures that a 3-part image can be created when each part |
103 | _3_creators                            |               | is being sourced from local files and 3 creators are used |
104 +----------------------------------------+---------------+-----------------------------------------------------------+
105 | test_create_three_part_image_from_url  | 1 & 2         | Ensures that a 3-part image can be created when each part |
106 | _3_creators                            |               | is being sourced from a URL and 3 creators are used       |
107 +----------------------------------------+---------------+-----------------------------------------------------------+
108
109 create_keypairs_tests.py - CreateKeypairsTests
110 ----------------------------------------------
111
112 +---------------------------------------+---------------+-----------------------------------------------------------+
113 | Test Name                             | Nova API      | Description                                               |
114 +=======================================+===============+===========================================================+
115 | test_create_keypair_only              | 2             | Ensures that a keypair object can be created simply by    |
116 |                                       |               | only configuring a name                                   |
117 +---------------------------------------+---------------+-----------------------------------------------------------+
118 | test_create_delete_keypair            | 2             | Ensures that a keypair object is deleted via              |
119 |                                       |               | OpenStackKeypair.clean() and subsequent calls do not      |
120 |                                       |               | result in exceptions                                      |
121 +---------------------------------------+---------------+-----------------------------------------------------------+
122 | test_create_keypair_save_pub_only     | 2             | Ensures that a keypair object can be created when the only|
123 |                                       |               | the public key is cached to disk                          |
124 +---------------------------------------+---------------+-----------------------------------------------------------+
125 | test_create_keypair_save_both         | 2             | Ensures that a keypair object can be created when both the|
126 |                                       |               | public and private keys are cached to disk                |
127 +---------------------------------------+---------------+-----------------------------------------------------------+
128 | test_create_keypair_from_file         | 2             | Ensures that a keypair object can be created with an      |
129 |                                       |               | existing public key file                                  |
130 +---------------------------------------+---------------+-----------------------------------------------------------+
131
132 create_keypairs_tests.py - CreateKeypairsCleanupTests
133 -----------------------------------------------------
134
135 +---------------------------------------+---------------+-----------------------------------------------------------+
136 | Test Name                             | Nova API      | Description                                               |
137 +=======================================+===============+===========================================================+
138 | test_create_keypair_gen_files_delete_1| 2             | Ensures that new keypair files are deleted by default     |
139 |                                       |               | by OpenStackKeypair#clean()                               |
140 +---------------------------------------+---------------+-----------------------------------------------------------+
141 | test_create_keypair_gen_files_delete_2| 2             | Ensures that new keypair files are deleted by         |
142 |                                       |               | OpenStackKeypair#clean() when the settings delete_on_clean|
143 |                                       |               | attribute is set to True                                  |
144 +---------------------------------------+---------------+-----------------------------------------------------------+
145 | test_create_keypair_gen_files_keep    | 2             | Ensures that new keypair files are not deleted by         |
146 |                                       |               | OpenStackKeypair#clean()                                  |
147 +---------------------------------------+---------------+-----------------------------------------------------------+
148 | test_create_keypair_exist_files_keep  | 2             | Ensures that existing keypair files are not deleted by    |
149 |                                       |               | OpenStackKeypair#clean()                                  |
150 +---------------------------------------+---------------+-----------------------------------------------------------+
151 | test_create_keypair_exist_files_delete| 2             | Ensures that existing keypair files are deleted by        |
152 |                                       |               | OpenStackKeypair#clean()                                  |
153 +---------------------------------------+---------------+-----------------------------------------------------------+
154
155 create_network_tests.py - CreateNetworkSuccessTests
156 ---------------------------------------------------
157
158 +---------------------------------------+---------------+-----------------------------------------------------------+
159 | Test Name                             | Neutron API   | Description                                               |
160 +=======================================+===============+===========================================================+
161 | test_create_network_without_router    | 2             | Ensures that a network can be created via the             |
162 |                                       |               | OpenStackNetwork class without any routers                |
163 +---------------------------------------+---------------+-----------------------------------------------------------+
164 | test_create_delete_network            | 2             | Ensures that a router can be deleted via the              |
165 |                                       |               | OpenStackNetwork.clean() method                           |
166 +---------------------------------------+---------------+-----------------------------------------------------------+
167 | test_create_network_with_router       | 2             | Ensures that a network can be created via the             |
168 |                                       |               | OpenStackNetwork class with a router                      |
169 +---------------------------------------+---------------+-----------------------------------------------------------+
170 | test_create_networks_same_name        | 2             | Ensures that the OpenStackNetwork.create() method will not|
171 |                                       |               | create a network with the same name                       |
172 +---------------------------------------+---------------+-----------------------------------------------------------+
173 | test_create_networks_router_admin_user| 2             | Ensures that the networks, subnets, and routers can be    |
174 | _to_new_project                       |               | create created by an admin user and assigned to a new     |
175 |                                       |               | project ID                                                |
176 +---------------------------------------+---------------+-----------------------------------------------------------+
177 | test_create_networks_router_new_user  | 2             | Ensures that the networks, subnets, and routers can be    |
178 | _to_admin_project                     |               | create created by a new admin user and assigned to the    |
179 |                                       |               | 'admin' project ID                                        |
180 +---------------------------------------+---------------+-----------------------------------------------------------+
181
182 create_router_tests.py - CreateRouterSuccessTests
183 -------------------------------------------------
184
185 +---------------------------------------+---------------+-----------------------------------------------------------+
186 | Test Name                             | Neutron API   | Description                                               |
187 +=======================================+===============+===========================================================+
188 | test_create_router_vanilla            | 2             | Ensures that a router can be created via the              |
189 |                                       |               | OpenStackRouter class with minimal settings               |
190 +---------------------------------------+---------------+-----------------------------------------------------------+
191 | test_create_router_admin_user_to_new  | 2             | Ensures that a router can be created by an admin user and |
192 | _project                              |               | assigned to a new project                                 |
193 +---------------------------------------+---------------+-----------------------------------------------------------+
194 | test_create_router_new_user_to_admin  | 2             | Ensures that a router can be created by a new user and    |
195 | _project                              |               | assigned to the admin project                             |
196 +---------------------------------------+---------------+-----------------------------------------------------------+
197 | test_create_delete_router             | 2             | Ensures that a router can be deleted via the              |
198 |                                       |               | OpenStackRouter.clean() method                            |
199 +---------------------------------------+---------------+-----------------------------------------------------------+
200 | test_create_router_admin_state_false  | 2             | Ensures that a router can created with                    |
201 |                                       |               | admin_state_up = False                                    |
202 +---------------------------------------+---------------+-----------------------------------------------------------+
203 | test_create_router_admin_state_True   | 2             | Ensures that a router can created with                    |
204 |                                       |               | admin_state_up = True                                     |
205 +---------------------------------------+---------------+-----------------------------------------------------------+
206 | test_create_router_private_network    | 2             | Ensures that a router port can be created against a       |
207 |                                       |               | private network                                           |
208 +---------------------------------------+---------------+-----------------------------------------------------------+
209 | test_create_router_external_network   | 2             | Ensures that a router can be created that is connected to |
210 |                                       |               | both external and private internal networks               |
211 +---------------------------------------+---------------+-----------------------------------------------------------+
212
213 create_router_tests.py - CreateRouterNegativeTests
214 --------------------------------------------------
215
216 +----------------------------------------+---------------+-----------------------------------------------------------+
217 | Test Name                              | Neutron API   | Description                                               |
218 +========================================+===============+===========================================================+
219 | test_create_router_noname              | 2             | Ensures that an exception is raised when attempting to    |
220 |                                        |               | create a router without a name                            |
221 +----------------------------------------+---------------+-----------------------------------------------------------+
222 | test_create_router_invalid_gateway_name| 2             | Ensures that an exception is raised when attempting to    |
223 |                                        |               | create a router to an external network that does not exist|
224 +----------------------------------------+---------------+-----------------------------------------------------------+
225
226 create_stack_tests.py - CreateStackSuccessTests
227 -----------------------------------------------
228
229 +---------------------------------------+---------------+-----------------------------------------------------------+
230 | Test Name                             | Neutron API   | Description                                               |
231 +=======================================+===============+===========================================================+
232 | test_create_stack_template_file       | 2             | Ensures that a Heat stack can be created with a file-based|
233 |                                       |               | Heat template file                                        |
234 +---------------------------------------+---------------+-----------------------------------------------------------+
235 | test_create_stack_template_dict       | 2             | Ensures that a Heat stack can be created with a dictionary|
236 |                                       |               | Heat template                                             |
237 +---------------------------------------+---------------+-----------------------------------------------------------+
238 | test_create_delete_stack              | 2             | Ensures that a Heat stack can be created and deleted      |
239 |                                       |               | while having clean() called 2x without an exception       |
240 +---------------------------------------+---------------+-----------------------------------------------------------+
241 | test_create_same_stack                | 2             | Ensures that a Heat stack with the same name cannot be    |
242 |                                       |               | created 2x                                                |
243 +---------------------------------------+---------------+-----------------------------------------------------------+
244
245 create_stack_tests.py - CreateStackNegativeTests
246 --------------------------------------------------
247
248 +----------------------------------------+---------------+-----------------------------------------------------------+
249 | Test Name                              | Neutron API   | Description                                               |
250 +========================================+===============+===========================================================+
251 | test_missing_dependencies              | 2             | Ensures that a Heat template fails to deploy when expected|
252 |                                        |               | dependencies are missing                                  |
253 +----------------------------------------+---------------+-----------------------------------------------------------+
254 | test_bad_stack_file                    | 2             | Ensures that a Heat template fails to deploy when the Heat|
255 |                                        |               | template file does not exist                              |
256 +----------------------------------------+---------------+-----------------------------------------------------------+
257
258 create_instance_tests.py - CreateInstanceSimpleTests
259 ----------------------------------------------------
260
261 +---------------------------------------+---------------+-----------------------------------------------------------+
262 | Test Name                             | API Versions  | Description                                               |
263 +=======================================+===============+===========================================================+
264 | test_create_delete_instance           | Nova 2        | Ensures that the OpenStackVmInstance.clean() method       |
265 |                                       | Neutron 2     | deletes the instance                                      |
266 +---------------------------------------+---------------+-----------------------------------------------------------+
267
268 create_instance_tests.py - SimpleHealthCheck
269 --------------------------------------------
270
271 +---------------------------------------+---------------+-----------------------------------------------------------+
272 | Test Name                             | API Versions  | Description                                               |
273 +=======================================+===============+===========================================================+
274 | test_check_vm_ip_dhcp                 | Nova 2        | Tests the creation of an OpenStack instance with a single |
275 |                                       | Neutron 2     | port and it's assigned IP address                         |
276 +---------------------------------------+---------------+-----------------------------------------------------------+
277
278 create_instance_tests.py - CreateInstanceSingleNetworkTests
279 -----------------------------------------------------------
280
281 +---------------------------------------+---------------+-----------------------------------------------------------+
282 | Test Name                             | API Versions  | Description                                               |
283 +=======================================+===============+===========================================================+
284 | test_single_port_static               | Nova 2        | Ensures that an instance with a single port/NIC with a    |
285 |                                       | Neutron 2     | static IP can be created                                  |
286 +---------------------------------------+---------------+-----------------------------------------------------------+
287 | test_ssh_client_fip_before_active     | Nova 2        | Ensures that an instance can be reached over SSH when the |
288 |                                       | Neutron 2     | floating IP is assigned prior to the VM becoming ACTIVE   |
289 +---------------------------------------+---------------+-----------------------------------------------------------+
290 | test_ssh_client_fip_after_active      | Nova 2        | Ensures that an instance can be reached over SSH when the |
291 |                                       | Neutron 2     | floating IP is assigned after to the VM becoming ACTIVE   |
292 +---------------------------------------+---------------+-----------------------------------------------------------+
293
294 create_instance_tests.py - CreateInstancePortManipulationTests
295 --------------------------------------------------------------
296
297 +---------------------------------------+---------------+-----------------------------------------------------------+
298 | Test Name                             | API Versions  | Description                                               |
299 +=======================================+===============+===========================================================+
300 | test_set_custom_valid_ip_one_subnet   | Nova 2        | Ensures that an instance's can have a valid static IP is  |
301 |                                       | Neutron 2     | properly assigned                                         |
302 +---------------------------------------+---------------+-----------------------------------------------------------+
303 | test_set_custom_invalid_ip_one_subnet | Nova 2        | Ensures that an instance's port with an invalid static IP |
304 |                                       | Neutron 2     | raises an exception                                       |
305 +---------------------------------------+---------------+-----------------------------------------------------------+
306 | test_set_custom_valid_mac             | Nova 2        | Ensures that an instance's port can have a valid MAC      |
307 |                                       | Neutron 2     | address properly assigned                                 |
308 +---------------------------------------+---------------+-----------------------------------------------------------+
309 | test_set_custom_invalid_mac           | Nova 2        | Ensures that an instance's port with an invalid MAC       |
310 |                                       | Neutron 2     | address raises and exception                              |
311 +---------------------------------------+---------------+-----------------------------------------------------------+
312 | test_set_custom_mac_and_ip            | Nova 2        | Ensures that an instance's port with a valid static IP and|
313 |                                       | Neutron 2     | MAC are properly assigned                                 |
314 +---------------------------------------+---------------+-----------------------------------------------------------+
315 | test_set_allowed_address_pairs        | Nova 2        | Ensures the configured allowed_address_pairs is properly  |
316 |                                       | Neutron 2     | set on a VMs port                                         |
317 +---------------------------------------+---------------+-----------------------------------------------------------+
318 | test_set_allowed_address_pairs_bad_mac| Nova 2        | Ensures the port cannot be created when a bad MAC address |
319 |                                       | Neutron 2     | format is used in the allowed_address_pairs port attribute|
320 +---------------------------------------+---------------+-----------------------------------------------------------+
321 | test_set_allowed_address_pairs_bad_ip | Nova 2        | Ensures the port cannot be created when a bad IP address  |
322 |                                       | Neutron 2     | format is used in the allowed_address_pairs port attribute|
323 +---------------------------------------+---------------+-----------------------------------------------------------+
324
325 create_instance_tests.py - CreateInstanceOnComputeHost
326 ------------------------------------------------------
327
328 +---------------------------------------+---------------+-----------------------------------------------------------+
329 | Test Name                             | API Versions  | Description                                               |
330 +=======================================+===============+===========================================================+
331 | test_deploy_vm_to_each_compute_node   | Nova 2        | Tests to ensure that one can fire up an instance on each  |
332 |                                       | Neutron 2     | active compute node                                       |
333 +---------------------------------------+---------------+-----------------------------------------------------------+
334
335 create_instance_tests.py - CreateInstanceFromThreePartImage
336 -----------------------------------------------------------
337
338 +-----------------------------------------------------+---------------+-----------------------------------------------------------+
339 | Test Name                                           | API Versions  | Description                                               |
340 +=====================================================+===============+===========================================================+
341 | test_create_delete_instance_from_three_part_image   | Nova 2        | Tests to ensure that one can fire up an instance then     |
342 |                                                     | Neutron 2     | delete it when using a 3-part image                       |
343 +-----------------------------------------------------+---------------+-----------------------------------------------------------+
344
345 create_instance_tests.py - CreateInstancePubPrivNetTests
346 --------------------------------------------------------
347
348 +---------------------------------------+---------------+-----------------------------------------------------------+
349 | Test Name                             | API Versions  | Description                                               |
350 +=======================================+===============+===========================================================+
351 | test_dual_ports_dhcp                  | Nova 2        | Ensures that a VM with two ports/NICs can have its second |
352 |                                       | Neutron 2     | NIC configured via SSH/Ansible after startup              |
353 +---------------------------------------+---------------+-----------------------------------------------------------+
354
355 create_instance_tests.py - InstanceSecurityGroupTests
356 -----------------------------------------------------
357
358 +---------------------------------------+---------------+-----------------------------------------------------------+
359 | Test Name                             | API Versions  | Description                                               |
360 +=======================================+===============+===========================================================+
361 | test_add_security_group               | Nova 2        | Ensures that a VM instance can have security group added  |
362 |                                       | Neutron 2     | to it while its running                                   |
363 +---------------------------------------+---------------+-----------------------------------------------------------+
364 | test_add_invalid_security_group       | Nova 2        | Ensures that a VM instance does not accept the addition of|
365 |                                       | Neutron 2     | a security group that no longer exists                    |
366 +---------------------------------------+---------------+-----------------------------------------------------------+
367 | test_remove_security_group            | Nova 2        | Ensures that a VM instance accepts the removal of a       |
368 |                                       | Neutron 2     | security group                                            |
369 +---------------------------------------+---------------+-----------------------------------------------------------+
370 | test_remove_security_group_never_added| Nova 2        | Ensures that a VM instance does not accept the removal of |
371 |                                       | Neutron 2     | a security group that was never added in the first place  |
372 +---------------------------------------+---------------+-----------------------------------------------------------+
373 | test_add_same_security_group          | Nova 2        | Ensures that a VM instance does not add a security group  |
374 |                                       | Neutron 2     | that has already been added to the instance               |
375 +---------------------------------------+---------------+-----------------------------------------------------------+
376
377 ansible_utils_tests.py - AnsibleProvisioningTests
378 -------------------------------------------------
379
380 +---------------------------------------+---------------+-----------------------------------------------------------+
381 | Test Name                             | API Versions  | Description                                               |
382 +=======================================+===============+===========================================================+
383 | test_apply_simple_playbook            | Nova 2        | Ensures that an instance assigned with a floating IP will |
384 |                                       | Neutron 2     | apply a simple Ansible playbook                           |
385 +---------------------------------------+---------------+-----------------------------------------------------------+
386 | test_apply_template_playbook          | Nova 2        | Ensures that an instance assigned with a floating IP will |
387 |                                       | Neutron 2     | apply a Ansible playbook containing Jinga2 substitution   |
388 |                                       |               | values                                                    |
389 +---------------------------------------+---------------+-----------------------------------------------------------+