Expand OpenStackSecurityGroup class tests.
[snaps.git] / docs / how-to-use / IntegrationTests.rst
1 SNAPS OpenStack Integration Testing
2 ===================================
3
4 These tests are ones designed to be run within their own dynamically created project along with a newly generated user
5 account and generally require other OpenStack object creators.
6
7 The Test Classes
8 ================
9
10 create_security_group_tests.py - CreateSecurityGroupTests
11 ---------------------------------------------------------
12
13 +---------------------------------------+---------------+-----------------------------------------------------------+
14 | Test Name                             | API Versions  | Description                                               |
15 +=======================================+===============+===========================================================+
16 | test_create_group_without_rules       | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can create a     |
17 |                                       | Neutron 2     | security group without any rules                          |
18 +---------------------------------------+---------------+-----------------------------------------------------------+
19 | test_create_group_admin_user_to_new   | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can be created   |
20 | _project                              | Neutron 2     | by the admin user and associated with a new project       |
21 +---------------------------------------+---------------+-----------------------------------------------------------+
22 | test_create_group_new_user_to_admin   | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can be created   |
23 | _project                              | Neutron 2     | by the new user and associated with the admin project     |
24 +---------------------------------------+---------------+-----------------------------------------------------------+
25 | test_create_delete_group              | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class clean() method   |
26 |                                       | Neutron 2     | will not raise an exception should the group be deleted by|
27 |                                       |               | some other process                                        |
28 +---------------------------------------+---------------+-----------------------------------------------------------+
29 | test_create_group_with_one_simple_rule| Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can create a     |
30 |                                       | Neutron 2     | security group with a single simple rule                  |
31 +---------------------------------------+---------------+-----------------------------------------------------------+
32 | test_create_group_with_one_complex    | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can create a     |
33 | _rule                                 | Neutron 2     | security group with a single complex rule                 |
34 +---------------------------------------+---------------+-----------------------------------------------------------+
35 | test_create_group_with_several_rules  | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup class can create a     |
36 |                                       | Neutron 2     | security group with several rules                         |
37 +---------------------------------------+---------------+-----------------------------------------------------------+
38 | test_add_rule                         | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup#add_rule() method      |
39 |                                       | Neutron 2     | properly creates and associates the new rule              |
40 +---------------------------------------+---------------+-----------------------------------------------------------+
41 | test_remove_rule_by_id                | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup#remove_rule() method   |
42 |                                       | Neutron 2     | properly deletes and disassociates the old rule via its ID|
43 +---------------------------------------+---------------+-----------------------------------------------------------+
44 | test_remove_rule_by_setting           | Keysone 2 & 3 | Ensures the OpenStackSecurityGroup#remove_rule() method   |
45 |                                       | Neutron 2     | properly deletes and disassociates the old rule via its   |
46 |                                       |               | setting object                                            |
47 +---------------------------------------+---------------+-----------------------------------------------------------+
48
49 create_image_tests.py - CreateImageSuccessTests
50 -----------------------------------------------
51
52 +---------------------------------------+---------------+-----------------------------------------------------------+
53 | Test Name                             | Glance API    | Description                                               |
54 +=======================================+===============+===========================================================+
55 | test_create_image_clean_url           | 1 & 2         | Ensures the OpenStackImage class can create an image from |
56 |                                       |               | a download URL location                                   |
57 +---------------------------------------+---------------+-----------------------------------------------------------+
58 | test_create_image_clean_url_properties| 1 & 2         | Ensures the OpenStackImage class can create an image from |
59 |                                       |               | a download URL location with custom properties            |
60 +---------------------------------------+---------------+-----------------------------------------------------------+
61 | test_create_image_clean_file          | 1 & 2         | Ensures the OpenStackImage class can create an image from |
62 |                                       |               | a locally sourced image file                              |
63 +---------------------------------------+---------------+-----------------------------------------------------------+
64 | test_create_delete_image              | 1 & 2         | Ensures the OpenStackImage.clean() method deletes an image|
65 |                                       |               | and does not raise an exception on subsequent calls to the|
66 |                                       |               | clean() method                                            |
67 +---------------------------------------+---------------+-----------------------------------------------------------+
68 | test_create_same_image                | 1 & 2         | Ensures the OpenStackImage.create() method does not create|
69 |                                       |               | another image when one already exists with the same name  |
70 +---------------------------------------+---------------+-----------------------------------------------------------+
71 | test_create_same_image_new_settings   | 1 & 2         | Tests the creation of an OpenStack image when the image   |
72 |                                       |               | already exists and the configuration only contains the    |
73 |                                       |               | the name.                                                 |
74 +---------------------------------------+---------------+-----------------------------------------------------------+
75
76 create_image_tests.py - CreateImageNegativeTests
77 ------------------------------------------------
78
79 +---------------------------------------+---------------+-----------------------------------------------------------+
80 | Test Name                             | Glance API    | Description                                               |
81 +=======================================+===============+===========================================================+
82 | test_bad_image_name                   | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
83 |                                       |               | being raised when the ImageSettings.name attribute has    |
84 |                                       |               | not been set                                              |
85 +---------------------------------------+---------------+-----------------------------------------------------------+
86 | test_bad_image_url                    | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
87 |                                       |               | being raised when the download URL is invalid             |
88 +---------------------------------------+---------------+-----------------------------------------------------------+
89 | test_bad_image_type                   | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
90 |                                       |               | being raised when the image format is 'foo'               |
91 +---------------------------------------+---------------+-----------------------------------------------------------+
92 | test_bad_image_file                   | 1 & 2         | Ensures OpenStackImage.create() results in an Exception   |
93 |                                       |               | being raised when the image file does not exist           |
94 +---------------------------------------+---------------+-----------------------------------------------------------+
95
96 create_image_tests.py - CreateMultiPartImageTests
97 -------------------------------------------------
98
99 +----------------------------------------+---------------+-----------------------------------------------------------+
100 | Test Name                              | Glance API    | Description                                               |
101 +========================================+===============+===========================================================+
102 | test_create_three_part_image_from_url  | 1 & 2         | Ensures that a 3-part image can be created when each part |
103 |                                        |               | is being sourced from URLs                                |
104 +----------------------------------------+---------------+-----------------------------------------------------------+
105 | test_create_three_part_image_from_file | 1 & 2         | Ensures that a 3-part image can be created when each part |
106 | _3_creators                            |               | is being sourced from local files and 3 creators are used |
107 +----------------------------------------+---------------+-----------------------------------------------------------+
108 | test_create_three_part_image_from_url  | 1 & 2         | Ensures that a 3-part image can be created when each part |
109 | _3_creators                            |               | is being sourced from a URL and 3 creators are used       |
110 +----------------------------------------+---------------+-----------------------------------------------------------+
111
112 create_keypairs_tests.py - CreateKeypairsTests
113 ----------------------------------------------
114
115 +---------------------------------------+---------------+-----------------------------------------------------------+
116 | Test Name                             | Nova API      | Description                                               |
117 +=======================================+===============+===========================================================+
118 | test_create_keypair_only              | 2             | Ensures that a keypair object can be created simply by    |
119 |                                       |               | only configuring a name                                   |
120 +---------------------------------------+---------------+-----------------------------------------------------------+
121 | test_create_delete_keypair            | 2             | Ensures that a keypair object is deleted via              |
122 |                                       |               | OpenStackKeypair.clean() and subsequent calls do not      |
123 |                                       |               | result in exceptions                                      |
124 +---------------------------------------+---------------+-----------------------------------------------------------+
125 | test_create_keypair_save_pub_only     | 2             | Ensures that a keypair object can be created when the only|
126 |                                       |               | the public key is cached to disk                          |
127 +---------------------------------------+---------------+-----------------------------------------------------------+
128 | test_create_keypair_save_both         | 2             | Ensures that a keypair object can be created when both the|
129 |                                       |               | public and private keys are cached to disk                |
130 +---------------------------------------+---------------+-----------------------------------------------------------+
131 | test_create_keypair_from_file         | 2             | Ensures that a keypair object can be created with an      |
132 |                                       |               | existing public key file                                  |
133 +---------------------------------------+---------------+-----------------------------------------------------------+
134
135 create_keypairs_tests.py - CreateKeypairsCleanupTests
136 -----------------------------------------------------
137
138 +---------------------------------------+---------------+-----------------------------------------------------------+
139 | Test Name                             | Nova API      | Description                                               |
140 +=======================================+===============+===========================================================+
141 | test_create_keypair_gen_files_delete_1| 2             | Ensures that new keypair files are deleted by default     |
142 |                                       |               | by OpenStackKeypair#clean()                               |
143 +---------------------------------------+---------------+-----------------------------------------------------------+
144 | test_create_keypair_gen_files_delete_2| 2             | Ensures that new keypair files are deleted by         |
145 |                                       |               | OpenStackKeypair#clean() when the settings delete_on_clean|
146 |                                       |               | attribute is set to True                                  |
147 +---------------------------------------+---------------+-----------------------------------------------------------+
148 | test_create_keypair_gen_files_keep    | 2             | Ensures that new keypair files are not deleted by         |
149 |                                       |               | OpenStackKeypair#clean()                                  |
150 +---------------------------------------+---------------+-----------------------------------------------------------+
151 | test_create_keypair_exist_files_keep  | 2             | Ensures that existing keypair files are not deleted by    |
152 |                                       |               | OpenStackKeypair#clean()                                  |
153 +---------------------------------------+---------------+-----------------------------------------------------------+
154 | test_create_keypair_exist_files_delete| 2             | Ensures that existing keypair files are deleted by        |
155 |                                       |               | OpenStackKeypair#clean()                                  |
156 +---------------------------------------+---------------+-----------------------------------------------------------+
157
158 create_network_tests.py - CreateNetworkSuccessTests
159 ---------------------------------------------------
160
161 +---------------------------------------+---------------+-----------------------------------------------------------+
162 | Test Name                             | Neutron API   | Description                                               |
163 +=======================================+===============+===========================================================+
164 | test_create_network_without_router    | 2             | Ensures that a network can be created via the             |
165 |                                       |               | OpenStackNetwork class without any routers                |
166 +---------------------------------------+---------------+-----------------------------------------------------------+
167 | test_create_delete_network            | 2             | Ensures that a router can be deleted via the              |
168 |                                       |               | OpenStackNetwork.clean() method                           |
169 +---------------------------------------+---------------+-----------------------------------------------------------+
170 | test_create_network_with_router       | 2             | Ensures that a network can be created via the             |
171 |                                       |               | OpenStackNetwork class with a router                      |
172 +---------------------------------------+---------------+-----------------------------------------------------------+
173 | test_create_networks_same_name        | 2             | Ensures that the OpenStackNetwork.create() method will not|
174 |                                       |               | create a network with the same name                       |
175 +---------------------------------------+---------------+-----------------------------------------------------------+
176 | test_create_networks_router_admin_user| 2             | Ensures that the networks, subnets, and routers can be    |
177 | _to_new_project                       |               | create created by an admin user and assigned to a new     |
178 |                                       |               | project ID                                                |
179 +---------------------------------------+---------------+-----------------------------------------------------------+
180 | test_create_networks_router_new_user  | 2             | Ensures that the networks, subnets, and routers can be    |
181 | _to_admin_project                     |               | create created by a new admin user and assigned to the    |
182 |                                       |               | 'admin' project ID                                        |
183 +---------------------------------------+---------------+-----------------------------------------------------------+
184
185 create_router_tests.py - CreateRouterSuccessTests
186 -------------------------------------------------
187
188 +---------------------------------------+---------------+-----------------------------------------------------------+
189 | Test Name                             | Neutron API   | Description                                               |
190 +=======================================+===============+===========================================================+
191 | test_create_router_vanilla            | 2             | Ensures that a router can be created via the              |
192 |                                       |               | OpenStackRouter class with minimal settings               |
193 +---------------------------------------+---------------+-----------------------------------------------------------+
194 | test_create_router_admin_user_to_new  | 2             | Ensures that a router can be created by an admin user and |
195 | _project                              |               | assigned to a new project                                 |
196 +---------------------------------------+---------------+-----------------------------------------------------------+
197 | test_create_router_new_user_to_admin  | 2             | Ensures that a router can be created by a new user and    |
198 | _project                              |               | assigned to the admin project                             |
199 +---------------------------------------+---------------+-----------------------------------------------------------+
200 | test_create_delete_router             | 2             | Ensures that a router can be deleted via the              |
201 |                                       |               | OpenStackRouter.clean() method                            |
202 +---------------------------------------+---------------+-----------------------------------------------------------+
203 | test_create_router_admin_state_false  | 2             | Ensures that a router can created with                    |
204 |                                       |               | admin_state_up = False                                    |
205 +---------------------------------------+---------------+-----------------------------------------------------------+
206 | test_create_router_admin_state_True   | 2             | Ensures that a router can created with                    |
207 |                                       |               | admin_state_up = True                                     |
208 +---------------------------------------+---------------+-----------------------------------------------------------+
209 | test_create_router_private_network    | 2             | Ensures that a router port can be created against a       |
210 |                                       |               | private network                                           |
211 +---------------------------------------+---------------+-----------------------------------------------------------+
212 | test_create_router_external_network   | 2             | Ensures that a router can be created that is connected to |
213 |                                       |               | both external and private internal networks               |
214 +---------------------------------------+---------------+-----------------------------------------------------------+
215
216 create_router_tests.py - CreateRouterNegativeTests
217 --------------------------------------------------
218
219 +----------------------------------------+---------------+-----------------------------------------------------------+
220 | Test Name                              | Neutron API   | Description                                               |
221 +========================================+===============+===========================================================+
222 | test_create_router_noname              | 2             | Ensures that an exception is raised when attempting to    |
223 |                                        |               | create a router without a name                            |
224 +----------------------------------------+---------------+-----------------------------------------------------------+
225 | test_create_router_invalid_gateway_name| 2             | Ensures that an exception is raised when attempting to    |
226 |                                        |               | create a router to an external network that does not exist|
227 +----------------------------------------+---------------+-----------------------------------------------------------+
228
229 create_stack_tests.py - CreateStackSuccessTests
230 -----------------------------------------------
231
232 +---------------------------------------+---------------+-----------------------------------------------------------+
233 | Test Name                             | Neutron API   | Description                                               |
234 +=======================================+===============+===========================================================+
235 | test_create_stack_template_file       | 2             | Ensures that a Heat stack can be created with a file-based|
236 |                                       |               | Heat template file                                        |
237 +---------------------------------------+---------------+-----------------------------------------------------------+
238 | test_create_stack_template_dict       | 2             | Ensures that a Heat stack can be created with a dictionary|
239 |                                       |               | Heat template                                             |
240 +---------------------------------------+---------------+-----------------------------------------------------------+
241 | test_create_delete_stack              | 2             | Ensures that a Heat stack can be created and deleted      |
242 |                                       |               | while having clean() called 2x without an exception       |
243 +---------------------------------------+---------------+-----------------------------------------------------------+
244 | test_create_same_stack                | 2             | Ensures that a Heat stack with the same name cannot be    |
245 |                                       |               | created 2x                                                |
246 +---------------------------------------+---------------+-----------------------------------------------------------+
247
248 create_stack_tests.py - CreateStackNegativeTests
249 ------------------------------------------------
250
251 +----------------------------------------+---------------+-----------------------------------------------------------+
252 | Test Name                              | Neutron API   | Description                                               |
253 +========================================+===============+===========================================================+
254 | test_missing_dependencies              | 2             | Ensures that a Heat template fails to deploy when expected|
255 |                                        |               | dependencies are missing                                  |
256 +----------------------------------------+---------------+-----------------------------------------------------------+
257 | test_bad_stack_file                    | 2             | Ensures that a Heat template fails to deploy when the Heat|
258 |                                        |               | template file does not exist                              |
259 +----------------------------------------+---------------+-----------------------------------------------------------+
260
261 create_instance_tests.py - CreateInstanceSimpleTests
262 ----------------------------------------------------
263
264 +---------------------------------------+---------------+-----------------------------------------------------------+
265 | Test Name                             | API Versions  | Description                                               |
266 +=======================================+===============+===========================================================+
267 | test_create_delete_instance           | Nova 2        | Ensures that the OpenStackVmInstance.clean() method       |
268 |                                       | Neutron 2     | deletes the instance                                      |
269 +---------------------------------------+---------------+-----------------------------------------------------------+
270
271 create_instance_tests.py - SimpleHealthCheck
272 --------------------------------------------
273
274 +---------------------------------------+---------------+-----------------------------------------------------------+
275 | Test Name                             | API Versions  | Description                                               |
276 +=======================================+===============+===========================================================+
277 | test_check_vm_ip_dhcp                 | Nova 2        | Tests the creation of an OpenStack instance with a single |
278 |                                       | Neutron 2     | port and it's assigned IP address                         |
279 +---------------------------------------+---------------+-----------------------------------------------------------+
280
281 create_instance_tests.py - CreateInstanceSingleNetworkTests
282 -----------------------------------------------------------
283
284 +---------------------------------------+---------------+-----------------------------------------------------------+
285 | Test Name                             | API Versions  | Description                                               |
286 +=======================================+===============+===========================================================+
287 | test_single_port_static               | Nova 2        | Ensures that an instance with a single port/NIC with a    |
288 |                                       | Neutron 2     | static IP can be created                                  |
289 +---------------------------------------+---------------+-----------------------------------------------------------+
290 | test_ssh_client_fip_before_active     | Nova 2        | Ensures that an instance can be reached over SSH when the |
291 |                                       | Neutron 2     | floating IP is assigned prior to the VM becoming ACTIVE   |
292 +---------------------------------------+---------------+-----------------------------------------------------------+
293 | test_ssh_client_fip_after_active      | Nova 2        | Ensures that an instance can be reached over SSH when the |
294 |                                       | Neutron 2     | floating IP is assigned after to the VM becoming ACTIVE   |
295 +---------------------------------------+---------------+-----------------------------------------------------------+
296
297 create_instance_tests.py - CreateInstancePortManipulationTests
298 --------------------------------------------------------------
299
300 +---------------------------------------+---------------+-----------------------------------------------------------+
301 | Test Name                             | API Versions  | Description                                               |
302 +=======================================+===============+===========================================================+
303 | test_set_custom_valid_ip_one_subnet   | Nova 2        | Ensures that an instance's can have a valid static IP is  |
304 |                                       | Neutron 2     | properly assigned                                         |
305 +---------------------------------------+---------------+-----------------------------------------------------------+
306 | test_set_custom_invalid_ip_one_subnet | Nova 2        | Ensures that an instance's port with an invalid static IP |
307 |                                       | Neutron 2     | raises an exception                                       |
308 +---------------------------------------+---------------+-----------------------------------------------------------+
309 | test_set_custom_valid_mac             | Nova 2        | Ensures that an instance's port can have a valid MAC      |
310 |                                       | Neutron 2     | address properly assigned                                 |
311 +---------------------------------------+---------------+-----------------------------------------------------------+
312 | test_set_custom_invalid_mac           | Nova 2        | Ensures that an instance's port with an invalid MAC       |
313 |                                       | Neutron 2     | address raises and exception                              |
314 +---------------------------------------+---------------+-----------------------------------------------------------+
315 | test_set_custom_mac_and_ip            | Nova 2        | Ensures that an instance's port with a valid static IP and|
316 |                                       | Neutron 2     | MAC are properly assigned                                 |
317 +---------------------------------------+---------------+-----------------------------------------------------------+
318 | test_set_allowed_address_pairs        | Nova 2        | Ensures the configured allowed_address_pairs is properly  |
319 |                                       | Neutron 2     | set on a VMs port                                         |
320 +---------------------------------------+---------------+-----------------------------------------------------------+
321 | test_set_allowed_address_pairs_bad_mac| Nova 2        | Ensures the port cannot be created when a bad MAC address |
322 |                                       | Neutron 2     | format is used in the allowed_address_pairs port attribute|
323 +---------------------------------------+---------------+-----------------------------------------------------------+
324 | test_set_allowed_address_pairs_bad_ip | Nova 2        | Ensures the port cannot be created when a bad IP address  |
325 |                                       | Neutron 2     | format is used in the allowed_address_pairs port attribute|
326 +---------------------------------------+---------------+-----------------------------------------------------------+
327
328 create_instance_tests.py - CreateInstanceOnComputeHost
329 ------------------------------------------------------
330
331 +---------------------------------------+---------------+-----------------------------------------------------------+
332 | Test Name                             | API Versions  | Description                                               |
333 +=======================================+===============+===========================================================+
334 | test_deploy_vm_to_each_compute_node   | Nova 2        | Tests to ensure that one can fire up an instance on each  |
335 |                                       | Neutron 2     | active compute node                                       |
336 +---------------------------------------+---------------+-----------------------------------------------------------+
337
338 create_instance_tests.py - CreateInstanceFromThreePartImage
339 -----------------------------------------------------------
340
341 +-----------------------------------------------------+---------------+-----------------------------------------------------------+
342 | Test Name                                           | API Versions  | Description                                               |
343 +=====================================================+===============+===========================================================+
344 | test_create_delete_instance_from_three_part_image   | Nova 2        | Tests to ensure that one can fire up an instance then     |
345 |                                                     | Neutron 2     | delete it when using a 3-part image                       |
346 +-----------------------------------------------------+---------------+-----------------------------------------------------------+
347
348 create_instance_tests.py - CreateInstancePubPrivNetTests
349 --------------------------------------------------------
350
351 +---------------------------------------+---------------+-----------------------------------------------------------+
352 | Test Name                             | API Versions  | Description                                               |
353 +=======================================+===============+===========================================================+
354 | test_dual_ports_dhcp                  | Nova 2        | Ensures that a VM with two ports/NICs can have its second |
355 |                                       | Neutron 2     | NIC configured via SSH/Ansible after startup              |
356 +---------------------------------------+---------------+-----------------------------------------------------------+
357
358 create_instance_tests.py - InstanceSecurityGroupTests
359 -----------------------------------------------------
360
361 +---------------------------------------+---------------+-----------------------------------------------------------+
362 | Test Name                             | API Versions  | Description                                               |
363 +=======================================+===============+===========================================================+
364 | test_add_security_group               | Nova 2        | Ensures that a VM instance can have security group added  |
365 |                                       | Neutron 2     | to it while its running                                   |
366 +---------------------------------------+---------------+-----------------------------------------------------------+
367 | test_add_invalid_security_group       | Nova 2        | Ensures that a VM instance does not accept the addition of|
368 |                                       | Neutron 2     | a security group that no longer exists                    |
369 +---------------------------------------+---------------+-----------------------------------------------------------+
370 | test_remove_security_group            | Nova 2        | Ensures that a VM instance accepts the removal of a       |
371 |                                       | Neutron 2     | security group                                            |
372 +---------------------------------------+---------------+-----------------------------------------------------------+
373 | test_remove_security_group_never_added| Nova 2        | Ensures that a VM instance does not accept the removal of |
374 |                                       | Neutron 2     | a security group that was never added in the first place  |
375 +---------------------------------------+---------------+-----------------------------------------------------------+
376 | test_add_same_security_group          | Nova 2        | Ensures that a VM instance does not add a security group  |
377 |                                       | Neutron 2     | that has already been added to the instance               |
378 +---------------------------------------+---------------+-----------------------------------------------------------+
379
380 ansible_utils_tests.py - AnsibleProvisioningTests
381 -------------------------------------------------
382
383 +---------------------------------------+---------------+-----------------------------------------------------------+
384 | Test Name                             | API Versions  | Description                                               |
385 +=======================================+===============+===========================================================+
386 | test_apply_simple_playbook            | Nova 2        | Ensures that an instance assigned with a floating IP will |
387 |                                       | Neutron 2     | apply a simple Ansible playbook                           |
388 +---------------------------------------+---------------+-----------------------------------------------------------+
389 | test_apply_template_playbook          | Nova 2        | Ensures that an instance assigned with a floating IP will |
390 |                                       | Neutron 2     | apply a Ansible playbook containing Jinga2 substitution   |
391 |                                       |               | values                                                    |
392 +---------------------------------------+---------------+-----------------------------------------------------------+