1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
7 This section gives some guidelines about how to troubleshoot the test cases
10 **IMPORTANT**: As in the previous section, the steps defined below must be
11 executed inside the Functest Docker container and after sourcing the OpenStack credentials::
17 source /home/opnfv/functest/conf/openstack.creds
22 This section covers the test cases related to the VIM (vPing, Tempest, Rally).
26 For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first steps are
31 * Create Security Group
34 After these actions, the test cases differ and will be explained in their respective section.
36 These test cases can be run inside the container as follows::
38 $repos_dir/functest/docker/run_tests.sh -t vping_ssh
39 $repos_dir/functest/docker/run_tests.sh -t vping_userdata
41 The **run_tests.sh** script is basically calling internally the corresponding
42 vPing scripts, located in
43 *$repos_dir/functest/testcases/vPing/CI/libraries/vPing_ssh.py* and
44 *$repos_dir/functest/testcases/vPing/CI/libraries/vPing_userdata.py* with the
47 After finishing the test execution, the corresponding script will remove all
48 created resources in OpenStack (image, instances, network and security group).
49 When troubleshooting, it is advisable sometimes to keep those resources in case the
50 test fails and a manual testing is needed. This can be achieved by adding the flag *-n*::
52 $repos_dir/functest/docker/run_tests.sh -n -t vping_ssh
53 $repos_dir/functest/docker/run_tests.sh -n -t vping_userdata
56 Some of the common errors that can appear in this test case are::
58 vPing_ssh- ERROR - There has been a problem when creating the neutron network....
60 This means that there has been some problems with Neutron, even before creating the
61 instances. Try to create manually a Neutron network and a Subnet to see if that works.
62 The debug messages will also help to see when it failed (subnet and router creation).
63 Example of Neutron commands (using 10.6.0.0/24 range for example)::
65 neutron net-create net-test
66 neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 --gateway 10.6.0.254 net-test 10.6.0.0/24
67 neutron router-create test_router
68 neutron router-interface-add <ROUTER_ID> test_subnet
69 neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME>
71 Another related error can occur while creating the Security Groups for the instances::
73 vPing_ssh- ERROR - Failed to create the security group...
75 In this case, proceed to create it manually. These are some hints::
77 neutron security-group-create sg-test
78 neutron security-group-rule-create sg-test --direction ingress --protocol icmp --remote-ip-prefix 0.0.0.0/0
79 neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
80 neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
82 The next step is to create the instances. The image used is located in
83 */home/opnfv/functest/data/cirros-0.3.4-x86_64-disk.img* and a Glance image is created
84 with the name **functest-vping**. If booting the instances fails (i.e. the status
85 is not **ACTIVE**), you can check why it failed by doing::
88 nova show <INSTANCE_ID>
90 It might show some messages about the booting failure. To try that manually::
92 nova boot --flavor 2 --image functest-vping --nic net-id=<NET_ID> nova-test
94 This will spawn a VM using the network created previously manually.
95 In all the OPNFV tested scenarios from CI, it never has been a problem with the
96 previous actions. Further possible problems are explained in the following sections.
101 This test case creates a floating IP on the external network and assigns it to
102 the second instance **opnfv-vping-2**. The purpose of this is to establish
103 a SSH connection to that instance and SCP a script that will ping the first instance.
104 This script is located in the repository under
105 *$repos_dir/functest/testcases/vPing/CI/libraries/ping.sh* and takes an IP as
106 a parameter. When the SCP is completed, the test will do an SSH call to that script
107 inside the second instance. Some problems can happen here::
109 vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting
111 If this is displayed, stop the test or wait for it to finish (if you have used the flag
112 *-n* in **run_tests.sh** explained previously) so that the test does not clean
113 the OpenStack resources. It means that the Container can not reach the public
114 IP assigned to the instance **opnfv-vping-2**. There are many possible reasons, and
115 they really depend on the chosen scenario. For most of the ODL-L3 and ONOS scenarios
116 this has been noticed and it is a known limitation.
118 First, make sure that the instance **opnfv-vping-2** succeeded to get an IP from
119 the DHCP agent. It can be checked by doing::
121 nova console-log opnfv-vping-2
123 If the message *Sending discover* and *No lease, failing* is shown, it probably
124 means that the Neutron dhcp-agent failed to assign an IP or even that it was not
125 responding. At this point it does not make sense to try to ping the floating IP.
127 If the instance got an IP properly, try to ping manually the VM from the container::
133 If the ping does not return anything, try to ping from the Host where the Docker
134 container is running. If that solves the problem, check the iptable rules because
135 there might be some rules rejecting ICMP or TCP traffic coming/going from/to the container.
137 At this point, if the ping does not work either, try to reproduce the test
138 manually with the steps described above in the vPing common section with the addition::
140 neutron floatingip-create <EXT_NET_NAME>
141 nova floating-ip-associate nova-test <FLOATING_IP>
144 Further troubleshooting is out of scope of this document, as it might be due to
145 problems with the SDN controller. Contact the installer team members or send an
146 email to the corresponding OPNFV mailing list for more information.
152 This test case does not create any floating IP neither establishes an SSH
153 connection. Instead, it uses nova-metadata service when creating an instance
154 to pass the same script as before (ping.sh) but as 1-line text. This script
155 will be executed automatically when the second instance **opnfv-vping-2** is booted.
157 The only known problem here for this test to fail is mainly the lack of support
158 of cloud-init (nova-metadata service). Check the console of the instance::
160 nova console-log opnfv-vping-2
162 If this text or similar is shown::
164 checking http://169.254.169.254/2009-04-04/instance-id
165 failed 1/20: up 1.13. request failed
166 failed 2/20: up 13.18. request failed
167 failed 3/20: up 25.20. request failed
168 failed 4/20: up 37.23. request failed
169 failed 5/20: up 49.25. request failed
170 failed 6/20: up 61.27. request failed
171 failed 7/20: up 73.29. request failed
172 failed 8/20: up 85.32. request failed
173 failed 9/20: up 97.34. request failed
174 failed 10/20: up 109.36. request failed
175 failed 11/20: up 121.38. request failed
176 failed 12/20: up 133.40. request failed
177 failed 13/20: up 145.43. request failed
178 failed 14/20: up 157.45. request failed
179 failed 15/20: up 169.48. request failed
180 failed 16/20: up 181.50. request failed
181 failed 17/20: up 193.52. request failed
182 failed 18/20: up 205.54. request failed
183 failed 19/20: up 217.56. request failed
184 failed 20/20: up 229.58. request failed
185 failed to read iid from metadata. tried 20
187 it means that the instance failed to read from the metadata service. Contact
188 the Functest or installer teams for more information.
190 NOTE: Cloud-init in not supported on scenario dealing with ONOS and the tests have been
191 excluded from CI in those scenarios.
197 In the upstream OpenStack CI all the Tempest test cases are supposed to pass.
198 If some test cases fail in an OPNFV deployment, the reason is very probably one
201 +-----------------------------+------------------------------------------------+
203 +=============================+================================================+
204 | Resources required for test | Such resources could be e.g. an external |
205 | case execution are missing | network and access to the management subnet |
206 | | (adminURL) from the Functest docker container. |
207 +-----------------------------+------------------------------------------------+
208 | OpenStack components or | Check running services in the controller and |
209 | services are missing or not | compute nodes (e.g. with "systemctl" or |
210 | configured properly | "service" commands). Configuration parameters |
211 | | can be verified from related .conf files |
212 | | located under /etc/<component> directories. |
213 +-----------------------------+------------------------------------------------+
214 | Some resources required for | The tempest.conf file, automatically generated |
215 | execution test cases are | by Rally in Functest, does not contain all the |
216 | missing | needed parameters or some parameters are not |
218 | | The tempest.conf file is located in /home/opnfv|
219 | | /.rally/tempest/for-deployment-<UUID> in |
220 | | Functest container |
221 | | Use "rally deployment list" command in order to|
222 | | check UUID of current deployment. |
223 +-----------------------------+------------------------------------------------+
226 When some Tempest test case fails, captured traceback and possibly also related
227 REST API requests/responses are output to the console.
228 More detailed debug information can be found from tempest.log file stored into
229 related Rally deployment folder.
235 Same error causes than for Tempest mentioned above may lead to errors in Rally.
237 It is possible to run only one Rally scenario, instead of the whole suite.
238 To do that, call the python script (instead of *run_tests.sh*) as follows::
240 python $repos_dir/functest/testcases/VIM/OpenStack/CI/libraries/run_rally-cert.py -h
241 usage: run_rally-cert.py [-h] [-d] [-r] [-s] [-v] [-n] test_name
243 positional arguments:
244 test_name Module name to be tested. Possible values are : [
245 authenticate | glance | cinder | heat | keystone | neutron |
246 nova | quotas | requests | vm | all ] The 'all' value
247 performs all possible test scenarios
250 -h, --help show this help message and exit
251 -d, --debug Debug mode
252 -r, --report Create json result file
253 -s, --smoke Smoke test mode
254 -v, --verbose Print verbose info about the progress
255 -n, --noclean Don't clean the created resources for this test.
257 For example, to run the Glance scenario with debug information::
259 python $repos_dir/functest/testcases/VIM/OpenStack/CI/libraries/run_rally-cert.py -d glance
261 Possible scenarios are:
273 To know more about what those scenarios are doing, they are defined in:
274 *$repos_dir/functest/testcases/VIM/OpenStack/CI/suites*. For more info about
275 Rally scenario definition please refer to the Rally official documentation.
277 If the flag *all* is specified, it will run all the scenarios one by one. Please
278 note that this might take some time (~1,5hr), taking around 1 hour to complete the
281 To check any possible problems with rally, the logs are stored under
282 */home/opnfv/functest/results/rally/* in the Functest container.
290 2 versions are supported in Brahmaputra depending on the scenario:
294 The upstream test suites have not been adapted, so you may get 18 or 15 tests
295 passed on 18 depending on your configuration. The 3 testcases are partly failed
296 due to wrong return code.
301 Please refer to the ONOS documentation.
311 vIMS deployment may fail for several reasons, the most frequent ones are
312 described in the following table:
314 +-----------------------------------+------------------------------------+
316 +===================================+====================================+
317 | Keystone admin API not reachable | Impossible to create vIMS user and |
319 +-----------------------------------+------------------------------------+
320 | Impossible to retrieve admin role | Impossible to create vIMS user and |
322 +-----------------------------------+------------------------------------+
323 | Error when uploading image from | impossible to deploy VNF |
324 | OpenStack to glance | |
325 +-----------------------------------+------------------------------------+
326 | Cinder quota cannot be updated | Default quotas not sufficient, they|
327 | | are adapted in the script |
328 +-----------------------------------+------------------------------------+
329 | Impossible to create a volume | VNF cannot be deployed |
330 +-----------------------------------+------------------------------------+
331 | SSH connection issue between the | if vPing test fails, vIMS test will|
332 | Test container and the VM | fail... |
333 +-----------------------------------+------------------------------------+
334 | No Internet access from a VM | the VMs of the VNF must have an |
335 | | external access to Internet |
336 +-----------------------------------+------------------------------------+
342 Please refer to the Promise documentation.