1 .. SPDX-License-Identifier: CC-BY-4.0
6 This section gives some guidelines about how to troubleshoot the test cases
9 **IMPORTANT**: As in the previous section, the steps defined below must be
10 executed inside the Functest Docker container and after sourcing the OpenStack
17 source /home/opnfv/functest/conf/env_file
22 This section covers the test cases related to the VIM (healthcheck, vping_ssh,
23 vping_userdata, tempest_smoke, tempest_full, rally_sanity, rally_full).
27 For both vPing test cases (**vPing_ssh**, and **vPing_userdata**), the first
32 * Create Security Group
35 After these actions, the test cases differ and will be explained in their
38 These test cases can be run inside the container, using new Functest CLI as
41 $ run_tests -t vping_ssh
42 $ run_tests -t vping_userdata
44 The Functest CLI is designed to route a call to the corresponding internal
45 python scripts, located in paths::
47 /usr/lib/python3.7/site-packages/functest/opnfv_tests/openstack/vping/vping_ssh.py
48 /usr/lib/python3.7/site-packages/functest/opnfv_tests/openstack/vping/vping_userdata.py
52 #. There is one difference, between the Functest CLI based test case
53 execution compared to the earlier used Bash shell script, which is
54 relevant to point out in troubleshooting scenarios:
56 The Functest CLI does **not yet** support the option to suppress
57 clean-up of the generated OpenStack resources, following the execution
60 Explanation: After finishing the test execution, the corresponding
61 script will remove, by default, all created resources in OpenStack
62 (image, instances, network and security group). When troubleshooting,
63 it is advisable sometimes to keep those resources in case the test
64 fails and a manual testing is needed.
66 It is actually still possible to invoke test execution, with suppression
67 of OpenStack resource cleanup, however this requires invocation of a
68 **specific Python script:** 'run_tests'.
69 The `OPNFV Functest Developer Guide`_ provides guidance on the use of that
70 Python script in such troubleshooting cases.
72 Some of the common errors that can appear in this test case are::
74 vPing_ssh- ERROR - There has been a problem when creating the neutron network....
76 This means that there has been some problems with Neutron, even before creating
77 the instances. Try to create manually a Neutron network and a Subnet to see if
78 that works. The debug messages will also help to see when it failed (subnet and
79 router creation). Example of Neutron commands (using 10.6.0.0/24 range for
82 neutron net-create net-test
83 neutron subnet-create --name subnet-test --allocation-pool start=10.6.0.2,end=10.6.0.100 \
84 --gateway 10.6.0.254 net-test 10.6.0.0/24
85 neutron router-create test_router
86 neutron router-interface-add <ROUTER_ID> test_subnet
87 neutron router-gateway-set <ROUTER_ID> <EXT_NET_NAME>
89 Another related error can occur while creating the Security Groups for the
92 vPing_ssh- ERROR - Failed to create the security group...
94 In this case, proceed to create it manually. These are some hints::
96 neutron security-group-create sg-test
97 neutron security-group-rule-create sg-test --direction ingress --protocol icmp \
98 --remote-ip-prefix 0.0.0.0/0
99 neutron security-group-rule-create sg-test --direction ingress --ethertype IPv4 \
100 --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
101 neutron security-group-rule-create sg-test --direction egress --ethertype IPv4 \
102 --protocol tcp --port-range-min 80 --port-range-max 80 --remote-ip-prefix 0.0.0.0/0
104 The next step is to create the instances. The image used is located in
105 */home/opnfv/functest/data/cirros-0.4.0-x86_64-disk.img* and a Glance image is
106 created with the name **functest-vping**. If booting the instances fails (i.e.
107 the status is not **ACTIVE**), you can check why it failed by doing::
110 nova show <INSTANCE_ID>
112 It might show some messages about the booting failure. To try that manually::
114 nova boot --flavor m1.small --image functest-vping --nic net-id=<NET_ID> nova-test
116 This will spawn a VM using the network created previously manually.
117 In all the OPNFV tested scenarios from CI, it never has been a problem with the
118 previous actions. Further possible problems are explained in the following
124 This test case creates a floating IP on the external network and assigns it to
125 the second instance **opnfv-vping-2**. The purpose of this is to establish
126 a SSH connection to that instance and SCP a script that will ping the first
127 instance. This script is located in the repository under
128 /usr/lib/python3.7/site-packages/functest/opnfv_tests/openstack/vping/ping.sh
129 and takes an IP as a parameter. When the SCP is completed, the test will do a
130 SSH call to that script inside the second instance. Some problems can happen
133 vPing_ssh- ERROR - Cannot establish connection to IP xxx.xxx.xxx.xxx. Aborting
135 If this is displayed, stop the test or wait for it to finish, if you have used
136 the special method of test invocation with specific supression of OpenStack
137 resource clean-up, as explained earler. It means that the Container can not
138 reach the Public/External IP assigned to the instance **opnfv-vping-2**. There
139 are many possible reasons, and they really depend on the chosen scenario. For
140 most of the ODL-L3 and ONOS scenarios this has been noticed and it is a known
143 First, make sure that the instance **opnfv-vping-2** succeeded to get an IP
144 from the DHCP agent. It can be checked by doing::
146 nova console-log opnfv-vping-2
148 If the message *Sending discover* and *No lease, failing* is shown, it probably
149 means that the Neutron dhcp-agent failed to assign an IP or even that it was
150 not responding. At this point it does not make sense to try to ping the
153 If the instance got an IP properly, try to ping manually the VM from the
160 If the ping does not return anything, try to ping from the Host where the
161 Docker container is running. If that solves the problem, check the iptable
162 rules because there might be some rules rejecting ICMP or TCP traffic
163 coming/going from/to the container.
165 At this point, if the ping does not work either, try to reproduce the test
166 manually with the steps described above in the vPing common section with the
169 neutron floatingip-create <EXT_NET_NAME>
170 nova floating-ip-associate nova-test <FLOATING_IP>
173 Further troubleshooting is out of scope of this document, as it might be due to
174 problems with the SDN controller. Contact the installer team members or send an
175 email to the corresponding OPNFV mailing list for more information.
181 This test case does not create any floating IP neither establishes an SSH
182 connection. Instead, it uses nova-metadata service when creating an instance
183 to pass the same script as before (ping.sh) but as 1-line text. This script
184 will be executed automatically when the second instance **opnfv-vping-2** is
187 The only known problem here for this test to fail is mainly the lack of support
188 of cloud-init (nova-metadata service). Check the console of the instance::
190 nova console-log opnfv-vping-2
192 If this text or similar is shown::
194 checking http://169.254.169.254/2009-04-04/instance-id
195 failed 1/20: up 1.13. request failed
196 failed 2/20: up 13.18. request failed
197 failed 3/20: up 25.20. request failed
198 failed 4/20: up 37.23. request failed
199 failed 5/20: up 49.25. request failed
200 failed 6/20: up 61.27. request failed
201 failed 7/20: up 73.29. request failed
202 failed 8/20: up 85.32. request failed
203 failed 9/20: up 97.34. request failed
204 failed 10/20: up 109.36. request failed
205 failed 11/20: up 121.38. request failed
206 failed 12/20: up 133.40. request failed
207 failed 13/20: up 145.43. request failed
208 failed 14/20: up 157.45. request failed
209 failed 15/20: up 169.48. request failed
210 failed 16/20: up 181.50. request failed
211 failed 17/20: up 193.52. request failed
212 failed 18/20: up 205.54. request failed
213 failed 19/20: up 217.56. request failed
214 failed 20/20: up 229.58. request failed
215 failed to read iid from metadata. tried 20
217 it means that the instance failed to read from the metadata service. Contact
218 the Functest or installer teams for more information.
224 In the upstream OpenStack CI all the Tempest test cases are supposed to pass.
225 If some test cases fail in an OPNFV deployment, the reason is very probably one
228 +----------------------------+------------------------------------------------+
230 +============================+================================================+
231 | Resources required for | Such resources could be e.g. an external |
232 | testcase execution are | network and access to the management subnet |
233 | missing | (adminURL) from the Functest docker container. |
234 +----------------------------+------------------------------------------------+
235 | OpenStack components or | Check running services in the controller and |
236 | services are missing or | compute nodes (e.g. with "systemctl" or |
237 | not configured properly | "service" commands). |
238 | | Configuration parameters can be verified from |
239 | | the related .conf files located under |
240 | | '/etc/<component>' directories. |
241 +----------------------------+------------------------------------------------+
242 | Some resources required | The tempest.conf file, automatically generated |
243 | for execution test cases | by Rally in Functest, does not contain all the |
244 | are missing | needed parameters or some parameters are not |
246 | | The tempest.conf file is located in directory |
247 | | 'root/.rally/verification/verifier-<UUID> |
248 | | /for-deployment-<UUID>' |
249 | | in the Functest Docker container. Use the |
250 | | "rally deployment list" command in order to |
251 | | check the UUID of the current deployment. |
252 +----------------------------+------------------------------------------------+
255 When some Tempest test case fails, captured traceback and possibly also the
256 related REST API requests/responses are output to the console. More detailed
257 debug information can be found from tempest.log file stored into related Rally
260 Functest offers a possibility to test a customized list of Tempest test cases.
261 To enable that, add a new entry in docker/smoke/testcases.yaml on the
262 "smoke" container with the following content::
265 case_name: tempest_custom
266 project_name: functest
270 The test case allows running a customized list of tempest
276 module: 'functest.opnfv_tests.openstack.tempest.tempest'
277 class: 'TempestCustom'
279 Also, a list of the Tempest test cases must be provided to the container or
280 modify the existing one in
281 /usr/lib/python3.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt
283 Example of custom list of tests 'my-custom-tempest-tests.txt'::
285 tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke]
286 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke]
288 This is an example of running a customized list of Tempest tests in Functest::
290 sudo docker run --env-file env \
291 -v $(pwd)/openstack.creds:/home/opnfv/functest/conf/env_file \
292 -v $(pwd)/images:/home/opnfv/functest/images \
293 -v $(pwd)/my-custom-testcases.yaml:/usr/lib/python3.7/site-packages/functest/ci/testcases.yaml \
294 -v $(pwd)/my-custom-tempest-tests.txt:/usr/lib/python3.7/site-packages/functest/opnfv_tests/openstack/tempest/custom_tests/test_list.txt \
295 opnfv/functest-smoke run_tests -t tempest_custom
301 The same error causes which were mentioned above for Tempest test cases, may
302 also lead to errors in Rally as well.
304 Possible scenarios are:
315 To know more about what those scenarios are doing, they are defined in
317 /usr/lib/python3.7/site-packages/functest/opnfv_tests/openstack/rally/scenario
318 For more info about Rally scenario definition please refer to the Rally
319 official documentation. `[3]`_
321 To check any possible problems with Rally, the logs are stored under
322 */home/opnfv/functest/results/rally/* in the Functest Docker container.
324 .. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
332 If the Basic Restconf test suite fails, check that the ODL controller is
333 reachable and its Restconf module has been installed.
335 If the Neutron Reachability test fails, verify that the modules
336 implementing Neutron requirements have been properly installed.
338 If any of the other test cases fails, check that Neutron and ODL have
339 been correctly configured to work together. Check Neutron configuration
340 files, accounts, IP addresses etc.).
346 Please refer to the dedicated feature user guides for details.
354 vIMS deployment may fail for several reasons, the most frequent ones are
355 described in the following table:
357 +-----------------------------------+------------------------------------+
359 +===================================+====================================+
360 | Keystone admin API not reachable | Impossible to create vIMS user and |
362 +-----------------------------------+------------------------------------+
363 | Impossible to retrieve admin role | Impossible to create vIMS user and |
365 +-----------------------------------+------------------------------------+
366 | Error when uploading image from | impossible to deploy VNF |
367 | OpenStack to glance | |
368 +-----------------------------------+------------------------------------+
369 | Cinder quota cannot be updated | Default quotas not sufficient, they|
370 | | are adapted in the script |
371 +-----------------------------------+------------------------------------+
372 | Impossible to create a volume | VNF cannot be deployed |
373 +-----------------------------------+------------------------------------+
374 | SSH connection issue between the | if vPing test fails, vIMS test will|
375 | Test Docker container and the VM | fail... |
376 +-----------------------------------+------------------------------------+
377 | No Internet access from the VM | the VMs of the VNF must have an |
378 | | external access to Internet |
379 +-----------------------------------+------------------------------------+
380 | No access to OpenStack API from | Orchestrator can be installed but |
381 | the VM | the vIMS VNF installation fails |
382 +-----------------------------------+------------------------------------+
384 Please note that this test case requires resources (8 VM (2Go) + 1 VM (4Go)),
385 it is there fore not recommended to run it on a light configuration.
387 .. _`OPNFV Functest Developer Guide`: http://artifacts.opnfv.org/functest/docs/testing_developer_devguide/index.html#