1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
4 =========================
5 OPNFV FUNCTEST user guide
6 =========================
13 +------------+----------+------------------+----------------------------------+
14 | **Date** | **Ver.** | **Author** | **Comment** |
16 +------------+----------+------------------+----------------------------------+
17 | 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release |
18 | | | Column Gaynor | |
19 +------------+----------+------------------+----------------------------------+
20 | 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube |
23 +------------+----------+------------------+----------------------------------+
29 The goal of this document is to describe the OPNFV Functest test cases and to
30 provide a procedure to execute them. In the OPNFV Danube system release,
31 a Functest CLI utility is introduced for an easier execution of test procedures.
33 **IMPORTANT**: It is assumed here that the Functest Docker container is already
34 properly deployed and that all instructions described in this guide are to be
35 performed from *inside* the deployed Functest Docker container.
37 .. include:: ./introduction.rst
39 The different test cases are described in the remaining sections of this document.
41 VIM (Virtualized Infrastructure Manager)
42 ----------------------------------------
46 In Danube, healthcheck tests have been refactored and rely on SNAPS, a
47 OPNFV middleware project.
49 SNAPS stands for "SDN/NFV Application development Platform and Stack".
50 SNAPS is an object-oriented OpenStack library packaged with tests that exercise
52 More information on SNAPS can be found in `[13]`_
54 Three tests are declared as healthcheck tests and can be used for gating by the
55 installer, they cover functionally the tests previously done by healthcheck
63 * *snaps_health_check*
65 Connection_check consists in 9 test cases (test duration < 5s) checking the
66 connectivity with Glance, Keystone, Neutron, Nova and the external network.
68 Api_check verifies the retrieval of OpenStack clients: Keystone, Glance,
69 Neutron and Nova and may perform some simple queries. When the config value of
70 snaps.use_keystone is True, functest must have access to the cloud's private
71 network. This suite consists in 49 tests (test duration < 2 minutes).
73 snaps_health_check creates instance, allocate floating IP, connect to the VM.
74 This test replaced the previous Colorado healthcheck test.
76 Self-obviously, successful completion of the 'healthcheck' testcase is a
77 necessary pre-requisite for the execution of all other test Tiers.
83 Given the script **ping.sh**::
87 ping -c 1 $1 2>&1 >/dev/null
89 if [ "Z$RES" = "Z0" ] ; then
99 The goal of this test is to establish an SSH connection using a floating IP
100 on the Public/External network and verify that 2 instances can talk over a Private
104 +-------------+ +-------------+
106 | | Boot VM1 with IP1 | |
107 | +------------------->| |
108 | Tester | | System |
109 | | Boot VM2 | Under |
110 | +------------------->| Test |
112 | | Create floating IP | |
113 | +------------------->| |
115 | | Assign floating IP | |
117 | +------------------->| |
119 | | Establish SSH | |
120 | | connection to VM2 | |
121 | | through floating IP| |
122 | +------------------->| |
124 | | SCP ping.sh to VM2 | |
125 | +------------------->| |
128 | | ping.sh to VM1 | |
129 | +------------------->| |
133 | | else (timeout): | |
136 +-------------+ +-------------+
138 This test can be considered as an "Hello World" example.
139 It is the first basic use case which **must** work on any deployment.
144 This test case is similar to vPing_ssh but without the use of Floating IPs
145 and the Public/External network to transfer the ping script.
146 Instead, it uses Nova metadata service to pass it to the instance at booting time.
147 As vPing_ssh, it checks that 2 instances can talk to
148 each other on a Private Tenant network::
150 vPing_userdata test case
151 +-------------+ +-------------+
153 | | Boot VM1 with IP1 | |
154 | +------------------->| |
156 | | Boot VM2 with | |
157 | | ping.sh as userdata| |
158 | | with IP1 as $1. | |
159 | +------------------->| |
160 | Tester | | System |
161 | | VM2 exeutes ping.sh| Under |
162 | | (ping IP1) | Test |
163 | +------------------->| |
166 | | console-log VM 2 | |
169 | | else (timeout) | |
172 +-------------+ +-------------+
174 When the second VM boots it will execute the script passed as userdata
175 automatically. The ping will be detected by periodically capturing the output
176 in the console-log of the second VM.
182 Tempest `[2]`_ is the reference OpenStack Integration test suite.
183 It is a set of integration tests to be run against a live OpenStack cluster.
184 Tempest has suites of tests for:
186 * OpenStack API validation
188 * Other specific tests useful in validating an OpenStack deployment
190 Functest uses Rally `[3]`_ to run the Tempest suite.
191 Rally generates automatically the Tempest configuration file **tempest.conf**.
192 Before running the actual test cases,
193 Functest creates the needed resources (user, tenant) and
194 updates the appropriate parameters into the configuration file.
196 When the Tempest suite is executed, each test duration is measured and the full
197 console output is stored to a *log* file for further analysis.
199 The Tempest testcases are distributed accross two
202 * Smoke Tier - Test Case 'tempest_smoke_serial'
203 * Components Tier - Test case 'tempest_full_parallel'
205 NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
206 tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
207 executes all defined Tempest tests using several concurrent threads
208 (i.e. parallel mode). The number of threads activated corresponds to the number
209 of available logical CPUs.
211 The goal of the Tempest test suite is to check the basic functionalities of the
212 different OpenStack components on an OPNFV fresh installation, using the
213 corresponding REST API interfaces.
216 Rally bench test suites
217 ^^^^^^^^^^^^^^^^^^^^^^^
219 Rally `[3]`_ is a benchmarking tool that answers the question:
221 *How does OpenStack work at scale?*
223 The goal of this test suite is to benchmark all the different OpenStack modules and
224 get significant figures that could help to define Telco Cloud KPIs.
226 The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
238 A basic SLA (stop test on errors) has been implemented.
240 The Rally testcases are distributed accross two Tiers:
242 * Smoke Tier - Test Case 'rally_sanity'
243 * Components Tier - Test case 'rally_full'
245 NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
246 cases. Test case 'rally_full' executes the full defined set of Rally tests.
251 This test case contains tests that setup and destroy environments with VMs with
252 and without Floating IPs with a newly created user and project. Set the config
253 value snaps.use_floating_ips (True|False) to toggle this functionality. When
254 the config value of snaps.use_keystone is True, functest must have access
255 the cloud's private network.
256 This suite consists in 38 tests (test duration < 10 minutes)
262 There are currently 3 available controllers:
271 The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
272 from the ODL project using the Robot `[11]`_ framework.
273 The suite verifies creation and deletion of networks, subnets and ports with
274 OpenDaylight and Neutron.
276 The list of tests can be described as follows:
278 * Basic Restconf test cases
279 * Connect to Restconf URL
280 * Check the HTTP code status
282 * Neutron Reachability test cases
283 * Get the complete list of neutron resources (networks, subnets, ports)
285 * Neutron Network test cases
286 * Check OpenStack networks
287 * Check OpenDaylight networks
288 * Create a new network via OpenStack and check the HTTP status code returned by Neutron
289 * Check that the network has also been successfully created in OpenDaylight
291 * Neutron Subnet test cases
292 * Check OpenStack subnets
293 * Check OpenDaylight subnets
294 * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
295 * Check that the subnet has also been successfully created in OpenDaylight
297 * Neutron Port test cases
298 * Check OpenStack Neutron for known ports
299 * Check OpenDaylight ports
300 * Create a new port via OpenStack and check the HTTP status code returned by Neutron
301 * Check that the new port has also been successfully created in OpenDaylight
304 * Delete the port previously created via OpenStack
305 * Check that the port has been also succesfully deleted in OpenDaylight
306 * Delete previously subnet created via OpenStack
307 * Check that the subnet has also been successfully deleted in OpenDaylight
308 * Delete the network created via OpenStack
309 * Check that the network has also been succesfully deleted in OpenDaylight
311 Note: the checks in OpenDaylight are based on the returned HTTP status
312 code returned by OpenDaylight.
318 TestON Framework is used to test the ONOS SDN controller functions.
319 The test cases deal with L2 and L3 functions.
320 The ONOS test suite can be run on any ONOS compliant scenario.
322 The test cases are described as follows:
324 * onosfunctest: The main executable file contains the initialization of
325 the docker environment and functions called by FUNCvirNetNB and
330 * Create Network: Post Network data and check it in ONOS
331 * Update Network: Update the Network and compare it in ONOS
332 * Delete Network: Delete the Network and check if it's NULL in ONOS or
334 * Create Subnet: Post Subnet data and check it in ONOS
335 * Update Subnet: Update the Subnet and compare it in ONOS
336 * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not
337 * Create Port: Post Port data and check it in ONOS
338 * Update Port: Update the Port and compare it in ONOS
339 * Delete Port: Delete the Port and check if it's NULL in ONOS or not
343 * Create Router: Post data for create Router and check it in ONOS
344 * Update Router: Update the Router and compare it in ONOS
345 * Delete Router: Delete the Router data and check it in ONOS
346 * Create RouterInterface: Post Router Interface data to an existing Router
348 * Delete RouterInterface: Delete the RouterInterface and check the Router
349 * Create FloatingIp: Post data for create FloatingIp and check it in ONOS
350 * Update FloatingIp: Update the FloatingIp and compare it in ONOS
351 * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in
353 * Create External Gateway: Post data to create an External Gateway for an
354 existing Router and check it in ONOS
355 * Update External Gateway: Update the External Gateway and compare the change
356 * Delete External Gateway: Delete the External Gateway and check that it is
363 In Danube, functest supports the integration of:
376 Note: copper is not supported in Danube.
378 Please refer to the dedicated feature user guides for details.
387 The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
388 architectural framework for delivering IP multimedia services.
390 vIMS has been integrated in Functest to demonstrate the capability to deploy a
391 relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
392 functional VNF allows the test of most of the essential functions needed for a
395 The goal of this test suite consists of:
397 * deploy a VNF orchestrator (Cloudify)
398 * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
399 orchestrator based on a TOSCA blueprint defined in `[5]`_
400 * run suite of signaling tests on top of this VNF
402 The Clearwater architecture is described as follows:
404 .. figure:: ../../../images/clearwater-architecture.png
406 :alt: vIMS architecture
410 Orchestra test case deals with the deployment of OpenIMS with OpenBaton
416 See parser user guide for details: `[12]`_
422 This test case deals with the deployment and the test of vyos vrouter with
423 Cloudify orchestrator. The test case can do testing for interchangeability of
424 BGP Protocol using vyos.
426 The Workflow is as follows:
428 Deploy VNF Testing topology by Cloudify using blueprint.
430 Setting configuration to Target VNF and reference VNF using ssh
432 Execution of test command for test item written YAML format file.
433 Check VNF status and behavior.
435 Output of report based on result using JSON format.
437 The vyos-vrouter architecture is described in `[14]`_
441 .. include:: ./runfunctest.rst
450 In manual mode test results are displayed in the console and result files
451 are put in /home/opnfv/functest/results.
456 In automated mode, test results are displayed in jenkins logs, a summary is provided
457 at the end of the job and can be described as follow::
459 +==================================================================================================================================================+
461 +==================================================================================================================================================+
463 | Deployment description: |
465 | SCENARIO: os-odl_l2-nofeature-ha |
466 | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 |
469 +=========================+===============+============+===============+===========================================================================+
470 | TEST CASE | TIER | DURATION | RESULT | URL |
471 +=========================+===============+============+===============+===========================================================================+
472 | connection_check | healthcheck | 00:02 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb62b34079ac000a42e3fe |
473 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
474 | api_check | healthcheck | 01:15 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb62fe4079ac000a42e3ff |
475 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
476 | snaps_health_check | healthcheck | 00:50 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63314079ac000a42e400 |
477 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
478 | vping_ssh | smoke | 01:10 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63654079ac000a42e401 |
479 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
480 | vping_userdata | smoke | 00:59 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63a14079ac000a42e403 |
481 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
482 | tempest_smoke_serial | smoke | 12:57 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb66bd4079ac000a42e408 |
483 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
484 | rally_sanity | smoke | 10:22 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb692b4079ac000a42e40a |
485 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
486 | refstack_defcore | smoke | 12:28 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6c184079ac000a42e40c |
487 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
488 | snaps_smoke | smoke | 12:04 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6eec4079ac000a42e40e |
489 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
490 | domino | features | 00:29 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6f044079ac000a42e40f |
491 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
494 Results are automatically pushed to the test results database, some additional
495 result files are pushed to OPNFV artifact web sites.
497 Based on the results stored in the result database, a `Functest reporting`_
498 portal is also automatically updated. This portal provides information on:
500 * The overall status per scenario and per installer
501 * Tempest: Tempest test case including reported errors per scenario and installer
502 * vIMS: vIMS details per scenario and installer
504 .. figure:: ../../../images/functest-reporting-status.png
506 :alt: Functest reporting portal Fuel status page
508 .. include:: ./troubleshooting.rst
514 .. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/#
515 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
516 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
517 .. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf
518 .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
519 .. _`[6]`: https://scap.nist.gov/
520 .. _`[7]`: https://github.com/OpenSCAP/openscap
521 .. _`[11]`: http://robotframework.org/
522 .. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
523 .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
524 .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
530 IRC support chan: #opnfv-testperf
532 .. _`OPNFV main site`: http://www.opnfv.org
533 .. _`Functest page`: https://wiki.opnfv.org/functest
534 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
535 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
536 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml
537 .. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html