1 .. _functest-userguide:
3 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
4 .. SPDX-License-Identifier: CC-BY-4.0
6 =========================
7 OPNFV FUNCTEST user guide
8 =========================
15 +------------+----------+------------------+----------------------------------+
16 | **Date** | **Ver.** | **Author** | **Comment** |
18 +------------+----------+------------------+----------------------------------+
19 | 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release |
20 | | | Column Gaynor | |
21 +------------+----------+------------------+----------------------------------+
22 | 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube |
25 +------------+----------+------------------+----------------------------------+
31 The goal of this document is to describe the OPNFV Functest test cases and to
32 provide a procedure to execute them. In the OPNFV Danube system release,
33 a Functest CLI utility is introduced for an easier execution of test procedures.
35 **IMPORTANT**: It is assumed here that the Functest Docker container is already
36 properly deployed and that all instructions described in this guide are to be
37 performed from *inside* the deployed Functest Docker container.
39 .. include:: ./introduction.rst
41 The different test cases are described in the remaining sections of this document.
43 VIM (Virtualized Infrastructure Manager)
44 ----------------------------------------
48 In Danube, healthcheck tests have been refactored and rely on SNAPS, a
49 OPNFV middleware project.
51 SNAPS stands for "SDN/NFV Application development Platform and Stack".
52 SNAPS is an object-oriented OpenStack library packaged with tests that exercise
54 More information on SNAPS can be found in `[13]`_
56 Three tests are declared as healthcheck tests and can be used for gating by the
57 installer, they cover functionally the tests previously done by healthcheck
65 * *snaps_health_check*
67 Connection_check consists in 9 test cases (test duration < 5s) checking the
68 connectivity with Glance, Keystone, Neutron, Nova and the external network.
70 Api_check verifies the retrieval of OpenStack clients: Keystone, Glance,
71 Neutron and Nova and may perform some simple queries. When the config value of
72 snaps.use_keystone is True, functest must have access to the cloud's private
73 network. This suite consists in 49 tests (test duration < 2 minutes).
75 snaps_health_check creates instance, allocate floating IP, connect to the VM.
76 This test replaced the previous Colorado healthcheck test.
78 Self-obviously, successful completion of the 'healthcheck' testcase is a
79 necessary pre-requisite for the execution of all other test Tiers.
85 Given the script **ping.sh**::
89 ping -c 1 $1 2>&1 >/dev/null
91 if [ "Z$RES" = "Z0" ] ; then
101 The goal of this test is to establish an SSH connection using a floating IP
102 on the Public/External network and verify that 2 instances can talk over a Private
106 +-------------+ +-------------+
108 | | Boot VM1 with IP1 | |
109 | +------------------->| |
110 | Tester | | System |
111 | | Boot VM2 | Under |
112 | +------------------->| Test |
114 | | Create floating IP | |
115 | +------------------->| |
117 | | Assign floating IP | |
119 | +------------------->| |
121 | | Establish SSH | |
122 | | connection to VM2 | |
123 | | through floating IP| |
124 | +------------------->| |
126 | | SCP ping.sh to VM2 | |
127 | +------------------->| |
130 | | ping.sh to VM1 | |
131 | +------------------->| |
135 | | else (timeout): | |
138 +-------------+ +-------------+
140 This test can be considered as an "Hello World" example.
141 It is the first basic use case which **must** work on any deployment.
146 This test case is similar to vPing_ssh but without the use of Floating IPs
147 and the Public/External network to transfer the ping script.
148 Instead, it uses Nova metadata service to pass it to the instance at booting time.
149 As vPing_ssh, it checks that 2 instances can talk to
150 each other on a Private Tenant network::
152 vPing_userdata test case
153 +-------------+ +-------------+
155 | | Boot VM1 with IP1 | |
156 | +------------------->| |
158 | | Boot VM2 with | |
159 | | ping.sh as userdata| |
160 | | with IP1 as $1. | |
161 | +------------------->| |
162 | Tester | | System |
163 | | VM2 exeutes ping.sh| Under |
164 | | (ping IP1) | Test |
165 | +------------------->| |
168 | | console-log VM 2 | |
171 | | else (timeout) | |
174 +-------------+ +-------------+
176 When the second VM boots it will execute the script passed as userdata
177 automatically. The ping will be detected by periodically capturing the output
178 in the console-log of the second VM.
184 Tempest `[2]`_ is the reference OpenStack Integration test suite.
185 It is a set of integration tests to be run against a live OpenStack cluster.
186 Tempest has suites of tests for:
188 * OpenStack API validation
190 * Other specific tests useful in validating an OpenStack deployment
192 Functest uses Rally `[3]`_ to run the Tempest suite.
193 Rally generates automatically the Tempest configuration file **tempest.conf**.
194 Before running the actual test cases,
195 Functest creates the needed resources (user, tenant) and
196 updates the appropriate parameters into the configuration file.
198 When the Tempest suite is executed, each test duration is measured and the full
199 console output is stored to a *log* file for further analysis.
201 The Tempest testcases are distributed accross two
204 * Smoke Tier - Test Case 'tempest_smoke_serial'
205 * Components Tier - Test case 'tempest_full_parallel'
207 NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
208 tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
209 executes all defined Tempest tests using several concurrent threads
210 (i.e. parallel mode). The number of threads activated corresponds to the number
211 of available logical CPUs.
213 The goal of the Tempest test suite is to check the basic functionalities of the
214 different OpenStack components on an OPNFV fresh installation, using the
215 corresponding REST API interfaces.
218 Rally bench test suites
219 ^^^^^^^^^^^^^^^^^^^^^^^
221 Rally `[3]`_ is a benchmarking tool that answers the question:
223 *How does OpenStack work at scale?*
225 The goal of this test suite is to benchmark all the different OpenStack modules and
226 get significant figures that could help to define Telco Cloud KPIs.
228 The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
240 A basic SLA (stop test on errors) has been implemented.
242 The Rally testcases are distributed accross two Tiers:
244 * Smoke Tier - Test Case 'rally_sanity'
245 * Components Tier - Test case 'rally_full'
247 NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
248 cases. Test case 'rally_full' executes the full defined set of Rally tests.
253 This test case contains tests that setup and destroy environments with VMs with
254 and without Floating IPs with a newly created user and project. Set the config
255 value snaps.use_floating_ips (True|False) to toggle this functionality. When
256 the config value of snaps.use_keystone is True, functest must have access
257 the cloud's private network.
258 This suite consists in 38 tests (test duration < 10 minutes)
264 There are currently 3 available controllers:
273 The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
274 from the ODL project using the Robot `[11]`_ framework.
275 The suite verifies creation and deletion of networks, subnets and ports with
276 OpenDaylight and Neutron.
278 The list of tests can be described as follows:
280 * Basic Restconf test cases
281 * Connect to Restconf URL
282 * Check the HTTP code status
284 * Neutron Reachability test cases
285 * Get the complete list of neutron resources (networks, subnets, ports)
287 * Neutron Network test cases
288 * Check OpenStack networks
289 * Check OpenDaylight networks
290 * Create a new network via OpenStack and check the HTTP status code returned by Neutron
291 * Check that the network has also been successfully created in OpenDaylight
293 * Neutron Subnet test cases
294 * Check OpenStack subnets
295 * Check OpenDaylight subnets
296 * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
297 * Check that the subnet has also been successfully created in OpenDaylight
299 * Neutron Port test cases
300 * Check OpenStack Neutron for known ports
301 * Check OpenDaylight ports
302 * Create a new port via OpenStack and check the HTTP status code returned by Neutron
303 * Check that the new port has also been successfully created in OpenDaylight
306 * Delete the port previously created via OpenStack
307 * Check that the port has been also succesfully deleted in OpenDaylight
308 * Delete previously subnet created via OpenStack
309 * Check that the subnet has also been successfully deleted in OpenDaylight
310 * Delete the network created via OpenStack
311 * Check that the network has also been succesfully deleted in OpenDaylight
313 Note: the checks in OpenDaylight are based on the returned HTTP status
314 code returned by OpenDaylight.
320 TestON Framework is used to test the ONOS SDN controller functions.
321 The test cases deal with L2 and L3 functions.
322 The ONOS test suite can be run on any ONOS compliant scenario.
324 The test cases are described as follows:
326 * onosfunctest: The main executable file contains the initialization of
327 the docker environment and functions called by FUNCvirNetNB and
332 * Create Network: Post Network data and check it in ONOS
333 * Update Network: Update the Network and compare it in ONOS
334 * Delete Network: Delete the Network and check if it's NULL in ONOS or
336 * Create Subnet: Post Subnet data and check it in ONOS
337 * Update Subnet: Update the Subnet and compare it in ONOS
338 * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not
339 * Create Port: Post Port data and check it in ONOS
340 * Update Port: Update the Port and compare it in ONOS
341 * Delete Port: Delete the Port and check if it's NULL in ONOS or not
345 * Create Router: Post data for create Router and check it in ONOS
346 * Update Router: Update the Router and compare it in ONOS
347 * Delete Router: Delete the Router data and check it in ONOS
348 * Create RouterInterface: Post Router Interface data to an existing Router
350 * Delete RouterInterface: Delete the RouterInterface and check the Router
351 * Create FloatingIp: Post data for create FloatingIp and check it in ONOS
352 * Update FloatingIp: Update the FloatingIp and compare it in ONOS
353 * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in
355 * Create External Gateway: Post data to create an External Gateway for an
356 existing Router and check it in ONOS
357 * Update External Gateway: Update the External Gateway and compare the change
358 * Delete External Gateway: Delete the External Gateway and check that it is
365 In Danube, functest supports the integration of:
378 Note: copper is not supported in Danube.
380 Please refer to the dedicated feature user guides for details.
389 The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
390 architectural framework for delivering IP multimedia services.
392 vIMS has been integrated in Functest to demonstrate the capability to deploy a
393 relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
394 functional VNF allows the test of most of the essential functions needed for a
397 The goal of this test suite consists of:
399 * deploy a VNF orchestrator (Cloudify)
400 * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
401 orchestrator based on a TOSCA blueprint defined in `[5]`_
402 * run suite of signaling tests on top of this VNF
404 The Clearwater architecture is described as follows:
406 .. figure:: ../../../images/clearwater-architecture.png
408 :alt: vIMS architecture
412 Orchestra test case deals with the deployment of OpenIMS with OpenBaton
418 See parser user guide for details: `[12]`_
424 This test case deals with the deployment and the test of vyos vrouter with
425 Cloudify orchestrator. The test case can do testing for interchangeability of
426 BGP Protocol using vyos.
428 The Workflow is as follows:
430 Deploy VNF Testing topology by Cloudify using blueprint.
432 Setting configuration to Target VNF and reference VNF using ssh
434 Execution of test command for test item written YAML format file.
435 Check VNF status and behavior.
437 Output of report based on result using JSON format.
439 The vyos-vrouter architecture is described in `[14]`_
443 .. include:: ./runfunctest.rst
452 In manual mode test results are displayed in the console and result files
453 are put in /home/opnfv/functest/results.
458 In automated mode, test results are displayed in jenkins logs, a summary is provided
459 at the end of the job and can be described as follow::
461 +==================================================================================================================================================+
463 +==================================================================================================================================================+
465 | Deployment description: |
467 | SCENARIO: os-odl_l2-nofeature-ha |
468 | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 |
471 +=========================+===============+============+===============+===========================================================================+
472 | TEST CASE | TIER | DURATION | RESULT | URL |
473 +=========================+===============+============+===============+===========================================================================+
474 | connection_check | healthcheck | 00:02 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb62b34079ac000a42e3fe |
475 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
476 | api_check | healthcheck | 01:15 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb62fe4079ac000a42e3ff |
477 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
478 | snaps_health_check | healthcheck | 00:50 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63314079ac000a42e400 |
479 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
480 | vping_ssh | smoke | 01:10 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63654079ac000a42e401 |
481 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
482 | vping_userdata | smoke | 00:59 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63a14079ac000a42e403 |
483 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
484 | tempest_smoke_serial | smoke | 12:57 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb66bd4079ac000a42e408 |
485 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
486 | rally_sanity | smoke | 10:22 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb692b4079ac000a42e40a |
487 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
488 | refstack_defcore | smoke | 12:28 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6c184079ac000a42e40c |
489 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
490 | snaps_smoke | smoke | 12:04 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6eec4079ac000a42e40e |
491 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
492 | domino | features | 00:29 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6f044079ac000a42e40f |
493 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
496 Results are automatically pushed to the test results database, some additional
497 result files are pushed to OPNFV artifact web sites.
499 Based on the results stored in the result database, a `Functest reporting`_
500 portal is also automatically updated. This portal provides information on:
502 * The overall status per scenario and per installer
503 * Tempest: Tempest test case including reported errors per scenario and installer
504 * vIMS: vIMS details per scenario and installer
506 .. figure:: ../../../images/functest-reporting-status.png
508 :alt: Functest reporting portal Fuel status page
510 .. include:: ./troubleshooting.rst
516 .. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/#
517 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
518 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
519 .. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf
520 .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
521 .. _`[6]`: https://scap.nist.gov/
522 .. _`[7]`: https://github.com/OpenSCAP/openscap
523 .. _`[11]`: http://robotframework.org/
524 .. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
525 .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
526 .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
532 IRC support chan: #opnfv-testperf
534 .. _`OPNFV main site`: http://www.opnfv.org
535 .. _`Functest page`: https://wiki.opnfv.org/functest
536 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
537 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
538 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml
539 .. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html