1 .. _functest-userguide:
3 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
4 .. SPDX-License-Identifier: CC-BY-4.0
6 =========================
7 OPNFV FUNCTEST user guide
8 =========================
15 +------------+----------+------------------+----------------------------------+
16 | **Date** | **Ver.** | **Author** | **Comment** |
18 +------------+----------+------------------+----------------------------------+
19 | 2016-08-17 | 1.0.0 | Juha Haapavirta | Colorado release |
20 | | | Column Gaynor | |
21 +------------+----------+------------------+----------------------------------+
22 | 2017-01-23 | 1.0.1 | Morgan Richomme | Adaptations for Danube |
25 +------------+----------+------------------+----------------------------------+
31 The goal of this document is to describe the OPNFV Functest test cases and to
32 provide a procedure to execute them. In the OPNFV Danube system release,
33 a Functest CLI utility is introduced for an easier execution of test procedures.
35 **IMPORTANT**: It is assumed here that the Functest Docker container is already
36 properly deployed and that all instructions described in this guide are to be
37 performed from *inside* the deployed Functest Docker container.
39 .. include:: ./introduction.rst
41 The different test cases are described in the remaining sections of this document.
43 VIM (Virtualized Infrastructure Manager)
44 ----------------------------------------
48 In Danube, healthcheck tests have been refactored and rely on SNAPS, a
49 OPNFV middleware project.
51 SNAPS stands for "SDN/NFV Application development Platform and Stack".
52 SNAPS is an object-oriented OpenStack library packaged with tests that exercise
54 More information on SNAPS can be found in `[13]`_
56 Three tests are declared as healthcheck tests and can be used for gating by the
57 installer, they cover functionally the tests previously done by healthcheck
65 * *snaps_health_check*
67 Connection_check consists in 9 test cases (test duration < 5s) checking the
68 connectivity with Glance, Keystone, Neutron, Nova and the external network.
70 Api_check verifies the retrieval of OpenStack clients: Keystone, Glance,
71 Neutron and Nova and may perform some simple queries. When the config value of
72 snaps.use_keystone is True, functest must have access to the cloud's private
73 network. This suite consists in 49 tests (test duration < 2 minutes).
75 snaps_health_check creates instance, allocate floating IP, connect to the VM.
76 This test replaced the previous Colorado healthcheck test.
78 Self-obviously, successful completion of the 'healthcheck' testcase is a
79 necessary pre-requisite for the execution of all other test Tiers.
85 Given the script **ping.sh**::
89 ping -c 1 $1 2>&1 >/dev/null
91 if [ "Z$RES" = "Z0" ] ; then
101 The goal of this test is to establish an SSH connection using a floating IP
102 on the Public/External network and verify that 2 instances can talk over a Private
106 +-------------+ +-------------+
108 | | Boot VM1 with IP1 | |
109 | +------------------->| |
110 | Tester | | System |
111 | | Boot VM2 | Under |
112 | +------------------->| Test |
114 | | Create floating IP | |
115 | +------------------->| |
117 | | Assign floating IP | |
119 | +------------------->| |
121 | | Establish SSH | |
122 | | connection to VM2 | |
123 | | through floating IP| |
124 | +------------------->| |
126 | | SCP ping.sh to VM2 | |
127 | +------------------->| |
130 | | ping.sh to VM1 | |
131 | +------------------->| |
135 | | else (timeout): | |
138 +-------------+ +-------------+
140 This test can be considered as an "Hello World" example.
141 It is the first basic use case which **must** work on any deployment.
146 This test case is similar to vPing_ssh but without the use of Floating IPs
147 and the Public/External network to transfer the ping script.
148 Instead, it uses Nova metadata service to pass it to the instance at booting time.
149 As vPing_ssh, it checks that 2 instances can talk to
150 each other on a Private Tenant network::
152 vPing_userdata test case
153 +-------------+ +-------------+
155 | | Boot VM1 with IP1 | |
156 | +------------------->| |
158 | | Boot VM2 with | |
159 | | ping.sh as userdata| |
160 | | with IP1 as $1. | |
161 | +------------------->| |
162 | Tester | | System |
163 | | VM2 exeutes ping.sh| Under |
164 | | (ping IP1) | Test |
165 | +------------------->| |
168 | | console-log VM 2 | |
171 | | else (timeout) | |
174 +-------------+ +-------------+
176 When the second VM boots it will execute the script passed as userdata
177 automatically. The ping will be detected by periodically capturing the output
178 in the console-log of the second VM.
184 Tempest `[2]`_ is the reference OpenStack Integration test suite.
185 It is a set of integration tests to be run against a live OpenStack cluster.
186 Tempest has suites of tests for:
188 * OpenStack API validation
190 * Other specific tests useful in validating an OpenStack deployment
192 Functest uses Rally `[3]`_ to run the Tempest suite.
193 Rally generates automatically the Tempest configuration file **tempest.conf**.
194 Before running the actual test cases,
195 Functest creates the needed resources (user, tenant) and
196 updates the appropriate parameters into the configuration file.
198 When the Tempest suite is executed, each test duration is measured and the full
199 console output is stored to a *log* file for further analysis.
201 The Tempest testcases are distributed accross two
204 * Smoke Tier - Test Case 'tempest_smoke_serial'
205 * Components Tier - Test case 'tempest_full_parallel'
207 NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
208 tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
209 executes all defined Tempest tests using several concurrent threads
210 (i.e. parallel mode). The number of threads activated corresponds to the number
211 of available logical CPUs.
213 The goal of the Tempest test suite is to check the basic functionalities of the
214 different OpenStack components on an OPNFV fresh installation, using the
215 corresponding REST API interfaces.
218 Rally bench test suites
219 ^^^^^^^^^^^^^^^^^^^^^^^
221 Rally `[3]`_ is a benchmarking tool that answers the question:
223 *How does OpenStack work at scale?*
225 The goal of this test suite is to benchmark all the different OpenStack modules and
226 get significant figures that could help to define Telco Cloud KPIs.
228 The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
240 A basic SLA (stop test on errors) has been implemented.
242 The Rally testcases are distributed accross two Tiers:
244 * Smoke Tier - Test Case 'rally_sanity'
245 * Components Tier - Test case 'rally_full'
247 NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
248 cases. Test case 'rally_full' executes the full defined set of Rally tests.
251 Refstack-client to run Defcore testcases
252 -----------------------------------------
254 Refstack-client `[8]`_ is a command line utility that allows you to
255 execute Tempest test runs based on configurations you specify.
256 It is the official tool to run Defcore `[9]`_ testcases,
257 which focuses on testing interoperability between OpenStack clouds.
259 Refstack-client is integrated in Functest, consumed by Dovetail, which
260 intends to define and provide a set of OPNFV related validation criteria
261 that will provide input for the evaluation of the use of OPNFV trademarks.
262 This progress is under the guideline of Compliance Verification Program(CVP).
269 Set of DefCore tempest test cases not flagged and required.
270 According to `[10]`_, some tests are still flagged due to outstanding bugs
271 in the Tempest library, particularly tests that require SSH. Refstack developers
272 are working on correcting these bugs upstream. Please note that although some tests
273 are flagged because of bugs, there is still an expectation that the capabilities
274 covered by the tests are available. It only contains Openstack core compute
275 (no object storage). The approved guidelines (2016.08) are valid for Kilo,
276 Liberty, Mitaka and Newton releases of OpenStack.
277 The list can be generated using the Rest API from RefStack project:
278 https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false
283 Two running methods are provided after refstack-client integrated into
284 Functest, Functest command line and manually, respectively.
286 By default, for Defcore test cases run by Functest command line,
287 are run followed with automatically generated
288 configuration file, i.e., refstack_tempest.conf. In some circumstances,
289 the automatic configuration file may not quite satisfied with the SUT,
290 Functest also inherits the refstack-client command line and provides a way
291 for users to set its configuration file according to its own SUT manually.
295 Inside the Functest container, first to prepare Functest environment:
299 cd /home/opnfv/repos/functest
303 then to run default defcore testcases by using refstack-client:
307 functest testcase run refstack_defcore
309 In OPNFV Continuous Integration(CI) system, the command line method is used.
313 Inside the Functest container, first to prepare the refstack virtualenv:
317 cd /home/opnfv/repos/refstack-client
318 source .venv/bin/activate
320 then prepare the tempest configuration file and the testcases want to run with the SUT,
321 run the testcases with:
325 ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>
327 using help for more information:
331 ./refstack-client --help
332 ./refstack-client test --help
334 Reference tempest configuration
335 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
337 *command line method*
339 When command line method is used, the default tempest configuration file
340 is generated by Rally.
344 When running manually is used, recommended way to generate tempest configuration
349 cd /home/opnfv/repos/functest/functest/opnfv_tests/openstack/refstack_client
350 python tempest_conf.py
352 a file called tempest.conf is stored in the current path by default, users can do
353 some adjustment according to the SUT:
357 vim refstack_tempest.conf
359 a reference article can be used `[15]`_.
365 This test case contains tests that setup and destroy environments with VMs with
366 and without Floating IPs with a newly created user and project. Set the config
367 value snaps.use_floating_ips (True|False) to toggle this functionality. When
368 the config value of snaps.use_keystone is True, Functest must have access
369 the cloud's private network.
370 This suite consists in 38 tests (test duration < 10 minutes)
376 There are currently 3 available controllers:
385 The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
386 from the ODL project using the Robot `[11]`_ framework.
387 The suite verifies creation and deletion of networks, subnets and ports with
388 OpenDaylight and Neutron.
390 The list of tests can be described as follows:
392 * Basic Restconf test cases
393 * Connect to Restconf URL
394 * Check the HTTP code status
396 * Neutron Reachability test cases
397 * Get the complete list of neutron resources (networks, subnets, ports)
399 * Neutron Network test cases
400 * Check OpenStack networks
401 * Check OpenDaylight networks
402 * Create a new network via OpenStack and check the HTTP status code returned by Neutron
403 * Check that the network has also been successfully created in OpenDaylight
405 * Neutron Subnet test cases
406 * Check OpenStack subnets
407 * Check OpenDaylight subnets
408 * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
409 * Check that the subnet has also been successfully created in OpenDaylight
411 * Neutron Port test cases
412 * Check OpenStack Neutron for known ports
413 * Check OpenDaylight ports
414 * Create a new port via OpenStack and check the HTTP status code returned by Neutron
415 * Check that the new port has also been successfully created in OpenDaylight
418 * Delete the port previously created via OpenStack
419 * Check that the port has been also succesfully deleted in OpenDaylight
420 * Delete previously subnet created via OpenStack
421 * Check that the subnet has also been successfully deleted in OpenDaylight
422 * Delete the network created via OpenStack
423 * Check that the network has also been succesfully deleted in OpenDaylight
425 Note: the checks in OpenDaylight are based on the returned HTTP status
426 code returned by OpenDaylight.
432 TestON Framework is used to test the ONOS SDN controller functions.
433 The test cases deal with L2 and L3 functions.
434 The ONOS test suite can be run on any ONOS compliant scenario.
436 The test cases are described as follows:
438 * onosfunctest: The main executable file contains the initialization of
439 the docker environment and functions called by FUNCvirNetNB and
444 * Create Network: Post Network data and check it in ONOS
445 * Update Network: Update the Network and compare it in ONOS
446 * Delete Network: Delete the Network and check if it's NULL in ONOS or
448 * Create Subnet: Post Subnet data and check it in ONOS
449 * Update Subnet: Update the Subnet and compare it in ONOS
450 * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not
451 * Create Port: Post Port data and check it in ONOS
452 * Update Port: Update the Port and compare it in ONOS
453 * Delete Port: Delete the Port and check if it's NULL in ONOS or not
457 * Create Router: Post data for create Router and check it in ONOS
458 * Update Router: Update the Router and compare it in ONOS
459 * Delete Router: Delete the Router data and check it in ONOS
460 * Create RouterInterface: Post Router Interface data to an existing Router
462 * Delete RouterInterface: Delete the RouterInterface and check the Router
463 * Create FloatingIp: Post data for create FloatingIp and check it in ONOS
464 * Update FloatingIp: Update the FloatingIp and compare it in ONOS
465 * Delete FloatingIp: Delete the FloatingIp and check that it is 'NULL' in
467 * Create External Gateway: Post data to create an External Gateway for an
468 existing Router and check it in ONOS
469 * Update External Gateway: Update the External Gateway and compare the change
470 * Delete External Gateway: Delete the External Gateway and check that it is
477 In Danube, Functest supports the integration of:
490 Note: copper is not supported in Danube.
492 Please refer to the dedicated feature user guides for details.
501 The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
502 architectural framework for delivering IP multimedia services.
504 vIMS has been integrated in Functest to demonstrate the capability to deploy a
505 relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
506 functional VNF allows the test of most of the essential functions needed for a
509 The goal of this test suite consists of:
511 * deploy a VNF orchestrator (Cloudify)
512 * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
513 orchestrator based on a TOSCA blueprint defined in `[5]`_
514 * run suite of signaling tests on top of this VNF
516 The Clearwater architecture is described as follows:
518 .. figure:: ../../../images/clearwater-architecture.png
520 :alt: vIMS architecture
524 Orchestra test case deals with the deployment of OpenIMS with OpenBaton
530 See parser user guide for details: `[12]`_
536 This test case deals with the deployment and the test of vyos vrouter with
537 Cloudify orchestrator. The test case can do testing for interchangeability of
538 BGP Protocol using vyos.
540 The Workflow is as follows:
542 Deploy VNF Testing topology by Cloudify using blueprint.
544 Setting configuration to Target VNF and reference VNF using ssh
546 Execution of test command for test item written YAML format file.
547 Check VNF status and behavior.
549 Output of report based on result using JSON format.
551 The vyos-vrouter architecture is described in `[14]`_
555 .. include:: ./runfunctest.rst
564 In manual mode test results are displayed in the console and result files
565 are put in /home/opnfv/functest/results.
570 In automated mode, test results are displayed in jenkins logs, a summary is provided
571 at the end of the job and can be described as follow::
573 +==================================================================================================================================================+
575 +==================================================================================================================================================+
577 | Deployment description: |
579 | SCENARIO: os-odl_l2-nofeature-ha |
580 | BUILD TAG: jenkins-functest-fuel-baremetal-daily-master-324 |
583 +=========================+===============+============+===============+===========================================================================+
584 | TEST CASE | TIER | DURATION | RESULT | URL |
585 +=========================+===============+============+===============+===========================================================================+
586 | connection_check | healthcheck | 00:02 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb62b34079ac000a42e3fe |
587 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
588 | api_check | healthcheck | 01:15 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb62fe4079ac000a42e3ff |
589 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
590 | snaps_health_check | healthcheck | 00:50 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63314079ac000a42e400 |
591 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
592 | vping_ssh | smoke | 01:10 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63654079ac000a42e401 |
593 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
594 | vping_userdata | smoke | 00:59 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb63a14079ac000a42e403 |
595 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
596 | tempest_smoke_serial | smoke | 12:57 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb66bd4079ac000a42e408 |
597 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
598 | rally_sanity | smoke | 10:22 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb692b4079ac000a42e40a |
599 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
600 | refstack_defcore | smoke | 12:28 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6c184079ac000a42e40c |
601 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
602 | snaps_smoke | smoke | 12:04 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6eec4079ac000a42e40e |
603 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
604 | domino | features | 00:29 | PASS | http://testresults.opnfv.org/test/api/v1/results/58cb6f044079ac000a42e40f |
605 +-------------------------+---------------+------------+---------------+---------------------------------------------------------------------------+
608 Results are automatically pushed to the test results database, some additional
609 result files are pushed to OPNFV artifact web sites.
611 Based on the results stored in the result database, a `Functest reporting`_
612 portal is also automatically updated. This portal provides information on:
614 * The overall status per scenario and per installer
615 * Tempest: Tempest test case including reported errors per scenario and installer
616 * vIMS: vIMS details per scenario and installer
618 .. figure:: ../../../images/functest-reporting-status.png
620 :alt: Functest reporting portal Fuel status page
622 .. include:: ./troubleshooting.rst
628 .. _`[1]`: http://artifacts.opnfv.org/functest/colorado/docs/configguide/#
629 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
630 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
631 .. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf
632 .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
633 .. _`[6]`: https://scap.nist.gov/
634 .. _`[7]`: https://github.com/OpenSCAP/openscap
635 .. _`[8]`: https://github.com/openstack/refstack-client
636 .. _`[9]`: https://github.com/openstack/defcore
637 .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
638 .. _`[11]`: http://robotframework.org/
639 .. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
640 .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
641 .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
642 .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/
648 IRC support chan: #opnfv-functest
650 .. _`OPNFV main site`: http://www.opnfv.org
651 .. _`Functest page`: https://wiki.opnfv.org/functest
652 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
653 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
654 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/functest/ci/config_functest.yaml
655 .. _`Functest reporting`: http://testresults.opnfv.org/reporting/functest/release/danube/index-status-fuel.html