1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 =========================
5 OPNFV FUNCTEST user guide
6 =========================
15 The goal of this document is to describe the Functest test cases as well as
16 provide a procedure to execute them.
18 A presentation has been created for the first OPNFV Summit `[4]`_.
20 This document is a continuation of the Functest Configuration Guide`[1]`_ and it
21 is assumed that the Functest Docker container is properly deployed.
23 **IMPORTANT**: All the instructions described in this guide must be performed
26 .. include:: ./introduction.rst
28 The different test cases are described in the section hereafter.
30 VIM (Virtualized Infrastructure Manager)
31 ----------------------------------------
36 Given the script **ping.sh**::
40 ping -c 1 $1 2>&1 >/dev/null
42 if [ "Z$RES" = "Z0" ] ; then
52 The goal of this test is to establish an SSH connection using a floating IP
53 on the public network and verify that 2 instances can talk on a private network::
56 +-------------+ +-------------+
58 | | Boot VM1 with IP1 | |
59 | +------------------->| |
61 | | Boot VM2 | Under |
62 | +------------------->| Test |
64 | | Create floating IP | |
65 | +------------------->| |
67 | | Assign floating IP | |
69 | +------------------->| |
72 | | connection to VM2 | |
73 | | through floating IP| |
74 | +------------------->| |
76 | | SCP ping.sh to VM2 | |
77 | +------------------->| |
80 | | ping.sh to VM1 | |
81 | +------------------->| |
85 | | else (timeout): | |
88 +-------------+ +-------------+
90 This test can be considered as an "Hello World" example.
91 It is the first basic use case which shall work on any deployment.
96 This test case is similar to vPing_ssh but without the use of floating ips
97 and the public network. It only checks that 2 instances can talk to each other
98 on a private network but it also verifies that the Nova metadata service is
101 vPing_userdata test case
102 +-------------+ +-------------+
104 | | Boot VM1 with IP1 | |
105 | +------------------->| |
107 | | Boot VM2 with | |
108 | | ping.sh as userdata| |
109 | | with IP1 as $1. | |
110 | +------------------->| |
111 | Tester | | System |
112 | | VM2 exeutes ping.sh| Under |
113 | | (ping IP1) | Test |
114 | +------------------->| |
117 | | console-log VM 2 | |
120 | | else (timeout) | |
123 +-------------+ +-------------+
125 When the second VM boots it will execute the script passed as userdata
126 automatically and the ping will be detected capturing periodically the output
127 in the console-log of the second VM.
133 Tempest `[2]`_ is the reference OpenStack Integration test suite.
134 It is a set of integration tests to be run against a live OpenStack cluster.
135 Tempest has batteries of tests for:
137 * OpenStack API validation
139 * Other specific tests useful in validating an OpenStack deployment
141 Functest uses Rally `[3]`_ to run the Tempest suite.
142 Rally generates automatically the Tempest configuration file **tempest.conf**.
143 Before running the actual test cases,
144 Functest creates the needed resources (user, tenant) and
145 updates the appropriate parameters into the configuration file.
146 When the Tempest suite is executed, each test duration is measured and the full
147 console output is stored in a *log* file for further analysis.
149 As an addition of Arno, Brahmaputra runs a customized set of Tempest test cases.
150 The list is specificed through *--tests-file* when executing the Rally command.
151 This option has been introduced in the version 0.1.2 of the Rally framework.
153 This customized list contains more than 200 Tempest test cases and can be divided
154 into two main sections:
156 1) Set of Tempest smoke test cases
157 2) Set of test cases from DefCore list `[8]`_
159 The goal of the Tempest test suite is to check the basic functionalities of the
160 different OpenStack components on an OPNFV fresh installation using
161 the corresponding REST API interfaces.
164 Rally bench test suites
165 ^^^^^^^^^^^^^^^^^^^^^^^
167 Rally `[3]`_ is a benchmarking tool that answers the question:
169 *How does OpenStack work at scale?*
171 The goal of this test suite is to benchmark all the different OpenStack modules and
172 get significant figures that could help to define Telco Cloud KPIs.
174 The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
186 A basic SLA (stop test on errors) have been implemented.
192 Brahmaputra introduces new SDN controllers.
193 There are currently 2 available controllers:
201 The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
202 from the ODL project using the Robot `[11]`_ framework.
203 The suite verifies creation and deletion of networks, subnets and ports with
204 OpenDaylight and Neutron.
206 The list of tests can be described as follows:
208 * Restconf.basic: Get the controller modules via Restconf
211 * Check OpenStack Networks :: Checking OpenStack Neutron for known networks
212 * Check OpenDaylight Networks :: Checking OpenDaylight Neutron API
213 * Create Network :: Create new network in OpenStack
214 * Check Network :: Check Network created in OpenDaylight
215 * Neutron.Networks :: Checking Network created in OpenStack are pushed
219 * Check OpenStack Subnets :: Checking OpenStack Neutron for known Subnets
220 * Check OpenDaylight subnets :: Checking OpenDaylight Neutron API
221 * Create New subnet :: Create new subnet in OpenStack
222 * Check New subnet :: Check new subnet created in OpenDaylight
223 * Neutron.Subnets :: Checking Subnets created in OpenStack are pushed
227 * Check OpenStack ports :: Checking OpenStack Neutron for known ports
228 * Check OpenDaylight ports :: Checking OpenDaylight Neutron API
229 * Create New Port :: Create new port in OpenStack
230 * Check New Port :: Check new subnet created in OpenDaylight
231 * Neutron.Ports :: Checking Port created in OpenStack are pushed
235 * Delete previously created subnet in OpenStack
236 * Check subnet deleted in OpenDaylight
237 * Check subnet deleted in OpenStack
241 * Delete previously created network in OpenStack
242 * Check network deleted in OpenDaylight
243 * Check network deleted in OpenStack
249 TestON Framework is used to test the ONOS SDN controller functions.
250 The test cases deal with L2 and L3 functions.
251 The ONOS test suite can be run on any ONOS compliant scenario.
253 The test cases are described as follows:
255 * onosfunctest: The main executable file contains the initialization of
256 the docker environment and functions called by FUNCvirNetNB and
261 * Create Network: Post Network data and check it in ONOS
262 * Update Network: Update the Network and compare it in ONOS
263 * Delete Network: Delete the Network and check if it's NULL in ONOS or
265 * Create Subnet: Post Subnet data and check it in ONOS
266 * Update Subnet: Update the Subnet and compare it in ONOS
267 * Delete Subnet: Delete the Subnet and check if it's NULL in ONOS or not
268 * Create Port: Post Port data and check it in ONOS
269 * Update Port: Update the Port and compare it in ONOS
270 * Delete Port: Delete the Port and check if it's NULL in ONOS or not
274 * Create Router: Post dataes for create Router and check it in ONOS
275 * Update Router: Update the Router and compare it in ONOS
276 * Delete Router: Delete the Router dataes and check it in ONOS
277 * Create RouterInterface: Post RouterInterface data to an exist Router
279 * Delete RouterInterface: Delete the RouterInterface and check the Router
280 * Create FloatingIp: Post dataes for create FloatingIp and check it in
282 * Update FloatingIp: Update the FloatingIp and compare it in ONOS
283 * Delete FloatingIp: Delete the FloatingIp and check if it's NULL in
285 * Create External Gateway: Post dataes for create External Gateway to an
286 exit Router and check it
287 * Update External Gateway: Update the External Gateway and compare it
288 * Delete External Gateway: Delete the External Gateway and check if it's
297 The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
298 architectural framework for delivering IP multimedia services.
300 vIMS has been integrated in Functest to demonstrate the capability to deploy a
301 relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
302 functional VNF allows the test of most of the
303 essential functions needed for a NFV platform.
305 The goal of this test suite consists of:
307 * deploy a VNF orchestrator (Cloudify)
308 * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
309 orchestrator based on a TOSCA blueprint defined in `[5]`_
310 * run suite of signaling tests on top of this VNF
312 The Clearwater architecture is described as follows:
314 .. figure:: ../images/clearwater-architecture.png
316 :alt: vIMS architecture
323 Promise provides a basic set of test cases as part of Brahmaputra.
325 The available 33 test cases can be grouped into 7 test suites:
327 #. Add a new OpenStack provider into resource pool: Registers
328 OpenStack into a new resource pool and adds more capacity associated
331 #. Allocation without reservation: Creates a new server in OpenStack
332 and adds a new allocation record in Promise shim-layer.
334 #. Allocation using reservation for immediate use: Creates a resource
335 reservation record with no start/end time and immediately creates a new
336 server in OpenStack and add a new allocation record in Promise
339 #. Reservation for future use: Creates a resource reservation record
340 for a future start time, queries, modifies and cancels the newly created
343 #. Capacity planning: Decreases and increases the available capacity
344 from a provider in the future and queries the available collections and
347 #. Reservation with conflict: Tries to create reservations for
348 immediate and future use with conflict.
350 #. Cleanup test allocations: Destroys all allocations in OpenStack.
355 .. include:: ./runfunctest.rst
360 For Brahmaputra test results, see the functest results document at `[12]`_
362 Note that the results are documented per scenario basis. Although most of the test
363 cases might show the same output, some of them are not supported by
364 certain scenario. Please select the appropriate scenario and compare the results
365 to the referenced in the documentation.
371 Based on results collected in CI, a test dashboard is dynamically generated.
372 The URL of this dashboard is TODO LF
375 .. include:: ./troubleshooting.rst
381 .. _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/#functional-testing-installation
382 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
383 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
384 .. _`[4]`: http://events.linuxfoundation.org/sites/events/files/slides/Functest%20in%20Depth_0.pdf
385 .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
386 .. _`[6]`: https://wiki.opnfv.org/opnfv_test_dashboard
387 .. _`[7]`: http://testresults.opnfv.org/testapi/test_projects/functest/cases
388 .. _`[8]`: https://wiki.openstack.org/wiki/Governance/DefCoreCommittee
389 .. _`[9]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/libraries/os_defaults.yaml
390 .. _`[10]`: https://git.opnfv.org/cgit/functest/tree/testcases/VIM/OpenStack/CI/rally_cert/task.yaml
391 .. _`[11]`: http://robotframework.org/
392 .. _`[12]`: http://artifacts.opnfv.org/functest/brahmaputra/docs/results/index.html
394 OPNFV main site: opnfvmain_.
396 OPNFV functional test page: opnfvfunctest_.
398 IRC support chan: #opnfv-testperf
400 .. _opnfvmain: http://www.opnfv.org
401 .. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing
402 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
403 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
404 .. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
405 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml