1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
5 The different test cases are described in the remaining sections of this document.
7 VIM (Virtualized Infrastructure Manager)
8 ----------------------------------------
12 In Danube, healthcheck tests have been refactored and rely on SNAPS, an
13 OPNFV middleware project.
15 SNAPS stands for "SDN/NFV Application development Platform and Stack".
16 SNAPS is an object-oriented OpenStack library packaged with tests that exercise
18 More information on SNAPS can be found in `[13]`_
20 Three tests are declared as healthcheck tests and can be used for gating by the
21 installer, they cover functionally the tests previously done by healthcheck
29 * *snaps_health_check*
31 Connection_check consists in 9 test cases (test duration < 5s) checking the
32 connectivity with Glance, Keystone, Neutron, Nova and the external network.
34 Api_check verifies the retrieval of OpenStack clients: Keystone, Glance,
35 Neutron and Nova and may perform some simple queries. When the config value of
36 snaps.use_keystone is True, functest must have access to the cloud's private
37 network. This suite consists in 49 tests (test duration < 2 minutes).
39 snaps_health_check creates instance, allocate floating IP, connect to the VM.
40 This test replaced the previous Colorado healthcheck test.
42 Self-obviously, successful completion of the 'healthcheck' testcase is a
43 necessary pre-requisite for the execution of all other test Tiers.
49 Given the script **ping.sh**::
52 ping -c 1 $1 2>&1 >/dev/null
54 if [ "Z$RES" = "Z0" ] ; then
61 The goal of this test is to establish an SSH connection using a floating IP
62 on the Public/External network and verify that 2 instances can talk over a Private
66 +-------------+ +-------------+
68 | | Boot VM1 with IP1 | |
69 | +------------------->| |
71 | | Boot VM2 | Under |
72 | +------------------->| Test |
74 | | Create floating IP | |
75 | +------------------->| |
77 | | Assign floating IP | |
79 | +------------------->| |
82 | | connection to VM2 | |
83 | | through floating IP| |
84 | +------------------->| |
86 | | SCP ping.sh to VM2 | |
87 | +------------------->| |
90 | | ping.sh to VM1 | |
91 | +------------------->| |
95 | | else (timeout): | |
98 +-------------+ +-------------+
100 This test can be considered as an "Hello World" example.
101 It is the first basic use case which **must** work on any deployment.
106 This test case is similar to vPing_ssh but without the use of Floating IPs
107 and the Public/External network to transfer the ping script.
108 Instead, it uses Nova metadata service to pass it to the instance at booting time.
109 As vPing_ssh, it checks that 2 instances can talk to
110 each other on a Private Tenant network::
112 vPing_userdata test case
113 +-------------+ +-------------+
115 | | Boot VM1 with IP1 | |
116 | +------------------->| |
118 | | Boot VM2 with | |
119 | | ping.sh as userdata| |
120 | | with IP1 as $1. | |
121 | +------------------->| |
122 | Tester | | System |
123 | | VM2 exeutes ping.sh| Under |
124 | | (ping IP1) | Test |
125 | +------------------->| |
128 | | console-log VM 2 | |
131 | | else (timeout) | |
134 +-------------+ +-------------+
136 When the second VM boots it will execute the script passed as userdata
137 automatically. The ping will be detected by periodically capturing the output
138 in the console-log of the second VM.
144 Tempest `[2]`_ is the reference OpenStack Integration test suite.
145 It is a set of integration tests to be run against a live OpenStack cluster.
146 Tempest has suites of tests for:
148 * OpenStack API validation
150 * Other specific tests useful in validating an OpenStack deployment
152 Functest uses Rally `[3]`_ to run the Tempest suite.
153 Rally generates automatically the Tempest configuration file **tempest.conf**.
154 Before running the actual test cases,
155 Functest creates the needed resources (user, tenant) and
156 updates the appropriate parameters into the configuration file.
158 When the Tempest suite is executed, each test duration is measured and the full
159 console output is stored to a *log* file for further analysis.
161 The Tempest testcases are distributed across two
164 * Smoke Tier - Test Case 'tempest_smoke_serial'
165 * Components Tier - Test case 'tempest_full_parallel'
167 NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
168 tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
169 executes all defined Tempest tests using several concurrent threads
170 (i.e. parallel mode). The number of threads activated corresponds to the number
171 of available logical CPUs.
173 The goal of the Tempest test suite is to check the basic functionalities of the
174 different OpenStack components on an OPNFV fresh installation, using the
175 corresponding REST API interfaces.
178 Rally bench test suites
179 ^^^^^^^^^^^^^^^^^^^^^^^
181 Rally `[3]`_ is a benchmarking tool that answers the question:
183 *How does OpenStack work at scale?*
185 The goal of this test suite is to benchmark all the different OpenStack modules and
186 get significant figures that could help to define Telco Cloud KPIs.
188 The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
199 A basic SLA (stop test on errors) has been implemented.
201 The Rally testcases are distributed across two Tiers:
203 * Smoke Tier - Test Case 'rally_sanity'
204 * Components Tier - Test case 'rally_full'
206 NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
207 cases. Test case 'rally_full' executes the full defined set of Rally tests.
210 Refstack-client to run Defcore testcases
211 -----------------------------------------
213 Refstack-client `[8]`_ is a command line utility that allows you to
214 execute Tempest test runs based on configurations you specify.
215 It is the official tool to run Defcore `[9]`_ testcases,
216 which focuses on testing interoperability between OpenStack clouds.
218 Refstack-client is integrated in Functest, consumed by Dovetail, which
219 intends to define and provide a set of OPNFV related validation criteria
220 that will provide input for the evaluation of the use of OPNFV trademarks.
221 This progress is under the guideline of Compliance Verification Program(CVP).
228 Set of DefCore tempest test cases not flagged and required.
229 According to `[10]`_, some tests are still flagged due to outstanding bugs
230 in the Tempest library, particularly tests that require SSH. Refstack developers
231 are working on correcting these bugs upstream. Please note that although some tests
232 are flagged because of bugs, there is still an expectation that the capabilities
233 covered by the tests are available. It only contains Openstack core compute
234 (no object storage). The approved guidelines (2016.08) are valid for Kilo,
235 Liberty, Mitaka and Newton releases of OpenStack.
236 The list can be generated using the Rest API from RefStack project:
237 https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false
242 Two running methods are provided after refstack-client integrated into
243 Functest, Functest command line and manually, respectively.
245 By default, for Defcore test cases run by Functest command line,
246 are run followed with automatically generated
247 configuration file, i.e., refstack_tempest.conf. In some circumstances,
248 the automatic configuration file may not quite satisfied with the SUT,
249 Functest also inherits the refstack-client command line and provides a way
250 for users to set its configuration file according to its own SUT manually.
254 Inside the Functest container, first to prepare Functest environment:
260 then to run default defcore testcases by using refstack-client:
264 functest testcase run refstack_defcore
266 In OPNFV Continuous Integration(CI) system, the command line method is used.
270 Prepare the tempest configuration file and the testcases want to run with the SUT,
271 run the testcases with:
275 ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>
277 using help for more information:
281 ./refstack-client --help
282 ./refstack-client test --help
284 Reference tempest configuration
285 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
287 *command line method*
289 When command line method is used, the default tempest configuration file
290 is generated by Rally.
294 When running manually is used, recommended way to generate tempest configuration
299 cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client
300 python tempest_conf.py
302 a file called tempest.conf is stored in the current path by default, users can do
303 some adjustment according to the SUT:
307 vim refstack_tempest.conf
309 a reference article can be used `[15]`_.
315 This test case contains tests that setup and destroy environments with VMs with
316 and without Floating IPs with a newly created user and project. Set the config
317 value snaps.use_floating_ips (True|False) to toggle this functionality. When
318 the config value of snaps.use_keystone is True, Functest must have access
319 the cloud's private network.
320 This suite consists in 38 tests (test duration < 10 minutes)
326 There are currently 3 available controllers:
335 The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
336 from the ODL project using the Robot `[11]`_ framework.
337 The suite verifies creation and deletion of networks, subnets and ports with
338 OpenDaylight and Neutron.
340 The list of tests can be described as follows:
342 * Basic Restconf test cases
343 * Connect to Restconf URL
344 * Check the HTTP code status
346 * Neutron Reachability test cases
347 * Get the complete list of neutron resources (networks, subnets, ports)
349 * Neutron Network test cases
350 * Check OpenStack networks
351 * Check OpenDaylight networks
352 * Create a new network via OpenStack and check the HTTP status code returned by Neutron
353 * Check that the network has also been successfully created in OpenDaylight
355 * Neutron Subnet test cases
356 * Check OpenStack subnets
357 * Check OpenDaylight subnets
358 * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
359 * Check that the subnet has also been successfully created in OpenDaylight
361 * Neutron Port test cases
362 * Check OpenStack Neutron for known ports
363 * Check OpenDaylight ports
364 * Create a new port via OpenStack and check the HTTP status code returned by Neutron
365 * Check that the new port has also been successfully created in OpenDaylight
368 * Delete the port previously created via OpenStack
369 * Check that the port has been also successfully deleted in OpenDaylight
370 * Delete previously subnet created via OpenStack
371 * Check that the subnet has also been successfully deleted in OpenDaylight
372 * Delete the network created via OpenStack
373 * Check that the network has also been successfully deleted in OpenDaylight
375 Note: the checks in OpenDaylight are based on the returned HTTP status
376 code returned by OpenDaylight.
382 Functest has been supporting several feature projects since Brahpamutra:
385 +-----------------+---------+----------+--------+-----------+
386 | Test | Brahma | Colorado | Danube | Euphrates |
387 +=================+=========+==========+========+===========+
388 | barometer | | | X | X |
389 +-----------------+---------+----------+--------+-----------+
390 | bgpvpn | | X | X | X |
391 +-----------------+---------+----------+--------+-----------+
393 +-----------------+---------+----------+--------+-----------+
394 | doctor | X | X | X | X |
395 +-----------------+---------+----------+--------+-----------+
396 | domino | | X | X | X |
397 +-----------------+---------+----------+--------+-----------+
399 +-----------------+---------+----------+--------+-----------+
401 +-----------------+---------+----------+--------+-----------+
402 | multisite | | X | X | |
403 +-----------------+---------+----------+--------+-----------+
404 | netready | | | X | |
405 +-----------------+---------+----------+--------+-----------+
406 | odl_sfc | | X | X | X |
407 +-----------------+---------+----------+--------+-----------+
409 +-----------------+---------+----------+--------+-----------+
410 | orchestra | | | X | X |
411 +-----------------+---------+----------+--------+-----------+
413 +-----------------+---------+----------+--------+-----------+
414 | promise | X | X | X | X |
415 +-----------------+---------+----------+--------+-----------+
416 | security_scan | | X | X | |
417 +-----------------+---------+----------+--------+-----------+
419 Please refer to the dedicated feature user guides for details.
428 The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
429 architectural framework for delivering IP multimedia services.
431 vIMS has been integrated in Functest to demonstrate the capability to deploy a
432 relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
433 functional VNF allows the test of most of the essential functions needed for a
436 The goal of this test suite consists of:
438 * deploy a VNF orchestrator (Cloudify)
439 * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
440 orchestrator based on a TOSCA blueprint defined in `[5]`_
441 * run suite of signaling tests on top of this VNF
443 The Clearwater architecture is described as follows:
445 .. figure:: ../../../images/clearwater-architecture.png
447 :alt: vIMS architecture
451 Orchestra test case deals with the deployment of OpenIMS with OpenBaton
454 orchestra_clearwaterims
455 ^^^^^^^^^^^^^^^^^^^^^^^
456 Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton
462 See parser user guide for details: `[12]`_
468 This test case deals with the deployment and the test of vyos vrouter with
469 Cloudify orchestrator. The test case can do testing for interchangeability of
470 BGP Protocol using vyos.
472 The Workflow is as follows:
474 Deploy VNF Testing topology by Cloudify using blueprint.
476 Setting configuration to Target VNF and reference VNF using ssh
478 Execution of test command for test item written YAML format file.
479 Check VNF status and behavior.
481 Output of report based on result using JSON format.
483 The vyos-vrouter architecture is described in `[14]`_
485 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
486 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
487 .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
488 .. _`[8]`: https://github.com/openstack/refstack-client
489 .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
490 .. _`[11]`: http://robotframework.org/
491 .. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
492 .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
493 .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
494 .. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/