1 .. SPDX-License-Identifier: CC-BY-4.0
3 The different test cases are described in the remaining sections of this
6 VIM (Virtualized Infrastructure Manager)
7 ----------------------------------------
11 Since Danube, healthcheck tests have been refactored and rely on SNAPS, an
12 OPNFV middleware project.
14 SNAPS stands for "SDN/NFV Application development Platform and Stack".
15 SNAPS is an object-oriented OpenStack library packaged with tests that exercise
17 More information on SNAPS can be found in `[13]`_
19 Three tests are declared as healthcheck tests and can be used for gating by the
20 installer, they cover functionally the tests previously done by healthcheck
28 Connection_check consists in test cases (test duration < 5s) checking the
29 connectivity with Glance, Keystone, Neutron, Nova and the external network.
31 Self-obviously, successful completion of the 'healthcheck' testcase is a
32 necessary pre-requisite for the execution of all other test Tiers.
38 Given the script **ping.sh**::
41 ping -c 1 $1 2>&1 >/dev/null
43 if [ "Z$RES" = "Z0" ] ; then
50 The goal of this test is to establish an SSH connection using a floating IP
51 on the Public/External network and verify that 2 instances can talk over a
52 Private Tenant network::
55 +-------------+ +-------------+
57 | | Boot VM1 with IP1 | |
58 | +------------------->| |
60 | | Boot VM2 | Under |
61 | +------------------->| Test |
63 | | Create floating IP | |
64 | +------------------->| |
66 | | Assign floating IP | |
68 | +------------------->| |
71 | | connection to VM2 | |
72 | | through floating IP| |
73 | +------------------->| |
75 | | SCP ping.sh to VM2 | |
76 | +------------------->| |
79 | | ping.sh to VM1 | |
80 | +------------------->| |
84 | | else (timeout): | |
87 +-------------+ +-------------+
89 This test can be considered as an "Hello World" example.
90 It is the first basic use case which **must** work on any deployment.
95 This test case is similar to vPing_ssh but without the use of Floating IPs
96 and the Public/External network to transfer the ping script.
97 Instead, it uses Nova metadata service to pass it to the instance at booting
99 As vPing_ssh, it checks that 2 instances can talk to
100 each other on a Private Tenant network::
102 vPing_userdata test case
103 +-------------+ +-------------+
105 | | Boot VM1 with IP1 | |
106 | +-------------------->| |
108 | | Boot VM2 with | |
109 | | ping.sh as userdata | |
110 | | with IP1 as $1. | |
111 | +-------------------->| |
112 | Tester | | System |
113 | | VM2 executes ping.sh| Under |
114 | | (ping IP1) | Test |
115 | +-------------------->| |
118 | | console-log VM 2 | |
121 | | else (timeout) | |
124 +-------------+ +-------------+
126 When the second VM boots it will execute the script passed as userdata
127 automatically. The ping will be detected by periodically capturing the output
128 in the console-log of the second VM.
134 Tempest `[2]`_ is the reference OpenStack Integration test suite.
135 It is a set of integration tests to be run against a live OpenStack cluster.
136 Tempest has suites of tests for:
138 * OpenStack API validation
140 * Other specific tests useful in validating an OpenStack deployment
142 Functest uses Rally `[3]`_ to run the Tempest suite.
143 Rally generates automatically the Tempest configuration file **tempest.conf**.
144 Before running the actual test cases,
145 Functest creates the needed resources (user, tenant) and
146 updates the appropriate parameters into the configuration file.
148 When the Tempest suite is executed, each test duration is measured and the full
149 console output is stored to a *log* file for further analysis.
151 The Tempest testcases are distributed across three
154 * Smoke Tier - Test Case 'tempest_smoke'
155 * Components Tier - Test case 'tempest_full'
156 * Neutron Trunk Port - Test case 'neutron_trunk'
157 * OpenStack interop testcases - Test case 'refstack_defcore'
158 * Testing and verifying RBAC policy enforcement - Test case 'patrole'
160 NOTE: Test case 'tempest_smoke' executes a defined set of tempest smoke
161 tests. Test case 'tempest_full' executes all defined Tempest tests.
163 NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN
164 separated networks using a single NIC. The feature neutron trunk ports have
165 been supported by Apex, Fuel and Compass, so the tempest testcases have been
168 NOTE: Rally is also used to run Openstack Interop testcases `[9]`_, which focus
169 on testing interoperability between OpenStack clouds.
171 NOTE: Patrole is a tempest plugin for testing and verifying RBAC policy
172 enforcement. It runs Tempest-based API tests using specified RBAC roles, thus
173 allowing deployments to verify that only intended roles have access to those
174 APIs. Patrole currently offers testing for the following OpenStack services:
175 Nova, Neutron, Glance, Cinder and Keystone. Currently in functest, only neutron
176 and glance are tested.
178 The goal of the Tempest test suite is to check the basic functionalities of the
179 different OpenStack components on an OPNFV fresh installation, using the
180 corresponding REST API interfaces.
183 Rally bench test suites
184 ^^^^^^^^^^^^^^^^^^^^^^^
186 Rally `[3]`_ is a benchmarking tool that answers the question:
188 *How does OpenStack work at scale?*
190 The goal of this test suite is to benchmark all the different OpenStack modules
191 and get significant figures that could help to define Telco Cloud KPIs.
193 The OPNFV Rally scenarios are based on the collection of the actual Rally
205 A basic SLA (stop test on errors) has been implemented.
207 The Rally testcases are distributed across two Tiers:
209 * Smoke Tier - Test Case 'rally_sanity'
210 * Components Tier - Test case 'rally_full'
212 NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
213 cases. Test case 'rally_full' executes the full defined set of Rally tests.
222 The OpenDaylight (ODL) test suite consists of a set of basic tests inherited
223 from the ODL project using the Robot `[11]`_ framework.
224 The suite verifies creation and deletion of networks, subnets and ports with
225 OpenDaylight and Neutron.
227 The list of tests can be described as follows:
229 * Basic Restconf test cases
231 * Connect to Restconf URL
232 * Check the HTTP code status
234 * Neutron Reachability test cases
236 * Get the complete list of neutron resources (networks, subnets, ports)
238 * Neutron Network test cases
240 * Check OpenStack networks
241 * Check OpenDaylight networks
242 * Create a new network via OpenStack and check the HTTP status code returned
244 * Check that the network has also been successfully created in OpenDaylight
246 * Neutron Subnet test cases
248 * Check OpenStack subnets
249 * Check OpenDaylight subnets
250 * Create a new subnet via OpenStack and check the HTTP status code returned
252 * Check that the subnet has also been successfully created in OpenDaylight
254 * Neutron Port test cases
256 * Check OpenStack Neutron for known ports
257 * Check OpenDaylight ports
258 * Create a new port via OpenStack and check the HTTP status code returned by
260 * Check that the new port has also been successfully created in OpenDaylight
264 * Delete the port previously created via OpenStack
265 * Check that the port has been also successfully deleted in OpenDaylight
266 * Delete previously subnet created via OpenStack
267 * Check that the subnet has also been successfully deleted in OpenDaylight
268 * Delete the network created via OpenStack
269 * Check that the network has also been successfully deleted in OpenDaylight
271 Note: the checks in OpenDaylight are based on the returned HTTP status
272 code returned by OpenDaylight.
281 The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
282 architectural framework for delivering IP multimedia services.
284 vIMS has been integrated in Functest to demonstrate the capability to deploy a
285 relatively complex NFV scenario on the OPNFV platform. The deployment of a
286 complete functional VNF allows the test of most of the essential functions
287 needed for a NFV platform.
289 The goal of this test suite consists of:
291 * deploy a VNF orchestrator (Cloudify)
292 * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF from this
293 orchestrator based on a TOSCA blueprint defined in `[5]`_
294 * run suite of signaling tests on top of this VNF
296 The Clearwater architecture is described as follows:
298 .. figure:: ../../../images/clearwater-architecture-v2.png
300 :alt: vIMS architecture
304 The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
305 architectural framework for delivering IP multimedia services.
307 vIMS has been integrated in Functest to demonstrate the capability to deploy a
308 relatively complex NFV scenario on the OPNFV platform. The deployment of a
309 complete functional VNF allows the test of most of the essential functions
310 needed for a NFV platform.
312 The goal of this test suite consists of:
314 * deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF using
315 OpenStack Heat orchestrator based on a HOT template defined in `[17]`_
316 * run suite of signaling tests on top of this VNF
318 The Clearwater architecture is described as follows:
320 .. figure:: ../../../images/clearwater-architecture-v2.png
322 :alt: vIMS architecture
326 This test case deals with the deployment and the test of vyos vrouter with
327 Cloudify orchestrator. The test case can do testing for interchangeability of
328 BGP Protocol using vyos.
330 The Workflow is as follows:
332 Deploy VNF Testing topology by Cloudify using blueprint.
334 Setting configuration to Target VNF and reference VNF using ssh
336 Execution of test command for test item written YAML format file.
337 Check VNF status and behavior.
339 Output of report based on result using JSON format.
341 The vyos-vrouter architecture is described in `[14]`_
345 The Evolved Packet Core (EPC) is the main component of the System Architecture
346 Evolution (SAE) which forms the core of the 3GPP LTE specification.
348 vEPC has been integrated in Functest to demonstrate the capability to deploy a
349 complex mobility-specific NFV scenario on the OPNFV platform. The OAI EPC
350 supports most of the essential functions defined by the 3GPP Technical Specs;
351 hence the successful execution of functional tests on the OAI EPC provides a
352 good endorsement of the underlying NFV platform.
354 This integration also includes ABot, a Test Orchestration system that enables
355 test scenarios to be defined in high-level DSL. ABot is also deployed as a
356 VM on the OPNFV platform; and this provides an example of the automation
357 driver and the Test VNF being both deployed as separate VNFs on the underlying
360 The Workflow is as follows:
361 * Deploy Orchestrator
362 Deploy Juju controller using Bootstrap command.
364 Deploy ABot orchestrator and OAI EPC as Juju charms.
365 Configuration of ABot and OAI EPC components is handled through
366 built-in Juju relations.
368 Execution of ABot feature files triggered by Juju actions.
369 This executes a suite of LTE signalling tests on the OAI EPC.
371 ABot test results are parsed accordingly and pushed to Functest Db.
373 Details of the ABot test orchestration tool may be found in `[15]`_
378 Kubernetes testing relies on sets of tests, which are part of the Kubernetes
379 source tree, such as the Kubernetes End-to-End (e2e) tests `[16]`_.
381 The kubernetes testcases are distributed across various Tiers:
385 * k8s_smoke Test Case: Creates a Guestbook application that contains redis
386 server, 2 instances of redis slave, frontend application, frontend service
387 and redis master service and redis slave service. Using frontend service,
388 the test will write an entry into the guestbook application which will
389 store the entry into the backend redis database. Application flow MUST
390 work as expected and the data written MUST be available to read.
394 * k8s_conformance Test Case: Runs a series of k8s e2e tests expected to
395 pass on any Kubernetes cluster. It is a subset of tests necessary to
396 demonstrate conformance grows with each release. Conformance is thus
397 considered versioned, with backwards compatibility guarantees and are
398 designed to be run with no cloud provider configured.
401 .. _`[2]`: https://docs.openstack.org/tempest/latest/
402 .. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
403 .. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
404 .. _`[8]`: https://github.com/openstack/refstack-client
405 .. _`[9]`: https://github.com/openstack/interop
406 .. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
407 .. _`[11]`: https://robotframework.org/
408 .. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
409 .. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
410 .. _`[15]`: https://www.rebaca.com/what-we-do/abot-5g-network-simulator/
411 .. _`[16]`: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md
412 .. _`[17]`: https://github.com/Metaswitch/clearwater-heat/blob/release-129/clearwater.yaml