-.. This work is licensed under a Creative Commons Attribution 4.0 International License.
-.. http://creativecommons.org/licenses/by/4.0
+.. SPDX-License-Identifier: CC-BY-4.0
-
-The different test cases are described in the remaining sections of this document.
+The different test cases are described in the remaining sections of this
+document.
VIM (Virtualized Infrastructure Manager)
----------------------------------------
* *connection_check*
- * *api_check*
- * *snaps_health_check*
-Connection_check consists in 9 test cases (test duration < 5s) checking the
+Connection_check consists in test cases (test duration < 5s) checking the
connectivity with Glance, Keystone, Neutron, Nova and the external network.
-Api_check verifies the retrieval of OpenStack clients: Keystone, Glance,
-Neutron and Nova and may perform some simple queries. When the config value of
-snaps.use_keystone is True, functest must have access to the cloud's private
-network. This suite consists in 49 tests (test duration < 2 minutes).
-
-Snaps_health_check creates a VM with a single port with an IPv4 address that
-is assigned by DHCP and then validates the expected IP with the actual.
-
Self-obviously, successful completion of the 'healthcheck' testcase is a
necessary pre-requisite for the execution of all other test Tiers.
The goal of this test is to establish an SSH connection using a floating IP
-on the Public/External network and verify that 2 instances can talk over a Private
-Tenant network::
+on the Public/External network and verify that 2 instances can talk over a
+Private Tenant network::
vPing_ssh test case
+-------------+ +-------------+
This test case is similar to vPing_ssh but without the use of Floating IPs
and the Public/External network to transfer the ping script.
-Instead, it uses Nova metadata service to pass it to the instance at booting time.
+Instead, it uses Nova metadata service to pass it to the instance at booting
+time.
As vPing_ssh, it checks that 2 instances can talk to
each other on a Private Tenant network::
The Tempest testcases are distributed across three
Tiers:
- * Smoke Tier - Test Case 'tempest_smoke_serial'
- * Components Tier - Test case 'tempest_full_parallel'
+ * Smoke Tier - Test Case 'tempest_smoke'
+ * Components Tier - Test case 'tempest_full'
* Neutron Trunk Port - Test case 'neutron_trunk'
+ * OpenStack interop testcases - Test case 'refstack_defcore'
+ * Testing and verifying RBAC policy enforcement - Test case 'patrole'
+
+NOTE: Test case 'tempest_smoke' executes a defined set of tempest smoke
+tests. Test case 'tempest_full' executes all defined Tempest tests.
+
+NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN
+separated networks using a single NIC. The feature neutron trunk ports have
+been supported by Apex, Fuel and Compass, so the tempest testcases have been
+integrated normally.
-NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
-tests with a single thread (i.e. serial mode). Test case 'tempest_full_parallel'
-executes all defined Tempest tests using several concurrent threads
-(i.e. parallel mode). The number of threads activated corresponds to the number
-of available logical CPUs.
+NOTE: Rally is also used to run Openstack Interop testcases `[9]`_, which focus
+on testing interoperability between OpenStack clouds.
-NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN separated
-networks using a single NIC. The feature neutron trunk ports have been supported
-by Apex, Fuel and Compass, so the tempest testcases have been integrated normally.
+NOTE: Patrole is a tempest plugin for testing and verifying RBAC policy
+enforcement. It runs Tempest-based API tests using specified RBAC roles, thus
+allowing deployments to verify that only intended roles have access to those
+APIs. Patrole currently offers testing for the following OpenStack services:
+Nova, Neutron, Glance, Cinder and Keystone. Currently in functest, only neutron
+and glance are tested.
The goal of the Tempest test suite is to check the basic functionalities of the
different OpenStack components on an OPNFV fresh installation, using the
*How does OpenStack work at scale?*
-The goal of this test suite is to benchmark all the different OpenStack modules and
-get significant figures that could help to define Telco Cloud KPIs.
+The goal of this test suite is to benchmark all the different OpenStack modules
+and get significant figures that could help to define Telco Cloud KPIs.
-The OPNFV Rally scenarios are based on the collection of the actual Rally scenarios:
+The OPNFV Rally scenarios are based on the collection of the actual Rally
+scenarios:
* authenticate
* cinder
* neutron
* nova
* quotas
- * ceilometer
A basic SLA (stop test on errors) has been implemented.
cases. Test case 'rally_full' executes the full defined set of Rally tests.
-Refstack-client to run OpenStack interop testcases
---------------------------------------------------
-
-Refstack-client `[8]`_ is a command line utility that allows you to
-execute Tempest test runs based on configurations you specify.
-It is the official tool to run Openstack Interop (previously known as Defcore)
-testcases `[9]`_, which focus on testing interoperability between OpenStack
-clouds.
-
-Refstack-client is integrated in Functest, consumed by Dovetail, which
-intends to define and provide a set of OPNFV related validation criteria
-that will provide input for the evaluation of the use of OPNFV trademarks.
-This progress is under the guideline of Compliance Verification Program(CVP).
-
-Running methods
-^^^^^^^^^^^^^^^
-
-Two running methods are provided after refstack-client integrated into
-Functest, Functest command line and manually, respectively.
-
-By default, for Defcore test cases run by Functest command line,
-are run followed with automatically generated
-configuration file, i.e., refstack_tempest.conf. In some circumstances,
-the automatic configuration file may not quite satisfied with the SUT,
-Functest also inherits the refstack-client command line and provides a way
-for users to set its configuration file according to its own SUT manually.
-
-*command line*
-
-Inside the Functest container, first to prepare Functest environment:
-
-::
-
- functest env prepare
-
-then to run default defcore testcases by using refstack-client:
-
-::
-
- functest testcase run refstack_defcore
-
-In OPNFV Continuous Integration(CI) system, the command line method is used.
-
-*manually*
-
-Prepare the tempest configuration file and the testcases want to run with the SUT,
-run the testcases with:
-
-::
-
- ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>
-
-using help for more information:
-
-::
-
- ./refstack-client --help
- ./refstack-client test --help
-
-Reference tempest configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-*command line method*
-
-When command line method is used, the default tempest configuration file
-is generated by Rally.
-
-*manually*
-
-When running manually is used, recommended way to generate tempest configuration
-file is:
-
-::
-
- cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client
- python tempest_conf.py
-
-a file called tempest.conf is stored in the current path by default, users can do
-some adjustment according to the SUT:
-
-::
-
- vim refstack_tempest.conf
-
-a reference article can be used `[15]`_.
-
-
-snaps_smoke
-------------
-
-This test case contains tests that setup and destroy environments with VMs with
-and without Floating IPs with a newly created user and project. Set the config
-value snaps.use_floating_ips (True|False) to toggle this functionality.
-Please note that When the configuration value of snaps.use_keystone is True, Functest must have access
-the cloud's private network.
-This suite consists in 38 tests (test duration < 10 minutes)
-
-
SDN Controllers
---------------
The list of tests can be described as follows:
* Basic Restconf test cases
+
* Connect to Restconf URL
* Check the HTTP code status
* Neutron Reachability test cases
+
* Get the complete list of neutron resources (networks, subnets, ports)
* Neutron Network test cases
+
* Check OpenStack networks
* Check OpenDaylight networks
- * Create a new network via OpenStack and check the HTTP status code returned by Neutron
+ * Create a new network via OpenStack and check the HTTP status code returned
+ by Neutron
* Check that the network has also been successfully created in OpenDaylight
* Neutron Subnet test cases
+
* Check OpenStack subnets
* Check OpenDaylight subnets
- * Create a new subnet via OpenStack and check the HTTP status code returned by Neutron
+ * Create a new subnet via OpenStack and check the HTTP status code returned
+ by Neutron
* Check that the subnet has also been successfully created in OpenDaylight
* Neutron Port test cases
+
* Check OpenStack Neutron for known ports
* Check OpenDaylight ports
- * Create a new port via OpenStack and check the HTTP status code returned by Neutron
+ * Create a new port via OpenStack and check the HTTP status code returned by
+ Neutron
* Check that the new port has also been successfully created in OpenDaylight
* Delete operations
+
* Delete the port previously created via OpenStack
* Check that the port has been also successfully deleted in OpenDaylight
* Delete previously subnet created via OpenStack
Features
--------
-Functest has been supporting several feature projects since Brahpamutra:
-
-
-+-----------------+---------+----------+--------+-----------+
-| Test | Brahma | Colorado | Danube | Euphrates |
-+=================+=========+==========+========+===========+
-| barometer | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| bgpvpn | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| copper | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| doctor | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| domino | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| fds | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| moon | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| multisite | | X | X | |
-+-----------------+---------+----------+--------+-----------+
-| netready | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| odl_sfc | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| opera | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| orchestra | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| parser | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| promise | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| security_scan | | X | X | |
-+-----------------+---------+----------+--------+-----------+
+Functest has been supporting several feature projects since Brahmaputra:
+
+
++-----------------+---------+----------+--------+-----------+-----------+
+| Test | Brahma | Colorado | Danube | Euphrates | Fraser |
++=================+=========+==========+========+===========+===========+
+| barometer | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| bgpvpn | | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| copper | | X | | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| doctor | X | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| domino | | X | X | X | |
++-----------------+---------+----------+--------+-----------+-----------+
+| fds | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| moon | | X | | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| multisite | | X | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| netready | | | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| odl_sfc | | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| opera | | | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| orchestra | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| parser | | | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| promise | X | X | X | X | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| security_scan | | X | X | | |
++-----------------+---------+----------+--------+-----------+-----------+
+| clover | | | | | X |
++-----------------+---------+----------+--------+-----------+-----------+
+| stor4nfv | | | | | X |
++-----------------+---------+----------+--------+-----------+-----------+
Please refer to the dedicated feature user guides for details.
architectural framework for delivering IP multimedia services.
vIMS has been integrated in Functest to demonstrate the capability to deploy a
-relatively complex NFV scenario on the OPNFV platform. The deployment of a complete
-functional VNF allows the test of most of the essential functions needed for a
-NFV platform.
+relatively complex NFV scenario on the OPNFV platform. The deployment of a
+complete functional VNF allows the test of most of the essential functions
+needed for a NFV platform.
The goal of this test suite consists of:
The Clearwater architecture is described as follows:
-.. figure:: ../../../images/clearwater-architecture.png
+.. figure:: ../../../images/clearwater-architecture-v2.png
:align: center
:alt: vIMS architecture
+heat_ims
+^^^^^^^^
+The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
+architectural framework for delivering IP multimedia services.
-cloudify_ims_perf
-^^^^^^^^^^^^^^^^^
-This testcase extends the cloudify_ims test case.
-The first part is similar but the testing part is different.
-The testing part consists in automating a realistic signaling load on the vIMS
-using an Ixia loader (proprietary tools)
- - You need to have access to an Ixia licence server defined in the configuration
- file and have ixia image locally.
-
-This test case is available but not declared in testcases.yaml. The declaration
-of the testcase is simple, connect to your functest-vnf docker, add the following
-section in /usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml::
-
- -
- case_name: cloudify_ims_perf
- project_name: functest
- criteria: 80
- blocking: false
- description: >-
- Stress tests based on Cloudify. Ixia loader images and access to Ixia
- server license.
- dependencies:
- installer: ''
- scenario: 'os-nosdn-nofeature-ha'
- run:
- module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf'
- class: 'CloudifyImsPerf'
-
-orchestra_openims
-^^^^^^^^^^^^^^^^^
-Orchestra test case deals with the deployment of OpenIMS with OpenBaton
-orchestrator.
+vIMS has been integrated in Functest to demonstrate the capability to deploy a
+relatively complex NFV scenario on the OPNFV platform. The deployment of a
+complete functional VNF allows the test of most of the essential functions
+needed for a NFV platform.
-orchestra_clearwaterims
-^^^^^^^^^^^^^^^^^^^^^^^
-Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton
-orchestrator.
+The goal of this test suite consists of:
+
+* deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF using
+ OpenStack Heat orchestrator based on a HOT template defined in `[17]`_
+* run suite of signaling tests on top of this VNF
+
+The Clearwater architecture is described as follows:
+
+.. figure:: ../../../images/clearwater-architecture-v2.png
+ :align: center
+ :alt: vIMS architecture
vyos-vrouter
^^^^^^^^^^^^
The vyos-vrouter architecture is described in `[14]`_
+juju_epc
+^^^^^^^^
+The Evolved Packet Core (EPC) is the main component of the System Architecture
+Evolution (SAE) which forms the core of the 3GPP LTE specification.
+
+vEPC has been integrated in Functest to demonstrate the capability to deploy a
+complex mobility-specific NFV scenario on the OPNFV platform. The OAI EPC
+supports most of the essential functions defined by the 3GPP Technical Specs;
+hence the successful execution of functional tests on the OAI EPC provides a
+good endorsement of the underlying NFV platform.
+
+This integration also includes ABot, a Test Orchestration system that enables
+test scenarios to be defined in high-level DSL. ABot is also deployed as a
+VM on the OPNFV platform; and this provides an example of the automation
+driver and the Test VNF being both deployed as separate VNFs on the underlying
+OPNFV platform.
+
+The Workflow is as follows:
+ * Deploy Orchestrator
+ Deploy Juju controller using Bootstrap command.
+ * Deploy VNF
+ Deploy ABot orchestrator and OAI EPC as Juju charms.
+ Configuration of ABot and OAI EPC components is handled through
+ built-in Juju relations.
+ * Test VNF
+ Execution of ABot feature files triggered by Juju actions.
+ This executes a suite of LTE signalling tests on the OAI EPC.
+ * Reporting
+ ABot test results are parsed accordingly and pushed to Functest Db.
+
+Details of the ABot test orchestration tool may be found in `[15]`_
+
+Kubernetes (K8s)
+----------------
+
+Kubernetes testing relies on sets of tests, which are part of the Kubernetes
+source tree, such as the Kubernetes End-to-End (e2e) tests `[16]`_.
+
+The kubernetes testcases are distributed across various Tiers:
+
+ * Healthcheck Tier
+
+ * k8s_smoke Test Case: Creates a Guestbook application that contains redis
+ server, 2 instances of redis slave, frontend application, frontend service
+ and redis master service and redis slave service. Using frontend service,
+ the test will write an entry into the guestbook application which will
+ store the entry into the backend redis database. Application flow MUST
+ work as expected and the data written MUST be available to read.
+
+ * Smoke Tier
+
+ * k8s_conformance Test Case: Runs a series of k8s e2e tests expected to
+ pass on any Kubernetes cluster. It is a subset of tests necessary to
+ demonstrate conformance grows with each release. Conformance is thus
+ considered versioned, with backwards compatibility guarantees and are
+ designed to be run with no cloud provider configured.
+
-.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
-.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
+.. _`[2]`: https://docs.openstack.org/tempest/latest/
+.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
.. _`[8]`: https://github.com/openstack/refstack-client
+.. _`[9]`: https://github.com/openstack/interop
.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
.. _`[11]`: http://robotframework.org/
-.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
-.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/
+.. _`[15]`: https://www.rebaca.com/what-we-do/abot-5g-network-simulator/
+.. _`[16]`: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md
+.. _`[17]`: https://github.com/Metaswitch/clearwater-heat/blob/release-129/clearwater.yaml