* *connection_check*
- * *api_check*
- * *snaps_health_check*
-Connection_check consists in 9 test cases (test duration < 5s) checking the
+Connection_check consists in test cases (test duration < 5s) checking the
connectivity with Glance, Keystone, Neutron, Nova and the external network.
-Api_check verifies the retrieval of OpenStack clients: Keystone, Glance,
-Neutron and Nova and may perform some simple queries. When the config value of
-snaps.use_keystone is True, functest must have access to the cloud's private
-network. This suite consists in 49 tests (test duration < 2 minutes).
-
-Snaps_health_check creates a VM with a single port with an IPv4 address that
-is assigned by DHCP and then validates the expected IP with the actual.
-
Self-obviously, successful completion of the 'healthcheck' testcase is a
necessary pre-requisite for the execution of all other test Tiers.
The Tempest testcases are distributed across three
Tiers:
- * Smoke Tier - Test Case 'tempest_smoke_serial'
- * Components Tier - Test case 'tempest_full_parallel'
+ * Smoke Tier - Test Case 'tempest_smoke'
+ * Components Tier - Test case 'tempest_full'
* Neutron Trunk Port - Test case 'neutron_trunk'
+ * OpenStack interop testcases - Test case 'refstack_defcore'
+ * Testing and verifying RBAC policy enforcement - Test case 'patrole'
-NOTE: Test case 'tempest_smoke_serial' executes a defined set of tempest smoke
-tests with a single thread (i.e. serial mode). Test case
-'tempest_full_parallel' executes all defined Tempest tests using several
-concurrent threads (i.e. parallel mode). The number of threads activated
-corresponds to the number of available logical CPUs.
+NOTE: Test case 'tempest_smoke' executes a defined set of tempest smoke
+tests. Test case 'tempest_full' executes all defined Tempest tests.
NOTE: The 'neutron_trunk' test set allows to connect a VM to multiple VLAN
separated networks using a single NIC. The feature neutron trunk ports have
been supported by Apex, Fuel and Compass, so the tempest testcases have been
integrated normally.
+NOTE: Rally is also used to run Openstack Interop testcases `[9]`_, which focus
+on testing interoperability between OpenStack clouds.
+
+NOTE: Patrole is a tempest plugin for testing and verifying RBAC policy
+enforcement. It runs Tempest-based API tests using specified RBAC roles, thus
+allowing deployments to verify that only intended roles have access to those
+APIs. Patrole currently offers testing for the following OpenStack services:
+Nova, Neutron, Glance, Cinder and Keystone. Currently in functest, only neutron
+and glance are tested.
+
The goal of the Tempest test suite is to check the basic functionalities of the
different OpenStack components on an OPNFV fresh installation, using the
corresponding REST API interfaces.
cases. Test case 'rally_full' executes the full defined set of Rally tests.
-Refstack-client to run OpenStack interop testcases
---------------------------------------------------
-
-Refstack-client `[8]`_ is a command line utility that allows you to
-execute Tempest test runs based on configurations you specify.
-It is the official tool to run Openstack Interop (previously known as Defcore)
-testcases `[9]`_, which focus on testing interoperability between OpenStack
-clouds.
-
-Refstack-client is integrated in Functest, consumed by Dovetail, which
-intends to define and provide a set of OPNFV related validation criteria
-that will provide input for the evaluation of the use of OPNFV trademarks.
-This progress is under the guideline of Compliance Verification Program(CVP).
-
-Running methods
-^^^^^^^^^^^^^^^
-
-Two running methods are provided after refstack-client integrated into
-Functest, Functest command line and manually, respectively.
-
-By default, for Defcore test cases run by Functest command line,
-are run followed with automatically generated
-configuration file, i.e., refstack_tempest.conf. In some circumstances,
-the automatic configuration file may not quite satisfied with the SUT,
-Functest also inherits the refstack-client command line and provides a way
-for users to set its configuration file according to its own SUT manually.
-
-*command line*
-
-Inside the Functest container, first to prepare Functest environment:
-
-::
-
- functest env prepare
-
-then to run default defcore testcases by using refstack-client:
-
-::
-
- functest testcase run refstack_defcore
-
-In OPNFV Continuous Integration(CI) system, the command line method is used.
-
-*manually*
-
-Prepare the tempest configuration file and the testcases want to run with the
-SUT, run the testcases with:
-
-::
-
- ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>
-
-using help for more information:
-
-::
-
- ./refstack-client --help
- ./refstack-client test --help
-
-Reference tempest configuration
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-*command line method*
-
-When command line method is used, the default tempest configuration file
-is generated by Rally.
-
-*manually*
-
-When running manually is used, recommended way to generate tempest
-configuration file is:
-
-::
-
- cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client
- python tempest_conf.py
-
-a file called tempest.conf is stored in the current path by default, users can
-do some adjustment according to the SUT:
-
-::
-
- vim refstack_tempest.conf
-
-a reference article can be used `[15]`_.
-
-
-snaps_smoke
-------------
-
-This test case contains tests that setup and destroy environments with VMs with
-and without Floating IPs with a newly created user and project. Set the config
-value snaps.use_floating_ips (True|False) to toggle this functionality.
-Please note that when the configuration value of snaps.use_keystone is True,
-Functest must have access the cloud's private network.
-This suite consists in 38 tests (test duration < 10 minutes)
-
-
SDN Controllers
---------------
code returned by OpenDaylight.
-Features
---------
-
-Functest has been supporting several feature projects since Brahmaputra:
-
-
-+-----------------+---------+----------+--------+-----------+
-| Test | Brahma | Colorado | Danube | Euphrates |
-+=================+=========+==========+========+===========+
-| barometer | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| bgpvpn | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| copper | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| doctor | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| domino | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| fds | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| moon | | X | | |
-+-----------------+---------+----------+--------+-----------+
-| multisite | | X | X | |
-+-----------------+---------+----------+--------+-----------+
-| netready | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| odl_sfc | | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| opera | | | X | |
-+-----------------+---------+----------+--------+-----------+
-| orchestra | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| parser | | | X | X |
-+-----------------+---------+----------+--------+-----------+
-| promise | X | X | X | X |
-+-----------------+---------+----------+--------+-----------+
-| security_scan | | X | X | |
-+-----------------+---------+----------+--------+-----------+
-
-Please refer to the dedicated feature user guides for details.
-
-
VNF
---
The Clearwater architecture is described as follows:
-.. figure:: ../../../images/clearwater-architecture.png
+.. figure:: ../../../images/clearwater-architecture-v2.png
:align: center
:alt: vIMS architecture
+heat_ims
+^^^^^^^^
+The IP Multimedia Subsystem or IP Multimedia Core Network Subsystem (IMS) is an
+architectural framework for delivering IP multimedia services.
+
+vIMS has been integrated in Functest to demonstrate the capability to deploy a
+relatively complex NFV scenario on the OPNFV platform. The deployment of a
+complete functional VNF allows the test of most of the essential functions
+needed for a NFV platform.
-cloudify_ims_perf
-^^^^^^^^^^^^^^^^^
-This testcase extends the cloudify_ims test case.
-The first part is similar but the testing part is different.
-The testing part consists in automating a realistic signaling load on the vIMS
-using an Ixia loader (proprietary tools)
-
- - You need to have access to an Ixia licence server defined in the
- configuration file and have ixia image locally.
-
-This test case is available but not declared in testcases.yaml. The declaration
-of the testcase is simple, connect to your functest-vnf docker, add the
-following section in
-/usr/lib/python2.7/site-packacges/functest/ci/testcases.yaml::
-
- -
- case_name: cloudify_ims_perf
- project_name: functest
- criteria: 80
- blocking: false
- description: >-
- Stress tests based on Cloudify. Ixia loader images and access to Ixia
- server license.
- dependencies:
- installer: ''
- scenario: 'os-nosdn-nofeature-ha'
- run:
- module: 'functest.opnfv_tests.vnf.ims.cloudify_ims_perf'
- class: 'CloudifyImsPerf'
-
-orchestra_openims
-^^^^^^^^^^^^^^^^^
-Orchestra test case deals with the deployment of OpenIMS with OpenBaton
-orchestrator.
+The goal of this test suite consists of:
-orchestra_clearwaterims
-^^^^^^^^^^^^^^^^^^^^^^^
-Orchestra test case deals with the deployment of Clearwater vIMS with OpenBaton
-orchestrator.
+* deploy a Clearwater vIMS (IP Multimedia Subsystem) VNF using
+ OpenStack Heat orchestrator based on a HOT template defined in `[17]`_
+* run suite of signaling tests on top of this VNF
+
+The Clearwater architecture is described as follows:
+
+.. figure:: ../../../images/clearwater-architecture-v2.png
+ :align: center
+ :alt: vIMS architecture
vyos-vrouter
^^^^^^^^^^^^
The vyos-vrouter architecture is described in `[14]`_
+juju_epc
+^^^^^^^^
+The Evolved Packet Core (EPC) is the main component of the System Architecture
+Evolution (SAE) which forms the core of the 3GPP LTE specification.
+
+vEPC has been integrated in Functest to demonstrate the capability to deploy a
+complex mobility-specific NFV scenario on the OPNFV platform. The OAI EPC
+supports most of the essential functions defined by the 3GPP Technical Specs;
+hence the successful execution of functional tests on the OAI EPC provides a
+good endorsement of the underlying NFV platform.
+
+This integration also includes ABot, a Test Orchestration system that enables
+test scenarios to be defined in high-level DSL. ABot is also deployed as a
+VM on the OPNFV platform; and this provides an example of the automation
+driver and the Test VNF being both deployed as separate VNFs on the underlying
+OPNFV platform.
+
+The Workflow is as follows:
+ * Deploy Orchestrator
+ Deploy Juju controller using Bootstrap command.
+ * Deploy VNF
+ Deploy ABot orchestrator and OAI EPC as Juju charms.
+ Configuration of ABot and OAI EPC components is handled through
+ built-in Juju relations.
+ * Test VNF
+ Execution of ABot feature files triggered by Juju actions.
+ This executes a suite of LTE signalling tests on the OAI EPC.
+ * Reporting
+ ABot test results are parsed accordingly and pushed to Functest Db.
+
+Details of the ABot test orchestration tool may be found in `[15]`_
+
+Kubernetes (K8s)
+----------------
+
+Kubernetes testing relies on sets of tests, which are part of the Kubernetes
+source tree, such as the Kubernetes End-to-End (e2e) tests `[16]`_.
+
+The kubernetes testcases are distributed across various Tiers:
+
+ * Healthcheck Tier
+
+ * k8s_smoke Test Case: Creates a Guestbook application that contains redis
+ server, 2 instances of redis slave, frontend application, frontend service
+ and redis master service and redis slave service. Using frontend service,
+ the test will write an entry into the guestbook application which will
+ store the entry into the backend redis database. Application flow MUST
+ work as expected and the data written MUST be available to read.
+
+ * Smoke Tier
+
+ * k8s_conformance Test Case: Runs a series of k8s e2e tests expected to
+ pass on any Kubernetes cluster. It is a subset of tests necessary to
+ demonstrate conformance grows with each release. Conformance is thus
+ considered versioned, with backwards compatibility guarantees and are
+ designed to be run with no cloud provider configured.
+
-.. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
-.. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
+.. _`[2]`: https://docs.openstack.org/tempest/latest/
+.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
.. _`[8]`: https://github.com/openstack/refstack-client
-.. _`[9]`: https://github.com/openstack/defcore
+.. _`[9]`: https://github.com/openstack/interop
.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
-.. _`[11]`: http://robotframework.org/
-.. _`[12]`: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
+.. _`[11]`: https://robotframework.org/
.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
-.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/
+.. _`[15]`: https://www.rebaca.com/what-we-do/abot-5g-network-simulator/
+.. _`[16]`: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md
+.. _`[17]`: https://github.com/Metaswitch/clearwater-heat/blob/release-129/clearwater.yaml