docs/testing/user/userguide/test_overview.rst

   1 .. SPDX-License-Identifier: CC-BY-4.0
   2
   3 Overview of the Functest suites
   4 ===============================
   5
   6 Functest is the OPNFV project primarily targeting functional testing.
   7 In the Continuous Integration pipeline, it is launched after an OPNFV fresh
   8 installation to validate and verify the basic functions of the
   9 infrastructure.
  10
  11 The current list of test suites can be distributed over 5 main domains:
  12   * VIM (Virtualised Infrastructure Manager)
  13   * Controllers (i.e. SDN Controllers)
  14   * VNF (Virtual Network Functions)
  15   * Kubernetes
  16
  17 Functest test suites are also distributed in the OPNFV testing categories:
  18 healthcheck, smoke, benchmarking, VNF, Stress tests.
  19
  20 All the Healthcheck and smoke tests of a given scenario must be successful to
  21 validate the scenario for the release.
  22
  23 +-------------+---------------+------------+----------------------------------+
  24 | Domain      | Tier          | Test case  | Comments                         |
  25 +=============+===============+============+==================================+
  26 | VIM         | healthcheck   | connection | Check OpenStack connectivity     |
  27 |             |               | \_check    |                                  |
  28 |             +---------------+------------+----------------------------------+
  29 |             | smoke         | vping_ssh  | NFV "Hello World" using an SSH   |
  30 |             |               |            | connection to a destination VM   |
  31 |             |               |            | over a created floating IP       |
  32 |             |               |            | address on the SUT Public /      |
  33 |             |               |            | External network. Using the SSH  |
  34 |             |               |            | connection a test script is then |
  35 |             |               |            | copied to the destination        |
  36 |             |               |            | VM and then executed via SSH.    |
  37 |             |               |            | The script will ping another     |
  38 |             |               |            | VM on a specified IP address over|
  39 |             |               |            | the SUT Private Tenant network   |
  40 |             |               +------------+----------------------------------+
  41 |             |               | vping      | Uses Ping with given userdata    |
  42 |             |               | \_userdata | to test intra-VM connectivity    |
  43 |             |               |            | over the SUT Private Tenant      |
  44 |             |               |            | network. The correct operation   |
  45 |             |               |            | of the NOVA Metadata service is  |
  46 |             |               |            | also verified in this test       |
  47 |             |               +------------+----------------------------------+
  48 |             |               | tempest    | Generate and run a relevant      |
  49 |             |               | \_smoke    | Tempest Test Suite in smoke mode.|
  50 |             |               |            | The generated test set is        |
  51 |             |               |            | dependent on the OpenStack       |
  52 |             |               |            | deployment environment           |
  53 |             |               +------------+----------------------------------+
  54 |             |               | rally      | Run a subset of the OpenStack    |
  55 |             |               | \_sanity   | Rally Test Suite in smoke mode   |
  56 |             |               +------------+----------------------------------+
  57 |             |               | refstack   | Reference RefStack suite         |
  58 |             |               | \_defcore  | tempest selection for NFV        |
  59 |             |               +------------+----------------------------------+
  60 |             |               | patrole    | Patrole is a tempest plugin for  |
  61 |             |               |            | testing and verifying RBAC policy|
  62 |             |               |            | enforcement, which offers testing|
  63 |             |               |            | for the following OpenStack      |
  64 |             |               |            | services: Nova, Neutron, Glance, |
  65 |             |               |            | Cinder and Keystone              |
  66 |             +---------------+------------+----------------------------------+
  67 |             |               | neutron    | The neutron trunk port testcases |
  68 |             |               | \_trunk    | have been introduced and they are|
  69 |             |               |            | supported by installers :        |
  70 |             |               |            | Apex, Fuel and Compass.          |
  71 |             +---------------+------------+----------------------------------+
  72 |             | components    | tempest    | Generate and run a full set of   |
  73 |             |               | \_full     | the OpenStack Tempest Test Suite.|
  74 |             |               | \_parallel | See the OpenStack reference test |
  75 |             |               |            | suite `[2]`_. The generated      |
  76 |             |               |            | test set is dependent on the     |
  77 |             |               |            | OpenStack deployment environment |
  78 |             |               +------------+----------------------------------+
  79 |             |               | rally_full | Run the OpenStack testing tool   |
  80 |             |               |            | benchmarking OpenStack modules   |
  81 |             |               |            | See the Rally documents `[3]`_   |
  82 +-------------+---------------+------------+----------------------------------+
  83 | Controllers | smoke         | odl        | Opendaylight Test suite          |
  84 |             |               |            | Limited test suite to check the  |
  85 |             |               |            | basic neutron (Layer 2)          |
  86 |             |               |            | operations mainly based on       |
  87 |             |               |            | upstream testcases. See below    |
  88 |             |               |            | for details                      |
  89 +-------------+---------------+------------+----------------------------------+
  90 | VNF         | vnf           | cloudify   | Example of a real VNF deployment |
  91 |             |               | \_ims      | to show the NFV capabilities of  |
  92 |             |               |            | the platform. The IP Multimedia  |
  93 |             |               |            | Subsystem is a typical Telco test|
  94 |             |               |            | case, referenced by ETSI.        |
  95 |             |               |            | It provides a fully functional   |
  96 |             |               |            | VoIP System                      |
  97 |             |               +------------+----------------------------------+
  98 |             |               | vyos       | vRouter testing                  |
  99 |             |               | \_vrouter  |                                  |
 100 |             |               +------------+----------------------------------+
 101 |             |               | juju_epc   | Validates deployment of a complex|
 102 |             |               |            | mobility VNF on OPNFV Platform.  |
 103 |             |               |            | Uses Juju for deploying the OAI  |
 104 |             |               |            | EPC  and ABot for defining test  |
 105 |             |               |            | scenarios using high-level DSL.  |
 106 |             |               |            | VNF tests reference 3GPP         |
 107 |             |               |            | Technical Specs and are executed |
 108 |             |               |            | through protocol drivers provided|
 109 |             |               |            | by ABot.                         |
 110 +-------------+---------------+------------+----------------------------------+
 111 | Kubernetes  | healthcheck   | k8s_smoke  | Test a running Kubernetes        |
 112 |             |               |            | cluster and ensure it satisfies  |
 113 |             |               |            | minimal functional requirements  |
 114 |             +---------------+------------+----------------------------------+
 115 |             | smoke         | k8s\_      | Run a subset of Kubernetes       |
 116 |             |               | conformance| End-to-End tests, expected to    |
 117 |             |               |            | pass on any Kubernetes cluster   |
 118 +-------------+---------------+------------+----------------------------------+
 119
 120
 121 As shown in the above table, Functest is structured into different 'domains',
 122 'tiers' and 'test cases'. Each 'test case' usually represents an actual
 123 'Test Suite' comprised -in turn- of several test cases internally.
 124
 125 Test cases also have an implicit execution order. For example, if the early
 126 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
 127 Tier testcases, there is little point to launch a full testcase execution
 128 round.
 129
 130 In Danube, we merged smoke and sdn controller tiers in smoke tier.
 131
 132 An overview of the Functest Structural Concept is depicted graphically below:
 133
 134 .. figure:: ../../../images/concepts_mapping_fraser.png
 135    :align: center
 136    :alt: Functest Concepts Structure
 137
 138 Some of the test cases are developed by Functest team members, whereas others
 139 are integrated from upstream communities or other OPNFV projects. For example,
 140 `Tempest <https://docs.openstack.org/tempest/latest/>`_ is the
 141 OpenStack integration test suite and Functest is in charge of the selection,
 142 integration and automation of those tests that fit suitably to OPNFV.
 143
 144 The Tempest test suite is the default OpenStack smoke test suite but no new
 145 test cases have been created in OPNFV Functest.
 146
 147 The results produced by the tests run from CI are pushed and collected into a
 148 NoSQL database. The goal is to populate the database with results from
 149 different sources and scenarios and to show them on a `Functest Dashboard`_. A
 150 screenshot of a live Functest Dashboard is shown below:
 151
 152 .. figure:: ../../../images/FunctestDashboardEuphrates.png
 153    :align: center
 154    :alt: Functest Dashboard
 155
 156
 157 Basic components (VIM, SDN controllers) are tested through their own suites.
 158 Feature projects also provide their own test suites with different ways of
 159 running their tests.
 160
 161 The notion of domain has been introduced in the description of the test cases
 162 stored in the Database.
 163 This parameters as well as possible tags can be used for the Test case catalog.
 164
 165 vIMS test case was integrated to demonstrate the capability to deploy a
 166 relatively complex NFV scenario on top of the OPNFV infrastructure.
 167
 168 Functest considers OPNFV as a black box. OPNFV offers a lot of potential
 169 combinations (which may change from one version to another):
 170
 171   * 3 controllers (OpenDaylight, ONOS, OpenContrail)
 172   * 5 installers (Apex, Compass, Daisy, Fuel, Joid)
 173
 174 Most of the tests are runnable by any combination, but some tests might have
 175 restrictions imposed by the utilized installers or due to the available
 176 deployed services. The system uses the environment variables to automatically
 177 determine the valid test cases, for each given environment.
 178
 179 A convenience Functest CLI utility is also available to simplify setting up the
 180 Functest environment, management of the OpenStack environment (e.g. resource
 181 clean-up) and for executing tests.
 182 The Functest CLI organised the testcase into logical Tiers, which contain in
 183 turn one or more testcases. The CLI allows execution of a single specified
 184 testcase, all test cases in a specified Tier, or the special case of execution
 185 of **ALL** testcases. The Functest CLI is introduced in more details in next
 186 section.
 187
 188 .. _`[2]`: https://docs.openstack.org/tempest/latest/
 189 .. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
 190 .. _`Functest Dashboard`: http://testresults.opnfv.org/