docs/testing/user/userguide/test_overview.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 Overview of the Functest suites
   5 ===============================
   6
   7 Functest is the OPNFV project primarily targeting functional testing.
   8 In the Continuous Integration pipeline, it is launched after an OPNFV fresh
   9 installation to validate and verify the basic functions of the
  10 infrastructure.
  11
  12 The current list of test suites can be distributed over 4 main domains:
  13   * VIM (Virtualised Infrastructure Manager)
  14   * Controllers (i.e. SDN Controllers)
  15   * Features
  16   * VNF (Virtual Network Functions)
  17
  18 Functest test suites are also distributed in the OPNFV testing categories:
  19 healthcheck, smoke, features, components, performance, VNF, Stress tests.
  20
  21 All the Healthcheck and smoke tests of a given scenario must be succesful to
  22 validate the scenario for the release.
  23
  24 +-------------+---------------+----------------+----------------------------------+
  25 | Domain      | Tier          | Test case      | Comments                         |
  26 +=============+===============+================+==================================+
  27 | VIM         | healthcheck   | connection     | Check OpenStack connectivity     |
  28 |             |               | _check         | through SNAPS framework          |
  29 |             |               +----------------+----------------------------------+
  30 |             |               | api_check      | Check OpenStack API through      |
  31 |             |               |                | SNAPS framework                  |
  32 |             |               +----------------+----------------------------------+
  33 |             |               | snaps_health   |  basic instance creation, check  |
  34 |             |               | \_check        |  DHCP                            |
  35 |             +---------------+----------------+----------------------------------+
  36 |             | smoke         | vping_ssh      | NFV "Hello World" using an SSH   |
  37 |             |               |                | connection to a destination VM   |
  38 |             |               |                | over a created floating IP       |
  39 |             |               |                | address on the SUT Public /      |
  40 |             |               |                | External network. Using the SSH  |
  41 |             |               |                | connection a test script is then |
  42 |             |               |                | copied to the destination        |
  43 |             |               |                | VM and then executed via SSH.    |
  44 |             |               |                | The script will ping another     |
  45 |             |               |                | VM on a specified IP address over|
  46 |             |               |                | the SUT Private Tenant network   |
  47 |             |               +----------------+----------------------------------+
  48 |             |               | vping_userdata | Uses Ping with given userdata    |
  49 |             |               |                | to test intra-VM connectivity    |
  50 |             |               |                | over the SUT Private Tenant      |
  51 |             |               |                | network. The correct operation   |
  52 |             |               |                | of the NOVA Metadata service is  |
  53 |             |               |                | also verified in this test       |
  54 |             |               +----------------+----------------------------------+
  55 |             |               | tempest_smoke  | Generate and run a relevant      |
  56 |             |               | \_serial       | Tempest Test Suite in smoke mode.|
  57 |             |               |                | The generated test set is        |
  58 |             |               |                | dependent on the OpenStack       |
  59 |             |               |                | deployment environment           |
  60 |             |               +----------------+----------------------------------+
  61 |             |               | rally_sanity   | Run a subset of the OpenStack    |
  62 |             |               |                | Rally Test Suite in smoke mode   |
  63 |             |               +----------------+----------------------------------+
  64 |             |               | snaps_smoke    | Run the SNAPS-OO integration     |
  65 |             |               |                | tests                            |
  66 |             |               +----------------+----------------------------------+
  67 |             |               | refstack       | Reference RefStack suite         |
  68 |             |               | \_defcore      | tempest selection for NFV        |
  69 |             |               +----------------+----------------------------------+
  70 |             |               | neutron_trunk  | The neutron trunk port testcases |
  71 |             |               |                | have been introduced and they are|
  72 |             |               |                | supported by installers :        |
  73 |             |               |                | Apex, Fuel and Compass.          |
  74 |             +---------------+----------------+----------------------------------+
  75 |             | components    | tempest_full   | Generate and run a full set of   |
  76 |             |               | \_parallel     | the OpenStack Tempest Test Suite.|
  77 |             |               |                | See the OpenStack reference test |
  78 |             |               |                | suite `[2]`_. The generated      |
  79 |             |               |                | test set is dependent on the     |
  80 |             |               |                | OpenStack deployment environment |
  81 |             |               +----------------+----------------------------------+
  82 |             |               | rally_full     | Run the OpenStack testing tool   |
  83 |             |               |                | benchmarking OpenStack modules   |
  84 |             |               |                | See the Rally documents `[3]`_   |
  85 +-------------+---------------+----------------+----------------------------------+
  86 | Controllers | smoke         | odl            | Opendaylight Test suite          |
  87 |             |               |                | Limited test suite to check the  |
  88 |             |               |                | basic neutron (Layer 2)          |
  89 |             |               |                | operations mainly based on       |
  90 |             |               |                | upstream testcases. See below    |
  91 |             |               |                | for details                      |
  92 |             |               +----------------+----------------------------------+
  93 |             |               | odl_netvirt    | Test Suite for the OpenDaylight  |
  94 |             |               |                | SDN Controller when the NetVirt  |
  95 |             |               |                | features are installed. It       |
  96 |             |               |                | integrates some test suites from |
  97 |             |               |                | upstream using Robot as the test |
  98 |             |               |                | framework                        |
  99 +-------------+---------------+----------------+----------------------------------+
 100 | Features    | features      | bgpvpn         | Implementation of the OpenStack  |
 101 |             |               |                | bgpvpn API from the SDNVPN       |
 102 |             |               |                | feature project. It allows for   |
 103 |             |               |                | the creation of BGP VPNs.        |
 104 |             |               |                | See `SDNVPN User Guide`_ for     |
 105 |             |               |                | details                          |
 106 |             |               +----------------+----------------------------------+
 107 |             |               | doctor         | Doctor platform, as of Colorado  |
 108 |             |               |                | release, provides the three      |
 109 |             |               |                | features:                        |
 110 |             |               |                | * Immediate Notification         |
 111 |             |               |                | * Consistent resource state      |
 112 |             |               |                | awareness for compute host down  |
 113 |             |               |                | * Valid compute host status      |
 114 |             |               |                | given to VM owner                |
 115 |             |               |                | See `Doctor User Guide`_ for     |
 116 |             |               |                | details                          |
 117 |             |               +----------------+----------------------------------+
 118 |             |               | odl-sfc        | SFC testing for odl scenarios    |
 119 |             |               |                | See `SFC User Guide`_ for details|
 120 |             |               +----------------+----------------------------------+
 121 |             |               | parser         | Parser is an integration project |
 122 |             |               |                | which aims to provide            |
 123 |             |               |                | placement/deployment templates   |
 124 |             |               |                | translation for OPNFV platform,  |
 125 |             |               |                | including TOSCA -> HOT, POLICY ->|
 126 |             |               |                | TOSCA and YANG -> TOSCA. it      |
 127 |             |               |                | deals with a fake vRNC.          |
 128 |             |               |                | See `Parser User Guide`_ for     |
 129 |             |               |                | details                          |
 130 |             |               +----------------+----------------------------------+
 131 |             |               | fds            | Test Suite for the OpenDaylight  |
 132 |             |               |                | SDN Controller when the GBP      |
 133 |             |               |                | features are installed. It       |
 134 |             |               |                | integrates some test suites from |
 135 |             |               |                | upstream using Robot as the test |
 136 |             |               |                | framework                        |
 137 +-------------+---------------+----------------+----------------------------------+
 138 | VNF         | vnf           | cloudify_ims   | Example of a real VNF deployment |
 139 |             |               |                | to show the NFV capabilities of  |
 140 |             |               |                | the platform. The IP Multimedia  |
 141 |             |               |                | Subsytem is a typical Telco test |
 142 |             |               |                | case, referenced by ETSI.        |
 143 |             |               |                | It provides a fully functional   |
 144 |             |               |                | VoIP System                      |
 145 |             |               +----------------+----------------------------------+
 146 |             |               | orchestra      | OpenIMS deployment using         |
 147 |             |               | \_openims      | Openbaton orchestrator           |
 148 |             |               +----------------+----------------------------------+
 149 |             |               | orchestra      | Cleawater IMS deployment using   |
 150 |             |               | \_cleawaterims | Openbaton orchestrator           |
 151 |             |               +----------------+----------------------------------+
 152 |             |               | vyos_vrouter   | vRouter testing                  |
 153 |             |               +----------------+----------------------------------+
 154 |             |               | cloudify_ims   | Based on cloudify_ims test case  |
 155 |             |               | perf           | cloudify_ims_perf substitutes    |
 156 |             |               |                | the signaling test suite by an   |
 157 |             |               |                | automatic deployment of an Ixia  |
 158 |             |               |                | loader and generic SIP stress    |
 159 |             |               |                | tests.                           |
 160 |             |               |                | This work has been initiated     |
 161 |             |               |                | during the plugfest and allows   |
 162 |             |               |                | realistic load tests on top of   |
 163 |             |               |                | cloudify_ims. Please note that   |
 164 |             |               |                | this test is available but not   |
 165 |             |               |                | declared in testcases.yaml as it |
 166 |             |               |                | requires access to proprietary   |
 167 |             |               |                | resources (Ixia loader)          |
 168 +-------------+---------------+----------------+----------------------------------+
 169
 170
 171 As shown in the above table, Functest is structured into different 'domains',
 172 'tiers' and 'test cases'. Each 'test case' usually represents an actual
 173 'Test Suite' comprised -in turn- of several test cases internally.
 174
 175 Test cases also have an implicit execution order. For example, if the early
 176 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
 177 Tier testcases, there is little point to launch a full testcase execution round.
 178
 179 In Danube, we merged smoke and sdn controller tiers in smoke tier.
 180
 181 An overview of the Functest Structural Concept is depicted graphically below:
 182
 183 .. figure:: ../../../images/concepts_mapping_final.png
 184    :align: center
 185    :alt: Functest Concepts Structure
 186
 187 Some of the test cases are developed by Functest team members, whereas others
 188 are integrated from upstream communities or other OPNFV projects. For example,
 189 `Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
 190 OpenStack integration test suite and Functest is in charge of the selection,
 191 integration and automation of those tests that fit suitably to OPNFV.
 192
 193 The Tempest test suite is the default OpenStack smoke test suite but no new test
 194 cases have been created in OPNFV Functest.
 195
 196 The results produced by the tests run from CI are pushed and collected into a
 197 NoSQL database. The goal is to populate the database with results from different
 198 sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
 199 of a live Functest Dashboard is shown below:
 200
 201 .. figure:: ../../../images/FunctestDashboardEuphrates.png
 202    :align: center
 203    :alt: Functest Dashboard
 204
 205
 206 Basic components (VIM, SDN controllers) are tested through their own suites.
 207 Feature projects also provide their own test suites with different ways of
 208 running their tests.
 209
 210 The notion of domain has been introduced in the description of the test cases
 211 stored in the Database.
 212 This parameters as well as possible tags can be used for the Test case catalog.
 213
 214 vIMS test case was integrated to demonstrate the capability to deploy a
 215 relatively complex NFV scenario on top of the OPNFV infrastructure.
 216
 217 Functest considers OPNFV as a black box. OPNFV offers a lot of potential
 218 combinations (which may change from one version to another):
 219
 220   * 3 controllers (OpenDaylight, ONOS, OpenContrail)
 221   * 5 installers (Apex, Compass, Daisy, Fuel, Joid)
 222
 223 Most of the tests are runnable by any combination, but some tests might have
 224 restrictions imposed by the utilized installers or due to the available
 225 deployed features. The system uses the environment variables (INSTALLER_TYPE and
 226 DEPLOY_SCENARIO) to automatically determine the valid test cases, for each given
 227 environment.
 228
 229 A convenience Functest CLI utility is also available to simplify setting up the
 230 Functest evironment, management of the OpenStack environment (e.g. resource
 231 clean-up) and for executing tests.
 232 The Functest CLI organised the testcase into logical Tiers, which contain in
 233 turn one or more testcases. The CLI allows execution of a single specified
 234 testcase, all test cases in a specified Tier, or the special case of execution
 235 of **ALL** testcases. The Functest CLI is introduced in more details in next
 236 section.
 237
 238 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
 239 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
 240 .. _`Doctor User Guide`: http://artifacts.opnfv.org/doctor/colorado/userguide/index.html
 241 .. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/colorado/docs/userguide/index.html
 242 .. _`Parser User Guide`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
 243 .. _`Functest Dashboard`: http://testresults.opnfv.org/kibana_dashboards/
 244 .. _`SFC User Guide`: http://artifacts.opnfv.org/sfc/colorado/userguide/index.html