docs/userguide/introduction.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 Overview of the Functest suites
   5 ===============================
   6
   7 Functest is the OPNFV project primarily targeting function testing.
   8 In the Continuous Integration pipeline, it is launched after an OPNFV fresh
   9 installation to validate and verify the basic functions of the
  10 infrastructure.
  11
  12 The current list of test suites can be distributed over 4 main domains: VIM
  13 (Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers),
  14 Features and VNF (Virtual Network Functions).
  15
  16 +-------------+---------------+----------------+----------------------------------+
  17 | Domain      | Tier          | Test case      | Comments                         |
  18 +=============+===============+================+==================================+
  19 | VIM         | healthcheck   | healthcheck    | Verify basic operation in VIM    |
  20 |             +---------------+----------------+----------------------------------+
  21 |             | smoke         | vPing_SSH      | NFV "Hello World" using an SSH   |
  22 |             |               |                | connection to a destination VM   |
  23 |             |               |                | over a created floating IP       |
  24 |             |               |                | address on the SUT Public /      |
  25 |             |               |                | External network. Using the SSH  |
  26 |             |               |                | connection a test script is then |
  27 |             |               |                | copied to the destination        |
  28 |             |               |                | VM and then executed via SSH.    |
  29 |             |               |                | The script will ping another     |
  30 |             |               |                | VM on a specified IP address over|
  31 |             |               |                | the SUT Private Tenant network.  |
  32 |             |               +----------------+----------------------------------+
  33 |             |               | vPing_userdata | Uses Ping with given userdata    |
  34 |             |               |                | to test intra-VM connectivity    |
  35 |             |               |                | over the SUT Private Tenant      |
  36 |             |               |                | network. The correct operation   |
  37 |             |               |                | of the NOVA Metadata service is  |
  38 |             |               |                | also verified in this test.      |
  39 |             |               +----------------+----------------------------------+
  40 |             |               | tempest_smoke  | Generate and run a relevant      |
  41 |             |               | \_serial       | Tempest Test Suite in smoke mode.|
  42 |             |               |                | The generated test set is        |
  43 |             |               |                | dependent on the OpenStack       |
  44 |             |               |                | deployment environment.          |
  45 |             |               +----------------+----------------------------------+
  46 |             |               | rally_sanity   | Run a subset of the OpenStack    |
  47 |             |               |                | Rally Test Suite in smoke mode   |
  48 |             +---------------+----------------+----------------------------------+
  49 |             | openstack     | tempest_full   | Generate and run a full set of   |
  50 |             |               | \_parallel     | the OpenStack Tempest Test Suite.|
  51 |             |               |                | See the OpenStack reference test |
  52 |             |               |                | suite `[2]`_. The generated      |
  53 |             |               |                | test set is dependent on the     |
  54 |             |               |                | OpenStack deployment environment.|
  55 |             |               +----------------+----------------------------------+
  56 |             |               | rally_full     | Run the OpenStack testing tool   |
  57 |             |               |                | benchmarking OpenStack modules   |
  58 |             |               |                | See the Rally documents `[3]`_.  |
  59 +-------------+---------------+----------------+----------------------------------+
  60 | Controllers | sdn_suites    | odl            | Opendaylight Test suite          |
  61 |             |               |                | TODO: Find a document reference! |
  62 |             |               +----------------+----------------------------------+
  63 |             |               | onos           | Test suite of ONOS L2 and L3     |
  64 |             |               |                | functions.                       |
  65 |             |               |                | See `ONOSFW User Guide`_  for    |
  66 |             |               |                | details.                         |
  67 +-------------+---------------+----------------+----------------------------------+
  68 | Features    | features      | Promise        | Resource reservation and         |
  69 |             |               |                | management project to identify   |
  70 |             |               |                | NFV related requirements and     |
  71 |             |               |                | realize resource reservation for |
  72 |             |               |                | future usage by capacity         |
  73 |             |               |                | management of resource pools     |
  74 |             |               |                | regarding compute, network and   |
  75 |             |               |                | storage.                         |
  76 |             |               |                | See `Promise User Guide`_ for    |
  77 |             |               |                | details.                         |
  78 |             |               +----------------+----------------------------------+
  79 |             |               | Doctor         | Doctor platform, as of Colorado  |
  80 |             |               |                | release, provides the two        |
  81 |             |               |                | features:                        |
  82 |             |               |                | * Immediate Notification         |
  83 |             |               |                | * Consistent resource state      |
  84 |             |               |                | awareness (compute). See the     |
  85 |             |               |                | See `Doctor User Guide`_ for     |
  86 |             |               |                | details                          |
  87 |             |               +----------------+----------------------------------+
  88 |             |               | bgpvpn         | Implementation of the OpenStack  |
  89 |             |               |                | bgpvpn API from the SDNVPN       |
  90 |             |               |                | feature project. It allows for   |
  91 |             |               |                | the creation of BGP VPNs.        |
  92 |             |               |                | See `SDNVPN User Guide`_ for     |
  93 |             |               |                | details                          |
  94 |             |               +----------------+----------------------------------+
  95 |             |               | security_scan  | Implementation of a simple       |
  96 |             |               |                | security scan. (Currently        |
  97 |             |               |                | available only for the Apex      |
  98 |             |               |                | installer environment)           |
  99 |             |               |                | TODO: Add document link from     |
 100 |             |               |                | Luke Hinds; when received.       |
 101 +-------------+---------------+----------------+----------------------------------+
 102 | VNF         | vnf           | vims           | Example of a real VNF deployment |
 103 |             |               |                | to show the NFV capabilities of  |
 104 |             |               |                | the platform. The IP Multimedia  |
 105 |             |               |                | Subsytem is a typical Telco test |
 106 |             |               |                | case, referenced by ETSI.        |
 107 |             |               |                | It provides a fully functional   |
 108 |             |               |                | VoIP System,                     |
 109 +-------------+---------------+----------------+----------------------------------+
 110
 111
 112 As shown in the above table, Functest is structured into different 'domains',
 113 'tiers' and 'test cases'. Each 'test case' usually represents an actual
 114 'Test Suite' comprised -in turn- of several test cases internally.
 115
 116 Test cases also have an implicit execution order. For example, if the early
 117 'healthcheck' Tier testcase fails, or if there are any failures in the 'smoke'
 118 Tier testcases, there is little point to launch a full testcase execution round.
 119
 120 An overview of the Functest Structural Concept is depicted graphically below:
 121
 122 .. figure:: ../images/concepts_mapping_final.png
 123    :align: center
 124    :alt: Functest Concepts Structure
 125
 126 Some of the test cases are developed by Functest team members, whereas others
 127 are integrated from upstream communities or other OPNFV projects. For example,
 128 `Tempest <http://docs.openstack.org/developer/tempest/overview.html>`_ is the
 129 OpenStack integration test suite and Functest is in charge of the selection,
 130 integration and automation of those tests that fit suitably to OPNFV.
 131
 132 The Tempest test suite has been customized but no new test cases have been
 133 created in OPNFV Functest.
 134
 135 The results produced by the tests run from CI are pushed and collected into a
 136 NoSQL database. The goal is to populate the database with results from different
 137 sources and scenarios and to show them on a `Functest Dashboard`_. A screenshot
 138 of a live Functest Dashboard is shown below:
 139
 140 .. figure:: ../images/FunctestDashboard.png
 141    :align: center
 142    :alt: Functest Dashboard
 143
 144
 145 There is no real notion of Test domain or Test coverage. Basic components
 146 (VIM, SDN controllers) are tested through their own suites. Feature projects
 147 also provide their own test suites with different ways of running their tests.
 148
 149 vIMS test case was integrated to demonstrate the capability to deploy a
 150 relatively complex NFV scenario on top of the OPNFV infrastructure.
 151
 152 Functest considers OPNFV as a black box. As of Colorado release the OPNFV
 153 offers a lot of potential combinations:
 154
 155   * 3 controllers (OpenDaylight, ONOS, OpenContrail)
 156   * 4 installers (Apex, Compass, Fuel, Joid)
 157
 158 Most of the tests are runnable by any combination, but some tests might have
 159 restrictions imposed by the utilized installers or due to the available
 160 deployed features. The system uses the environment variables (INSTALLER_IP and
 161 DEPLOY_SCENARIO) to automatically determine the valid test cases; for each given
 162 environment.
 163
 164 In the Colorado OPNFV System release a convenience Functest CLI utility is also
 165 introduced to simplify setting up the Functest evironment, management of the
 166 OpenStack environment (e.g. resource clean-up) and for executing tests.
 167 The Functest CLI organised the testcase into logical Tiers, which contain in
 168 turn one or more testcases. The CLI allow execution of a single specified
 169 testcase, all test cases in a specified Tier, or the special case of execution
 170 of **ALL** testcases. The Functest CLI is introduced in more detail in the
 171 section `Executing the functest suites`_ of this document.
 172
 173 .. _`[2]`: http://docs.openstack.org/developer/tempest/overview.html
 174 .. _`[3]`: https://rally.readthedocs.org/en/latest/index.html
 175 .. _`Doctor User Guide`: http://artifacts.opnfv.org/opnfvdocs/brahmaputra/docs/userguide/featureusage-doctor.html
 176 .. _`Promise User Guide`: http://artifacts.opnfv.org/promise/brahmaputra/docs/userguide/index.html
 177 .. _`ONOSFW User Guide`: http://artifacts.opnfv.org/onosfw/brahmaputra/docs/userguide/index.html
 178 .. _`SDNVPN User Guide`: http://artifacts.opnfv.org/sdnvpn/brahmaputra/docs/userguide/featureusage.html
 179 .. _`Functest Dashboard`: http://testresults.opnfv.org/dashboard/
 180
 181