docs/testing/user/userguide/test_results.rst

   1 .. SPDX-License-Identifier: CC-BY-4.0
   2
   3 Test results
   4 ============
   5
   6 Manual testing
   7 --------------
   8
   9 In manual mode test results are displayed in the console and result files
  10 are put in /home/opnfv/functest/results.
  11
  12 If you want additional logs, you may configure the logging.ini under
  13 /usr/lib/python3.8/site-packages/xtesting/ci.
  14
  15 Automated testing
  16 -----------------
  17
  18 In automated mode, tests are run within split Alpine containers, and test
  19 results are displayed in jenkins logs. The result summary is provided at the
  20 end of each suite and can be described as follow.
  21
  22 Healthcheck suite::
  23
  24   +--------------------------+------------------+---------------------+------------------+----------------+
  25   |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
  26   +--------------------------+------------------+---------------------+------------------+----------------+
  27   |     connection_check     |     functest     |     healthcheck     |      00:03       |      PASS      |
  28   |      tenantnetwork1      |     functest     |     healthcheck     |      00:05       |      PASS      |
  29   |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
  30   |         vmready1         |     functest     |     healthcheck     |      00:06       |      PASS      |
  31   |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
  32   |        singlevm1         |     functest     |     healthcheck     |      00:32       |      PASS      |
  33   |        singlevm2         |     functest     |     healthcheck     |      00:37       |      PASS      |
  34   |        vping_ssh         |     functest     |     healthcheck     |      00:46       |      PASS      |
  35   |      vping_userdata      |     functest     |     healthcheck     |      00:39       |      PASS      |
  36   |       cinder_test        |     functest     |     healthcheck     |      01:05       |      PASS      |
  37   |      tempest_smoke       |     functest     |     healthcheck     |      05:39       |      PASS      |
  38   |     tempest_horizon      |     functest     |     healthcheck     |      01:05       |      PASS      |
  39   |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
  40   +--------------------------+------------------+---------------------+------------------+----------------+
  41
  42 Smoke suite::
  43
  44   +---------------------------+------------------+---------------+------------------+----------------+
  45   |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
  46   +---------------------------+------------------+---------------+------------------+----------------+
  47   |      tempest_neutron      |     functest     |     smoke     |      15:30       |      PASS      |
  48   |       tempest_cinder      |     functest     |     smoke     |      02:01       |      PASS      |
  49   |      tempest_keystone     |     functest     |     smoke     |      01:17       |      PASS      |
  50   |        tempest_heat       |     functest     |     smoke     |      22:14       |      PASS      |
  51   |     tempest_telemetry     |     functest     |     smoke     |      00:00       |      SKIP      |
  52   |        rally_sanity       |     functest     |     smoke     |      17:24       |      PASS      |
  53   |      refstack_compute     |     functest     |     smoke     |      07:03       |      PASS      |
  54   |      refstack_object      |     functest     |     smoke     |      02:09       |      PASS      |
  55   |     refstack_platform     |     functest     |     smoke     |      07:31       |      PASS      |
  56   |        tempest_full       |     functest     |     smoke     |      41:52       |      PASS      |
  57   |      tempest_scenario     |     functest     |     smoke     |      08:42       |      PASS      |
  58   |        tempest_slow       |     functest     |     smoke     |      43:42       |      PASS      |
  59   |          patrole          |     functest     |     smoke     |      02:42       |      PASS      |
  60   |      tempest_barbican     |     functest     |     smoke     |      02:30       |      PASS      |
  61   |      tempest_octavia      |     functest     |     smoke     |      00:00       |      SKIP      |
  62   +---------------------------+------------------+---------------+------------------+----------------+
  63
  64 Smoke CNTT suite::
  65
  66   +-------------------------------+------------------+---------------+------------------+----------------+
  67   |           TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
  68   +-------------------------------+------------------+---------------+------------------+----------------+
  69   |      tempest_neutron_cntt     |     functest     |     smoke     |      11:35       |      PASS      |
  70   |      tempest_cinder_cntt      |     functest     |     smoke     |      01:58       |      PASS      |
  71   |     tempest_keystone_cntt     |     functest     |     smoke     |      01:13       |      PASS      |
  72   |       tempest_heat_cntt       |     functest     |     smoke     |      22:32       |      PASS      |
  73   |       rally_sanity_cntt       |     functest     |     smoke     |      17:16       |      PASS      |
  74   |       tempest_full_cntt       |     functest     |     smoke     |      41:13       |      PASS      |
  75   |     tempest_scenario_cntt     |     functest     |     smoke     |      08:57       |      PASS      |
  76   |       tempest_slow_cntt       |     functest     |     smoke     |      35:58       |      PASS      |
  77   +-------------------------------+------------------+---------------+------------------+----------------+
  78
  79 Benchmarking suite::
  80
  81   +--------------------+------------------+----------------------+------------------+----------------+
  82   |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
  83   +--------------------+------------------+----------------------+------------------+----------------+
  84   |     rally_full     |     functest     |     benchmarking     |      93:03       |      PASS      |
  85   |     rally_jobs     |     functest     |     benchmarking     |      27:05       |      PASS      |
  86   |        vmtp        |     functest     |     benchmarking     |      17:56       |      PASS      |
  87   |       shaker       |     functest     |     benchmarking     |      24:02       |      PASS      |
  88   +--------------------+------------------+----------------------+------------------+----------------+
  89
  90 Benchmarking CNTT suite::
  91
  92   +-------------------------+------------------+----------------------+------------------+----------------+
  93   |        TEST CASE        |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
  94   +-------------------------+------------------+----------------------+------------------+----------------+
  95   |     rally_full_cntt     |     functest     |     benchmarking     |      89:52       |      PASS      |
  96   |     rally_jobs_cntt     |     functest     |     benchmarking     |      19:39       |      PASS      |
  97   |           vmtp          |     functest     |     benchmarking     |      16:59       |      PASS      |
  98   |          shaker         |     functest     |     benchmarking     |      23:43       |      PASS      |
  99   +-------------------------+------------------+----------------------+------------------+----------------+
 100
 101 Vnf suite::
 102
 103   +----------------------+------------------+--------------+------------------+----------------+
 104   |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 105   +----------------------+------------------+--------------+------------------+----------------+
 106   |       cloudify       |     functest     |     vnf      |      05:08       |      PASS      |
 107   |     cloudify_ims     |     functest     |     vnf      |      24:46       |      PASS      |
 108   |       heat_ims       |     functest     |     vnf      |      33:12       |      PASS      |
 109   |     vyos_vrouter     |     functest     |     vnf      |      15:53       |      PASS      |
 110   |       juju_epc       |     functest     |     vnf      |      27:52       |      PASS      |
 111   +----------------------+------------------+--------------+------------------+----------------+
 112
 113 Kubernetes healthcheck suite::
 114
 115   +-------------------+------------------+---------------------+------------------+----------------+
 116   |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
 117   +-------------------+------------------+---------------------+------------------+----------------+
 118   |     k8s_quick     |     functest     |     healthcheck     |      00:18       |      PASS      |
 119   |     k8s_smoke     |     functest     |     healthcheck     |      01:14       |      PASS      |
 120   +-------------------+------------------+---------------------+------------------+----------------+
 121
 122 Kubernetes smoke suite::
 123
 124   +---------------------------+------------------+---------------+------------------+----------------+
 125   |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
 126   +---------------------------+------------------+---------------+------------------+----------------+
 127   |      k8s_conformance      |     functest     |     smoke     |      94:26       |      PASS      |
 128   |     xrally_kubernetes     |     functest     |     smoke     |      13:05       |      PASS      |
 129   +---------------------------+------------------+---------------+------------------+----------------+
 130
 131 Kubernetes security suite::
 132
 133   +---------------------+------------------+------------------+------------------+----------------+
 134   |      TEST CASE      |     PROJECT      |       TIER       |     DURATION     |     RESULT     |
 135   +---------------------+------------------+------------------+------------------+----------------+
 136   |     kube_hunter     |     functest     |     security     |      00:24       |      PASS      |
 137   |      kube_bench     |     functest     |     security     |      00:18       |      PASS      |
 138   +---------------------+------------------+------------------+------------------+----------------+
 139
 140 Kubernetes benchmarking suite::
 141
 142   +--------------------------------+------------------+----------------------+------------------+----------------+
 143   |           TEST CASE            |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
 144   +--------------------------------+------------------+----------------------+------------------+----------------+
 145   |     xrally_kubernetes_full     |     functest     |     benchmarking     |      34:16       |      PASS      |
 146   +--------------------------------+------------------+----------------------+------------------+----------------+
 147
 148 Kubernetes cnf suite::
 149
 150   +-------------------------+------------------+--------------+------------------+----------------+
 151   |        TEST CASE        |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
 152   +-------------------------+------------------+--------------+------------------+----------------+
 153   |     cnf_conformance     |     functest     |     cnf      |      05:30       |      PASS      |
 154   |         k8s_vims        |     functest     |     cnf      |      20:28       |      PASS      |
 155   +-------------------------+------------------+--------------+------------------+----------------+
 156
 157 Results are automatically pushed to the test results database, some additional
 158 result files are pushed to OPNFV artifact web sites.
 159
 160 Based on the results stored in the result database, a `Functest reporting`_
 161 portal is also automatically updated. This portal provides information on the
 162 overall status per scenario and per installer
 163
 164 .. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html