Update Xtesting to 0.98
[functest.git] / docs / testing / user / userguide / test_results.rst
1 .. SPDX-License-Identifier: CC-BY-4.0
2
3 Test results
4 ============
5
6 Manual testing
7 --------------
8
9 In manual mode test results are displayed in the console and result files
10 are put in /home/opnfv/functest/results.
11
12 If you want additional logs, you may configure the logging.ini under
13 /usr/lib/python3.8/site-packages/xtesting/ci.
14
15 Automated testing
16 -----------------
17
18 In automated mode, tests are run within split Alpine containers, and test
19 results are displayed in jenkins logs. The result summary is provided at the
20 end of each suite and can be described as follow.
21
22 Healthcheck suite::
23
24   +--------------------------+------------------+---------------------+------------------+----------------+
25   |        TEST CASE         |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
26   +--------------------------+------------------+---------------------+------------------+----------------+
27   |     connection_check     |     functest     |     healthcheck     |      00:03       |      PASS      |
28   |      tenantnetwork1      |     functest     |     healthcheck     |      00:05       |      PASS      |
29   |      tenantnetwork2      |     functest     |     healthcheck     |      00:06       |      PASS      |
30   |         vmready1         |     functest     |     healthcheck     |      00:06       |      PASS      |
31   |         vmready2         |     functest     |     healthcheck     |      00:08       |      PASS      |
32   |        singlevm1         |     functest     |     healthcheck     |      00:32       |      PASS      |
33   |        singlevm2         |     functest     |     healthcheck     |      00:37       |      PASS      |
34   |        vping_ssh         |     functest     |     healthcheck     |      00:46       |      PASS      |
35   |      vping_userdata      |     functest     |     healthcheck     |      00:39       |      PASS      |
36   |       cinder_test        |     functest     |     healthcheck     |      01:05       |      PASS      |
37   |      tempest_smoke       |     functest     |     healthcheck     |      05:39       |      PASS      |
38   |     tempest_horizon      |     functest     |     healthcheck     |      01:05       |      PASS      |
39   |           odl            |     functest     |     healthcheck     |      00:00       |      SKIP      |
40   +--------------------------+------------------+---------------------+------------------+----------------+
41
42 Smoke suite::
43
44   +---------------------------+------------------+---------------+------------------+----------------+
45   |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
46   +---------------------------+------------------+---------------+------------------+----------------+
47   |      tempest_neutron      |     functest     |     smoke     |      15:30       |      PASS      |
48   |       tempest_cinder      |     functest     |     smoke     |      02:01       |      PASS      |
49   |      tempest_keystone     |     functest     |     smoke     |      01:17       |      PASS      |
50   |        tempest_heat       |     functest     |     smoke     |      22:14       |      PASS      |
51   |     tempest_telemetry     |     functest     |     smoke     |      00:00       |      SKIP      |
52   |        rally_sanity       |     functest     |     smoke     |      17:24       |      PASS      |
53   |      refstack_compute     |     functest     |     smoke     |      07:03       |      PASS      |
54   |      refstack_object      |     functest     |     smoke     |      02:09       |      PASS      |
55   |     refstack_platform     |     functest     |     smoke     |      07:31       |      PASS      |
56   |        tempest_full       |     functest     |     smoke     |      41:52       |      PASS      |
57   |      tempest_scenario     |     functest     |     smoke     |      08:42       |      PASS      |
58   |        tempest_slow       |     functest     |     smoke     |      43:42       |      PASS      |
59   |       patrole_admin       |     functest     |     smoke     |      21:06       |      PASS      |
60   |       patrole_member      |     functest     |     smoke     |      21:23       |      PASS      |
61   |       patrole_reader      |     functest     |     smoke     |      21:56       |      PASS      |
62   |      tempest_barbican     |     functest     |     smoke     |      02:30       |      PASS      |
63   |      tempest_octavia      |     functest     |     smoke     |      00:00       |      SKIP      |
64   |       tempest_cyborg      |     functest     |     smoke     |      00:00       |      SKIP      |
65   +---------------------------+------------------+---------------+------------------+----------------+
66
67 Smoke CNTT suite::
68
69   +-------------------------------+------------------+---------------+------------------+----------------+
70   |           TEST CASE           |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
71   +-------------------------------+------------------+---------------+------------------+----------------+
72   |      tempest_neutron_cntt     |     functest     |     smoke     |      11:35       |      PASS      |
73   |      tempest_cinder_cntt      |     functest     |     smoke     |      01:58       |      PASS      |
74   |     tempest_keystone_cntt     |     functest     |     smoke     |      01:13       |      PASS      |
75   |       tempest_heat_cntt       |     functest     |     smoke     |      22:32       |      PASS      |
76   |       rally_sanity_cntt       |     functest     |     smoke     |      17:16       |      PASS      |
77   |       tempest_full_cntt       |     functest     |     smoke     |      41:13       |      PASS      |
78   |     tempest_scenario_cntt     |     functest     |     smoke     |      08:57       |      PASS      |
79   |       tempest_slow_cntt       |     functest     |     smoke     |      35:58       |      PASS      |
80   +-------------------------------+------------------+---------------+------------------+----------------+
81
82 Benchmarking suite::
83
84   +--------------------+------------------+----------------------+------------------+----------------+
85   |     TEST CASE      |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
86   +--------------------+------------------+----------------------+------------------+----------------+
87   |     rally_full     |     functest     |     benchmarking     |      93:03       |      PASS      |
88   |     rally_jobs     |     functest     |     benchmarking     |      27:05       |      PASS      |
89   |        vmtp        |     functest     |     benchmarking     |      17:56       |      PASS      |
90   |       shaker       |     functest     |     benchmarking     |      24:02       |      PASS      |
91   +--------------------+------------------+----------------------+------------------+----------------+
92
93 Benchmarking CNTT suite::
94
95   +-------------------------+------------------+----------------------+------------------+----------------+
96   |        TEST CASE        |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
97   +-------------------------+------------------+----------------------+------------------+----------------+
98   |     rally_full_cntt     |     functest     |     benchmarking     |      89:52       |      PASS      |
99   |     rally_jobs_cntt     |     functest     |     benchmarking     |      19:39       |      PASS      |
100   |           vmtp          |     functest     |     benchmarking     |      16:59       |      PASS      |
101   |          shaker         |     functest     |     benchmarking     |      23:43       |      PASS      |
102   +-------------------------+------------------+----------------------+------------------+----------------+
103
104 Vnf suite::
105
106   +----------------------+------------------+--------------+------------------+----------------+
107   |      TEST CASE       |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
108   +----------------------+------------------+--------------+------------------+----------------+
109   |       cloudify       |     functest     |     vnf      |      05:08       |      PASS      |
110   |     cloudify_ims     |     functest     |     vnf      |      24:46       |      PASS      |
111   |       heat_ims       |     functest     |     vnf      |      33:12       |      PASS      |
112   |     vyos_vrouter     |     functest     |     vnf      |      15:53       |      PASS      |
113   |       juju_epc       |     functest     |     vnf      |      27:52       |      PASS      |
114   +----------------------+------------------+--------------+------------------+----------------+
115
116 Kubernetes healthcheck suite::
117
118   +-------------------+------------------+---------------------+------------------+----------------+
119   |     TEST CASE     |     PROJECT      |         TIER        |     DURATION     |     RESULT     |
120   +-------------------+------------------+---------------------+------------------+----------------+
121   |     k8s_quick     |     functest     |     healthcheck     |      00:18       |      PASS      |
122   |     k8s_smoke     |     functest     |     healthcheck     |      01:14       |      PASS      |
123   +-------------------+------------------+---------------------+------------------+----------------+
124
125 Kubernetes smoke suite::
126
127   +---------------------------+------------------+---------------+------------------+----------------+
128   |         TEST CASE         |     PROJECT      |      TIER     |     DURATION     |     RESULT     |
129   +---------------------------+------------------+---------------+------------------+----------------+
130   |      k8s_conformance      |     functest     |     smoke     |      94:26       |      PASS      |
131   |     xrally_kubernetes     |     functest     |     smoke     |      13:05       |      PASS      |
132   +---------------------------+------------------+---------------+------------------+----------------+
133
134 Kubernetes security suite::
135
136   +---------------------------+------------------+------------------+------------------+----------------+
137   |         TEST CASE         |     PROJECT      |       TIER       |     DURATION     |     RESULT     |
138   +---------------------------+------------------+------------------+------------------+----------------+
139   |        kube_hunter        |     functest     |     security     |      00:19       |      PASS      |
140   |     kube_bench_master     |     functest     |     security     |      00:02       |      PASS      |
141   |      kube_bench_node      |     functest     |     security     |      00:01       |      PASS      |
142   +---------------------------+------------------+------------------+------------------+----------------+
143
144 Kubernetes benchmarking suite::
145
146   +--------------------------------+------------------+----------------------+------------------+----------------+
147   |           TEST CASE            |     PROJECT      |         TIER         |     DURATION     |     RESULT     |
148   +--------------------------------+------------------+----------------------+------------------+----------------+
149   |     xrally_kubernetes_full     |     functest     |     benchmarking     |      34:16       |      PASS      |
150   +--------------------------------+------------------+----------------------+------------------+----------------+
151
152 Kubernetes cnf suite::
153
154   +-------------------------+------------------+--------------+------------------+----------------+
155   |        TEST CASE        |     PROJECT      |     TIER     |     DURATION     |     RESULT     |
156   +-------------------------+------------------+--------------+------------------+----------------+
157   |         k8s_vims        |     functest     |     cnf      |      09:06       |      PASS      |
158   |        helm_vims        |     functest     |     cnf      |      08:54       |      PASS      |
159   |     cnf_conformance     |     functest     |     cnf      |      02:00       |      PASS      |
160   +-------------------------+------------------+--------------+------------------+----------------+
161
162 Results are automatically pushed to the test results database, some additional
163 result files are pushed to OPNFV artifact web sites.
164
165 Based on the results stored in the result database, a `Functest reporting`_
166 portal is also automatically updated. This portal provides information on the
167 overall status per scenario and per installer
168
169 .. _`Functest reporting`: http://testresults.opnfv.org/reporting/master/functest/status-apex.html