1 .. http://creativecommons.org/licenses/by/4.0
6 An automatic reporting page has been created in order to provide a consistent
7 view of the Functest tests on the different scenarios.
9 In this page, each scenario is evaluated according to test criteria.
11 The results are collected from the centralized database every day and, per
12 scenario. A score is calculated based on the results from the last 10 days.
13 This score is the addition of single test scores. Each test case has a success
14 criteria reflected in the criteria field from the results.
16 As an illustration, let's consider the scenario
17 os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the
18 scores of all the runnable tests from the categories (tiers, healthcheck, smoke
19 and features) corresponding to this scenario.
21 +---------------------+---------+---------+---------+---------+
22 | Test | Apex | Compass | Fuel | Joid |
23 +=====================+=========+=========+=========+=========+
24 | vPing_ssh | X | X | X | X |
25 +---------------------+---------+---------+---------+---------+
26 | vPing_userdata | X | X | X | X |
27 +---------------------+---------+---------+---------+---------+
28 | tempest_smoke | X | X | X | X |
29 +---------------------+---------+---------+---------+---------+
30 | rally_sanity | X | X | X | X |
31 +---------------------+---------+---------+---------+---------+
32 | odl | X | X | X | X |
33 +---------------------+---------+---------+---------+---------+
34 | promise | | | X | X |
35 +---------------------+---------+---------+---------+---------+
36 | doctor | X | | X | |
37 +---------------------+---------+---------+---------+---------+
38 | security_scan | X | | | |
39 +---------------------+---------+---------+---------+---------+
41 +---------------------+---------+---------+---------+---------+
42 | copper | X | | | X |
43 +---------------------+---------+---------+---------+---------+
45 src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix
48 All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature
50 Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration
51 addons and, as a consequence, specific scenario.
52 There are not considered as runnable on the generic odl_l2 scenario.
55 If no result is available or if all the results are failed, the test case get 0
57 If it was successful at least once but not anymore during the 4 runs, the case
58 get 1 point (it worked once).
59 If at least 3 of the last 4 runs were successful, the case get 2 points.
60 If the last 4 runs of the test are successful, the test get 3 points.
62 In the example above, the target score for fuel/os-odl_l2-nofeature-ha is
63 3 x 8 = 24 points and for compass it is 3 x 5 = 15 points .
65 The scenario is validated per installer when we got 3 points for all individual
66 test cases (e.g 24/24 for fuel, 15/15 for compass).
68 Please note that complex or long duration tests are not considered yet for the
69 scoring. In fact the success criteria are not always easy to define and may
70 require specific hardware configuration.
72 Please also note that all the test cases have the same "weight" for the score
73 calculation whatever the complexity of the test case. Concretely a vping has
74 the same weight than the 200 tempest tests.
75 Moreover some installers support more features than others. The more cases your
76 scenario is dealing with, the most difficult to rich a good scoring.
78 Therefore the scoring provides 3 types of indicators:
80 * the richness of the scenario: if the target scoring is high, it means that
81 the scenario includes lots of features
82 * the maturity: if the percentage (scoring/target scoring * 100) is high, it
83 means that all the tests are PASS
84 * the stability: as the number of iteration is included in the calculation,
85 the pecentage can be high only if the scenario is run regularly (at least
86 more than 4 iterations over the last 10 days in CI)
88 In any case, the scoring is used to give feedback to the other projects and
89 does not represent an absolute value of the scenario.
91 See `reporting page`_ for details. For the status, click on the version,
92 Functest then the Status menu.
94 .. _`reporting page`: http://testresults.opnfv.org/reporting/
96 .. figure:: ../../../images/functest-reporting-status.png
98 :alt: Functest reporting portal Fuel status page