docs/results/overview.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6 Yardstick test tesult document overview
   7 =======================================
   8
   9 .. _`Yardstick user guide`: artifacts.opnfv.org/yardstick/docs/userguide/index.html
  10 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  11 .. _Jenkins: https://build.opnfv.org/ci/view/yardstick/
  12 .. _Scenarios: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-scenarios
  13
  14 This document provides an overview of the results of test cases developed by
  15 the OPNFV Yardstick Project, executed on OPNFV community labs.
  16
  17 Yardstick project is described in `Yardstick user guide`_.
  18
  19 Yardstick is run systematically at the end of an OPNFV fresh installation.
  20 The system under test (SUT) is installed by the installer Apex, Compass, Fuel
  21 or Joid on Performance Optimized Datacenter (POD); One single installer per
  22 POD. All the runnable test cases are run sequentially. The installer and the
  23 POD are considered to evaluate whether the test case can be run or not. That is
  24 why all the number of test cases may vary from 1 installer to another and from
  25 1 POD to POD.
  26
  27 OPNFV CI provides automated build, deploy and testing for
  28 the software developed in OPNFV. Unless stated, the reported tests are
  29 automated via Jenkins Jobs. Yardsrick test results from OPNFV Continous
  30 Integration can be found in the following dashboard:
  31
  32 * *Yardstick* Dashboard_:  uses influx DB to store Yardstick CI test results and
  33    Grafana for visualization (user: opnfv/ password: opnfv)
  34
  35 The results of executed test cases are available in Dashboard_ and all logs are
  36 stored in Jenkins_.
  37
  38 It was not possible to execute the entire Yardstick test cases suite on the
  39 PODs assigned for release verification over a longer period of time, due to
  40 continuous work on the software components and blocking faults either on
  41 environment, features or test framework.
  42
  43 The list of scenarios supported by each installer can be described as follows:
  44
  45 +-------------------------+---------+---------+---------+---------+
  46 |         Scenario        |  Apex   | Compass |  Fuel   |   Joid  |
  47 +=========================+=========+=========+=========+=========+
  48 | os-nosdn-nofeature-noha |         |         |    X    |    X    |
  49 +-------------------------+---------+---------+---------+---------+
  50 | os-nosdn-nofeature-ha   |    X    |    X    |    X    |    X    |
  51 +-------------------------+---------+---------+---------+---------+
  52 | os-odl_l2-nofeature-ha  |    X    |    X    |    X    |    X    |
  53 +-------------------------+---------+---------+---------+---------+
  54 | os-odl_l2-nofeature-noha|         |         |    X    |         |
  55 +-------------------------+---------+---------+---------+---------+
  56 | os-odl_l3-nofeature-ha  |    X    |    X    |    X    |         |
  57 +-------------------------+---------+---------+---------+---------+
  58 | os-odl_l3-nofeature-noha|         |         |    X    |         |
  59 +-------------------------+---------+---------+---------+---------+
  60 | os-onos-sfc-ha          |    X    |    X    |    X    |    X    |
  61 +-------------------------+---------+---------+---------+---------+
  62 | os-onos-sfc-noha        |         |         |    X    |         |
  63 +-------------------------+---------+---------+---------+---------+
  64 | os-onos-nofeature-ha    |    X    |    X    |    X    |    X    |
  65 +-------------------------+---------+---------+---------+---------+
  66 | os-onos-nofeature-noha  |         |         |    X    |         |
  67 +-------------------------+---------+---------+---------+---------+
  68 | os-odl_l2-sfc-ha        |         |         |    X    |         |
  69 +-------------------------+---------+---------+---------+---------+
  70 | os-odl_l2-sfc-noha      |    X    |    X    |    X    |         |
  71 +-------------------------+---------+---------+---------+---------+
  72 | os-odl_l2-bgpvpn-ha     |    X    |         |    X    |         |
  73 +-------------------------+---------+---------+---------+---------+
  74 | os-odl_l2-bgpvpn-noha   |         |    X    |    X    |         |
  75 +-------------------------+---------+---------+---------+---------+
  76 | os-nosdn-kvm-ha         |         |         |    X    |         |
  77 +-------------------------+---------+---------+---------+---------+
  78 | os-nosdn-kvm-noha       |         |    X    |    X    |         |
  79 +-------------------------+---------+---------+---------+---------+
  80 | os-nosdn-ovs-ha         |         |         |    X    |         |
  81 +-------------------------+---------+---------+---------+---------+
  82 | os-nosdn-ovs-noha       |    X    |         |    X    |         |
  83 +-------------------------+---------+---------+---------+---------+
  84 | os-ocl-nofeature-ha     |         |         |         |         |
  85 +-------------------------+---------+---------+---------+---------+
  86 | os-nosdn-lxd-ha         |         |         |         |    X    |
  87 +-------------------------+---------+---------+---------+---------+
  88 | os-nosdn-lxd-noha       |         |         |         |    X    |
  89 +-------------------------+---------+---------+---------+---------+
  90 | os-odl_l2-fdio-noha     |    X    |         |         |         |
  91 +-------------------------+---------+---------+---------+---------+
  92 | os-odl_l2-moon-ha       |         |    X    |         |         |
  93 +-------------------------+---------+---------+---------+---------+
  94
  95 To qualify for release, the scenarios must have deployed and been successfully
  96 tested in four consecutive installations to establish stability of deployment
  97 and feature capability. It is a recommendation to run Yardstick test
  98 cases over a longer period of time in order to better understand the behavior
  99 of the system under test.
 100
 101 References
 102 ----------
 103
 104 * IEEE Std 829-2008. "Standard for Software and System Test Documentation".
 105
 106 * OPNFV Colorado release note for Yardstick.