docs/release/results/overview.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6 ..
   7       Convention for heading levels in Yardstick:
   8       =======  Heading 0 (reserved for the title in a document)
   9       -------  Heading 1
  10       ^^^^^^^  Heading 2
  11       +++++++  Heading 3
  12       '''''''  Heading 4
  13       Avoid deeper levels because they do not render well.
  14
  15 Yardstick test tesult document overview
  16 =======================================
  17
  18 .. _`Yardstick user guide`: artifacts.opnfv.org/yardstick/docs/userguide/index.html
  19 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  20 .. _Jenkins: https://build.opnfv.org/ci/view/yardstick/
  21 .. _Scenarios: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-scenarios
  22
  23 This document provides an overview of the results of test cases developed by
  24 the OPNFV Yardstick Project, executed on OPNFV community labs.
  25
  26 Yardstick project is described in `Yardstick user guide`_.
  27
  28 Yardstick is run systematically at the end of an OPNFV fresh installation.
  29 The system under test (SUT) is installed by the installer Apex, Compass, Fuel
  30 or Joid on Performance Optimized Datacenter (POD); One single installer per
  31 POD. All the runnable test cases are run sequentially. The installer and the
  32 POD are considered to evaluate whether the test case can be run or not. That is
  33 why all the number of test cases may vary from 1 installer to another and from
  34 1 POD to POD.
  35
  36 OPNFV CI provides automated build, deploy and testing for
  37 the software developed in OPNFV. Unless stated, the reported tests are
  38 automated via Jenkins Jobs. Yardsrick test results from OPNFV Continous
  39 Integration can be found in the following dashboard:
  40
  41 * *Yardstick* Dashboard_:  uses influx DB to store Yardstick CI test results and
  42    Grafana for visualization (user: opnfv/ password: opnfv)
  43
  44 The results of executed test cases are available in Dashboard_ and all logs are
  45 stored in Jenkins_.
  46
  47 It was not possible to execute the entire Yardstick test cases suite on the
  48 PODs assigned for release verification over a longer period of time, due to
  49 continuous work on the software components and blocking faults either on
  50 environment, features or test framework.
  51
  52 The list of scenarios supported by each installer can be described as follows:
  53
  54 +-------------------------+------+---------+----------+------+------+-------+
  55 |        Scenario         | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy |
  56 +=========================+======+=========+==========+======+======+=======+
  57 | os-nosdn-nofeature-noha |  X   |         |          |      |  X   |       |
  58 +-------------------------+------+---------+----------+------+------+-------+
  59 | os-nosdn-nofeature-ha   |  X   |         |    X     |  X   |  X   |   X   |
  60 +-------------------------+------+---------+----------+------+------+-------+
  61 | os-nosdn-bar-noha       |  X   |         |          |      |      |       |
  62 +-------------------------+------+---------+----------+------+------+-------+
  63 | os-nosdn-bar-ha         |  X   |         |          |      |      |       |
  64 +-------------------------+------+---------+----------+------+------+-------+
  65 | os-odl-bgpvpn-ha        |  X   |         |          |      |      |       |
  66 +-------------------------+------+---------+----------+------+------+-------+
  67 | os-nosdn-calipso-noha   |  X   |         |          |      |      |       |
  68 +-------------------------+------+---------+----------+------+------+-------+
  69 | os-nosdn-kvm-ha         |      |    X    |          |      |      |       |
  70 +-------------------------+------+---------+----------+------+------+-------+
  71 | os-odl_l3-nofeature-ha  |      |    X    |          |      |      |       |
  72 +-------------------------+------+---------+----------+------+------+-------+
  73 | os-odl-sfc-ha           |      |    X    |          |      |      |       |
  74 +-------------------------+------+---------+----------+------+------+-------+
  75 | os-odl-nofeature-ha     |      |         |          |  X   |      |   X   |
  76 +-------------------------+------+---------+----------+------+------+-------+
  77 | os-nosdn-ovs-ha         |      |         |          |  X   |      |       |
  78 +-------------------------+------+---------+----------+------+------+-------+
  79
  80 To qualify for release, the scenarios must have deployed and been successfully
  81 tested in four consecutive installations to establish stability of deployment
  82 and feature capability. It is a recommendation to run Yardstick test
  83 cases over a longer period of time in order to better understand the behavior
  84 of the system under test.
  85
  86 References
  87 ----------
  88
  89 * IEEE Std 829-2008. "Standard for Software and System Test Documentation".
  90
  91 * OPNFV Fraser release note for Yardstick.