docs/results/overview.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6 =====================
   7 Yardstick Test Report
   8 =====================
   9
  10 .. toctree::
  11    :maxdepth: 2
  12
  13 Introduction
  14 ============
  15
  16 Document Identifier
  17 -------------------
  18
  19 This document is part of deliverables of the OPNFV release brahmaputra.3.0
  20
  21 Scope
  22 -----
  23
  24 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  25 .. _Jenkins: https://build.opnfv.org/ci/view/yardstick/
  26 .. _Scenarios: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-scenarios
  27
  28 This document provides an overview of the results of test cases developed by
  29 the OPNFV Yardstick Project, executed on OPNFV community labs.
  30
  31 OPNFV Continous Integration provides automated build, deploy and testing for
  32 the software developed in OPNFV. Unless stated, the reported tests are
  33 automated via Jenkins Jobs.
  34
  35 Test results are visible in the following dashboard:
  36
  37 * *Yardstick* Dashboard_:  uses influx DB to store test results and Grafana for
  38    visualization (user: opnfv/ password: opnfv)
  39
  40
  41 References
  42 ----------
  43
  44 * IEEE Std 829-2008. "Standard for Software and System Test Documentation".
  45
  46 * OPNFV Brahamputra release note for Yardstick.
  47
  48
  49
  50 General
  51 =======
  52
  53 Yardstick Test Cases have been executed for scenarios and features defined in
  54 this OPNFV release.
  55
  56 The test environments were installed by one of the following: Apex, Compass,
  57 Fuel or Joid; one single installer per POD.
  58
  59 The results of executed tests are available in Dashboard_ and all logs stored
  60 in Jenkins_.
  61
  62 After one week of measurments, in general, SDN ONOS showed lower latency than
  63 SDN ODL, which showed lower latency than an environment installed with pure
  64 OpenStack. Additional time and PODs make this a non-conclusive statement, see
  65 Scenarios_ for a snapshot and Dashboard_ for complete results.
  66
  67 It was not possible to execute the entire Yardstick test cases suite on the
  68 PODs assigned for release verification over a longer period of time, due to
  69 continuous work on the software components and blocking faults either on
  70 environment, feature or test framework.
  71
  72 Four consecutive successful runs was defined as criteria for release.
  73 It is a recommendation to run Yardstick test cases over a longer period
  74 of time in order to better understand the behavior of the system.
  75
  76
  77
  78 Document change procedures and history
  79 --------------------------------------
  80
  81 +--------------------------------------+--------------------------------------+
  82 | **Project**                          | Yardstick                            |
  83 |                                      |                                      |
  84 +--------------------------------------+--------------------------------------+
  85 | **Repo/tag**                         | yardstick/brahmaputra.3.0            |
  86 |                                      |                                      |
  87 +--------------------------------------+--------------------------------------+
  88 | **Release designation**              | Brahmaputra                          |
  89 |                                      |                                      |
  90 +--------------------------------------+--------------------------------------+
  91 | **Release date**                     | Apr 27th, 2016                       |
  92 |                                      |                                      |
  93 +--------------------------------------+--------------------------------------+
  94 | **Purpose of the delivery**          | OPNFV Brahmaputra release test       |
  95 |                                      | results.                             |
  96 |                                      |                                      |
  97 +--------------------------------------+--------------------------------------+