docs/release/release-notes/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
   5  licensed under the terms of the Apache License, Version 2.0.
   6
   7 =======================
   8 Yardstick Release Notes
   9 =======================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/display/yardstick
  15
  16 .. _Dashboard: http://testresults.opnfv.org/grafana/
  17
  18 .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the Iruya release of OPNFV Yardstick.
  25
  26
  27 Version History
  28 ===============
  29 +-------------------+-----------+---------------------------------+
  30 | *Date*            | *Version* | *Comment*                       |
  31 |                   |           |                                 |
  32 +-------------------+-----------+---------------------------------+
  33 | Jan 10, 2020      | 9.0.0     | Yardstick for Iruya release     |
  34 |                   |           |                                 |
  35 +-------------------+-----------+---------------------------------+
  36
  37
  38 Important Notes
  39 ===============
  40
  41 The software delivered in the OPNFV Yardstick_ Project, comprising the
  42 *Yardstick framework*, and the *Yardstick test cases* is a realization of
  43 the methodology in ETSI-ISG NFV-TST001_.
  44
  45 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  46 independent.
  47
  48 OPNFV Iruya Release
  49 ====================
  50
  51 This Iruya release provides *Yardstick* as a framework for NFVI testing
  52 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  53
  54 * Documentation generated with Sphinx
  55
  56   * User Guide
  57   * Developer Guide
  58   * Release notes (this document)
  59   * Results
  60
  61 * Automated Yardstick test suite (daily, weekly)
  62
  63   * Jenkins Jobs for OPNFV community labs
  64
  65 * Automated Yardstick test results visualization
  66
  67   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  68     backend
  69
  70 * Yardstick framework source code
  71
  72 * Yardstick test cases yaml files
  73
  74 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  75
  76 For Iruya release, the *Yardstick framework* is used for the following
  77 testing:
  78
  79 * OPNFV platform testing - generic test cases to measure the categories:
  80
  81   * Compute
  82   * Network
  83   * Storage
  84
  85 * OPNFV platform network service benchmarking (NSB)
  86
  87   * NSB
  88
  89 * Test cases for the following OPNFV Projects:
  90
  91   * Container4NFV
  92   * High Availability
  93   * IPv6
  94   * KVM
  95   * Parser
  96   * StorPerf
  97   * VSperf
  98
  99 The *Yardstick framework* is developed in the OPNFV community, by the
 100 Yardstick_ team.
 101
 102 .. note:: The test case description template used for the Yardstick test cases
 103   is based on the document ETSI-ISG NFV-TST001_; the results report template
 104   used for the Yardstick results is based on the IEEE Std 829-2008.
 105
 106
 107 Release Data
 108 ============
 109
 110 +--------------------------------+-----------------------+
 111 | **Project**                    | Yardstick             |
 112 |                                |                       |
 113 +--------------------------------+-----------------------+
 114 | **Repo/tag**                   | yardstick/opnfv-9.0.0 |
 115 |                                |                       |
 116 +--------------------------------+-----------------------+
 117 | **Yardstick Docker image tag** | opnfv-9.0.0           |
 118 |                                |                       |
 119 +--------------------------------+-----------------------+
 120 | **Release designation**        | Iruya 9.0             |
 121 |                                |                       |
 122 +--------------------------------+-----------------------+
 123 | **Release date**               | Jan 10, 2020          |
 124 |                                |                       |
 125 +--------------------------------+-----------------------+
 126 | **Purpose of the delivery**    | OPNFV Iruya 9.0.0     |
 127 |                                |                       |
 128 +--------------------------------+-----------------------+
 129
 130
 131 Deliverables
 132 ============
 133
 134 Documents
 135 ---------
 136
 137  - User Guide: :ref:`<yardstick:userguide>`
 138
 139  - Developer Guide: :ref:`<yardstick:devguide>`
 140
 141
 142 Software Deliverables
 143 ---------------------
 144
 145  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-9.0.0)
 146
 147 List of Contexts
 148 ^^^^^^^^^^^^^^^^
 149
 150 +--------------+-------------------------------------------+
 151 | **Context**  | **Description**                           |
 152 |              |                                           |
 153 +--------------+-------------------------------------------+
 154 | *Heat*       | Models orchestration using OpenStack Heat |
 155 |              |                                           |
 156 +--------------+-------------------------------------------+
 157 | *Node*       | Models Baremetal, Controller, Compute     |
 158 |              |                                           |
 159 +--------------+-------------------------------------------+
 160 | *Standalone* | Models VM running on Non-Managed NFVi     |
 161 |              |                                           |
 162 +--------------+-------------------------------------------+
 163 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 164 |              |                                           |
 165 +--------------+-------------------------------------------+
 166
 167
 168 List of Runners
 169 ^^^^^^^^^^^^^^^
 170
 171 +----------------+-------------------------------------------------------+
 172 | **Runner**     | **Description**                                       |
 173 |                |                                                       |
 174 +----------------+-------------------------------------------------------+
 175 | *Arithmetic*   | Steps every run arithmetically according to specified |
 176 |                | input value                                           |
 177 |                |                                                       |
 178 +----------------+-------------------------------------------------------+
 179 | *Duration*     | Runs for a specified period of time                   |
 180 |                |                                                       |
 181 +----------------+-------------------------------------------------------+
 182 | *Iteration*    | Runs for a specified number of iterations             |
 183 |                |                                                       |
 184 +----------------+-------------------------------------------------------+
 185 | *IterationIPC* | Runs a configurable number of times before it         |
 186 |                | returns. Each iteration has a configurable timeout.   |
 187 |                |                                                       |
 188 +----------------+-------------------------------------------------------+
 189 | *Sequence*     | Selects input value to a scenario from an input file  |
 190 |                | and runs all entries sequentially                     |
 191 |                |                                                       |
 192 +----------------+-------------------------------------------------------+
 193 | *Dynamictp*    | A runner that searches for the max throughput with    |
 194 |                | binary search                                         |
 195 |                |                                                       |
 196 +----------------+-------------------------------------------------------+
 197 | *Search*       | A runner that runs a specific time before it returns  |
 198 |                |                                                       |
 199 +----------------+-------------------------------------------------------+
 200
 201
 202 List of Scenarios
 203 ^^^^^^^^^^^^^^^^^
 204
 205 +----------------+-----------------------------------------------------+
 206 | **Category**   | **Delivered**                                       |
 207 |                |                                                     |
 208 +----------------+-----------------------------------------------------+
 209 | *Availability* | Attacker:                                           |
 210 |                |                                                     |
 211 |                | * baremetal, process                                |
 212 |                |                                                     |
 213 |                | HA tools:                                           |
 214 |                |                                                     |
 215 |                | * check host, openstack, process, service           |
 216 |                | * kill process                                      |
 217 |                | * start/stop service                                |
 218 |                |                                                     |
 219 |                | Monitor:                                            |
 220 |                |                                                     |
 221 |                | * command, process                                  |
 222 |                |                                                     |
 223 +----------------+-----------------------------------------------------+
 224 | *Compute*      | * cpuload                                           |
 225 |                | * cyclictest                                        |
 226 |                | * lmbench                                           |
 227 |                | * lmbench_cache                                     |
 228 |                | * perf                                              |
 229 |                | * unixbench                                         |
 230 |                | * ramspeed                                          |
 231 |                | * cachestat                                         |
 232 |                | * memeoryload                                       |
 233 |                | * computecapacity                                   |
 234 |                | * SpecCPU2006                                       |
 235 |                |                                                     |
 236 +----------------+-----------------------------------------------------+
 237 | *Networking*   | * iperf3                                            |
 238 |                | * netperf                                           |
 239 |                | * netperf_node                                      |
 240 |                | * ping                                              |
 241 |                | * ping6                                             |
 242 |                | * pktgen                                            |
 243 |                | * sfc                                               |
 244 |                | * sfc with tacker                                   |
 245 |                | * networkcapacity                                   |
 246 |                | * netutilization                                    |
 247 |                | * nstat                                             |
 248 |                | * pktgenDPDK                                        |
 249 |                |                                                     |
 250 +----------------+-----------------------------------------------------+
 251 | *Parser*       | Tosca2Heat                                          |
 252 |                |                                                     |
 253 +----------------+-----------------------------------------------------+
 254 | *Storage*      | * fio                                               |
 255 |                | * bonnie++                                          |
 256 |                | * storagecapacity                                   |
 257 |                |                                                     |
 258 +----------------+-----------------------------------------------------+
 259 | *StorPerf*     | storperf                                            |
 260 |                |                                                     |
 261 +----------------+-----------------------------------------------------+
 262 | *NSB*          | vFW thoughput test case                             |
 263 |                |                                                     |
 264 +----------------+-----------------------------------------------------+
 265
 266
 267 New Test cases
 268 --------------
 269
 270 opnfv_yardstick_tc015: Processing speed with impact on energy consumption
 271 and CPU load.
 272
 273 The purpose of TC015 is to evaluate the IaaS compute performance with
 274 regards to CPU processing speed with its impact on the energy consumption.
 275 It measures score of single cpu running and parallel running. Energy
 276 consumption and cpu load are monitored while the cpu test is running.
 277 The purpose is also to be able to spot the trends. Test results, graphs
 278 and similar shall be stored for comparison reasons and product evolution
 279 understanding between different OPNFV versions and/or configurations,
 280 different server types.
 281
 282
 283 Version Change
 284 ==============
 285
 286 Module Version Changes
 287 ----------------------
 288
 289 This is the seventh tracked release of Yardstick. It is based on following
 290 upstream versions:
 291
 292 - OpenStack Stein
 293
 294
 295 Document Version Changes
 296 ------------------------
 297
 298 This is the seventh tracked version of the Yardstick framework in OPNFV.
 299 It includes the following documentation updates:
 300
 301 - Yardstick User Guide:
 302 - Yardstick Developer Guide
 303 - Yardstick Release Notes for Yardstick: this document
 304
 305
 306 Feature additions
 307 -----------------
 308
 309
 310 Scenario Matrix
 311 ===============
 312
 313
 314 Test results
 315 ============
 316
 317 Test results are available in:
 318
 319  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 320
 321
 322 Known Issues/Faults
 323 -------------------
 324
 325
 326 Corrected Faults
 327 ----------------
 328
 329
 330 Iruya 9.0.0 known restrictions/issues
 331 ======================================
 332
 333
 334 Useful links
 335 ============
 336
 337  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 338
 339  - wiki Yardstick Iruya release planning page: https://wiki.opnfv.org/display/yardstick/Release+Iruya
 340
 341  - Yardstick repo: https://git.opnfv.org/yardstick
 342
 343  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 344
 345  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 346
 347  - Yardstick IRC channel: #opnfv-yardstick