docs/release/release-notes/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
   5  licensed under the terms of the Apache License, Version 2.0.
   6
   7 =======================
   8 Yardstick Release Notes
   9 =======================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/display/yardstick
  15
  16 .. _Dashboard: http://testresults.opnfv.org/grafana/
  17
  18 .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the Gambia release of OPNFV Yardstick.
  25
  26
  27 Version History
  28 ===============
  29 +-------------------+-----------+---------------------------------+
  30 | *Date*            | *Version* | *Comment*                       |
  31 |                   |           |                                 |
  32 +-------------------+-----------+---------------------------------+
  33 | November 9, 2018  | 7.0.0     | Yardstick for Gambia release    |
  34 |                   |           |                                 |
  35 +-------------------+-----------+---------------------------------+
  36 | December 14, 2018 | 7.1.0     | Yardstick for Gambia release    |
  37 |                   |           |                                 |
  38 +-------------------+-----------+---------------------------------+
  39 | January 25, 2019  | 7.2.0     | Yardstick for Gambia release    |
  40 |                   |           |                                 |
  41 +-------------------+-----------+---------------------------------+
  42
  43
  44 Important Notes
  45 ===============
  46
  47 The software delivered in the OPNFV Yardstick_ Project, comprising the
  48 *Yardstick framework*, and the *Yardstick test cases* is a realization of
  49 the methodology in ETSI-ISG NFV-TST001_.
  50
  51 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  52 independent.
  53
  54 OPNFV Gambia Release
  55 ====================
  56
  57 This Gambia release provides *Yardstick* as a framework for NFVI testing
  58 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  59
  60 * Documentation generated with Sphinx
  61
  62   * User Guide
  63   * Developer Guide
  64   * Release notes (this document)
  65   * Results
  66
  67 * Automated Yardstick test suite (daily, weekly)
  68
  69   * Jenkins Jobs for OPNFV community labs
  70
  71 * Automated Yardstick test results visualization
  72
  73   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  74     backend
  75
  76 * Yardstick framework source code
  77
  78 * Yardstick test cases yaml files
  79
  80 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  81
  82 For Gambia release, the *Yardstick framework* is used for the following
  83 testing:
  84
  85 * OPNFV platform testing - generic test cases to measure the categories:
  86
  87   * Compute
  88   * Network
  89   * Storage
  90
  91 * OPNFV platform network service benchmarking (NSB)
  92
  93   * NSB
  94
  95 * Test cases for the following OPNFV Projects:
  96
  97   * Container4NFV
  98   * High Availability
  99   * IPv6
 100   * KVM
 101   * Parser
 102   * StorPerf
 103   * VSperf
 104
 105 The *Yardstick framework* is developed in the OPNFV community, by the
 106 Yardstick_ team.
 107
 108 .. note:: The test case description template used for the Yardstick test cases
 109   is based on the document ETSI-ISG NFV-TST001_; the results report template
 110   used for the Yardstick results is based on the IEEE Std 829-2008.
 111
 112
 113 Release Data
 114 ============
 115
 116 +--------------------------------+-----------------------+
 117 | **Project**                    | Yardstick             |
 118 |                                |                       |
 119 +--------------------------------+-----------------------+
 120 | **Repo/tag**                   | yardstick/opnfv-7.2.0 |
 121 |                                |                       |
 122 +--------------------------------+-----------------------+
 123 | **Yardstick Docker image tag** | opnfv-7.2.0           |
 124 |                                |                       |
 125 +--------------------------------+-----------------------+
 126 | **Release designation**        | Gambia 7.2            |
 127 |                                |                       |
 128 +--------------------------------+-----------------------+
 129 | **Release date**               | January 25, 2019     |
 130 |                                |                       |
 131 +--------------------------------+-----------------------+
 132 | **Purpose of the delivery**    | OPNFV Gambia 7.2.0    |
 133 |                                |                       |
 134 +--------------------------------+-----------------------+
 135
 136
 137 Deliverables
 138 ============
 139
 140 Documents
 141 ---------
 142
 143  - User Guide: :ref:`<yardstick:userguide>`
 144
 145  - Developer Guide: :ref:`<yardstick:devguide>`
 146
 147
 148 Software Deliverables
 149 ---------------------
 150
 151  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)
 152
 153 List of Contexts
 154 ^^^^^^^^^^^^^^^^
 155
 156 +--------------+-------------------------------------------+
 157 | **Context**  | **Description**                           |
 158 |              |                                           |
 159 +--------------+-------------------------------------------+
 160 | *Heat*       | Models orchestration using OpenStack Heat |
 161 |              |                                           |
 162 +--------------+-------------------------------------------+
 163 | *Node*       | Models Baremetal, Controller, Compute     |
 164 |              |                                           |
 165 +--------------+-------------------------------------------+
 166 | *Standalone* | Models VM running on Non-Managed NFVi     |
 167 |              |                                           |
 168 +--------------+-------------------------------------------+
 169 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 170 |              |                                           |
 171 +--------------+-------------------------------------------+
 172
 173
 174 List of Runners
 175 ^^^^^^^^^^^^^^^
 176
 177 .. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC".
 178
 179 +----------------+-------------------------------------------------------+
 180 | **Runner**     | **Description**                                       |
 181 |                |                                                       |
 182 +----------------+-------------------------------------------------------+
 183 | *Arithmetic*   | Steps every run arithmetically according to specified |
 184 |                | input value                                           |
 185 |                |                                                       |
 186 +----------------+-------------------------------------------------------+
 187 | *Duration*     | Runs for a specified period of time                   |
 188 |                |                                                       |
 189 +----------------+-------------------------------------------------------+
 190 | *Iteration*    | Runs for a specified number of iterations             |
 191 |                |                                                       |
 192 +----------------+-------------------------------------------------------+
 193 | *IterationIPC* | Runs a configurable number of times before it         |
 194 |                | returns. Each iteration has a configurable timeout.   |
 195 |                |                                                       |
 196 +----------------+-------------------------------------------------------+
 197 | *Sequence*     | Selects input value to a scenario from an input file  |
 198 |                | and runs all entries sequentially                     |
 199 |                |                                                       |
 200 +----------------+-------------------------------------------------------+
 201 | *Dynamictp*    | A runner that searches for the max throughput with    |
 202 |                | binary search                                         |
 203 |                |                                                       |
 204 +----------------+-------------------------------------------------------+
 205 | *Search*       | A runner that runs a specific time before it returns  |
 206 |                |                                                       |
 207 +----------------+-------------------------------------------------------+
 208
 209
 210 List of Scenarios
 211 ^^^^^^^^^^^^^^^^^
 212
 213 +----------------+-----------------------------------------------------+
 214 | **Category**   | **Delivered**                                       |
 215 |                |                                                     |
 216 +----------------+-----------------------------------------------------+
 217 | *Availability* | Attacker:                                           |
 218 |                |                                                     |
 219 |                | * baremetal, process                                |
 220 |                |                                                     |
 221 |                | HA tools:                                           |
 222 |                |                                                     |
 223 |                | * check host, openstack, process, service           |
 224 |                | * kill process                                      |
 225 |                | * start/stop service                                |
 226 |                |                                                     |
 227 |                | Monitor:                                            |
 228 |                |                                                     |
 229 |                | * command, process                                  |
 230 |                |                                                     |
 231 +----------------+-----------------------------------------------------+
 232 | *Compute*      | * cpuload                                           |
 233 |                | * cyclictest                                        |
 234 |                | * lmbench                                           |
 235 |                | * lmbench_cache                                     |
 236 |                | * perf                                              |
 237 |                | * unixbench                                         |
 238 |                | * ramspeed                                          |
 239 |                | * cachestat                                         |
 240 |                | * memeoryload                                       |
 241 |                | * computecapacity                                   |
 242 |                | * SpecCPU2006                                       |
 243 |                |                                                     |
 244 +----------------+-----------------------------------------------------+
 245 | *Networking*   | * iperf3                                            |
 246 |                | * netperf                                           |
 247 |                | * netperf_node                                      |
 248 |                | * ping                                              |
 249 |                | * ping6                                             |
 250 |                | * pktgen                                            |
 251 |                | * sfc                                               |
 252 |                | * sfc with tacker                                   |
 253 |                | * networkcapacity                                   |
 254 |                | * netutilization                                    |
 255 |                | * nstat                                             |
 256 |                | * pktgenDPDK                                        |
 257 |                |                                                     |
 258 +----------------+-----------------------------------------------------+
 259 | *Parser*       | Tosca2Heat                                          |
 260 |                |                                                     |
 261 +----------------+-----------------------------------------------------+
 262 | *Storage*      | * fio                                               |
 263 |                | * bonnie++                                          |
 264 |                | * storagecapacity                                   |
 265 |                |                                                     |
 266 +----------------+-----------------------------------------------------+
 267 | *StorPerf*     | storperf                                            |
 268 |                |                                                     |
 269 +----------------+-----------------------------------------------------+
 270 | *NSB*          | vFW thoughput test case                             |
 271 |                |                                                     |
 272 +----------------+-----------------------------------------------------+
 273
 274
 275 New Test cases
 276 --------------
 277
 278 .. note:: Yardstick Gambia 7.2.0 adds no new test cases.
 279
 280 * Generic NFVI test cases
 281
 282  * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
 283
 284 * HA Test cases
 285
 286  * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
 287
 288
 289 Version Change
 290 ==============
 291
 292 Module Version Changes
 293 ----------------------
 294
 295 This is the seventh tracked release of Yardstick. It is based on following
 296 upstream versions:
 297
 298 - OpenStack Queens
 299 - OpenDayLight Oxygen
 300
 301
 302 Document Version Changes
 303 ------------------------
 304
 305 This is the seventh tracked version of the Yardstick framework in OPNFV.
 306 It includes the following documentation updates:
 307
 308 - Yardstick User Guide:
 309
 310   - Remove vTC chapter;
 311
 312 - Yardstick Developer Guide
 313 - Yardstick Release Notes for Yardstick: this document
 314
 315
 316 Feature additions
 317 -----------------
 318
 319 - Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)....
 320 - Spirent support.
 321 - vEPC testcases.
 322 - Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
 323 - PROX enhancements and the addition of Standalone test case using SRIOV and
 324   OVS-DPDK.
 325 - Ixia enhancements for vBNG and PPPoE traffic.
 326 - Improvements of unit tests and gating.
 327 - Add DPDK pktgen traffic generator.
 328 - Kubernetes context enhancements.
 329 - Kubernetes sample test cases of fio and lmbench added.
 330
 331
 332 Scenario Matrix
 333 ===============
 334
 335 For Gambia 7.2.0, Yardstick was tested on the following scenarios:
 336
 337 +-------------------------+------+---------+----------+------+
 338 |        Scenario         | Apex | Compass | Fuel-arm | Fuel |
 339 +=========================+======+=========+==========+======+
 340 | os-nosdn-nofeature-noha |   X  |         |          |      |
 341 +-------------------------+------+---------+----------+------+
 342 | os-nosdn-nofeature-ha   |   X  |         |          |      |
 343 +-------------------------+------+---------+----------+------+
 344 | os-odl-bgpvpn-noha      |   X  |         |          |      |
 345 +-------------------------+------+---------+----------+------+
 346 | os-nosdn-calipso-noha   |   X  |         |          |      |
 347 +-------------------------+------+---------+----------+------+
 348 | os-nosdn-kvm-ha         |      |    X    |          |      |
 349 +-------------------------+------+---------+----------+------+
 350 | os-odl-nofeature-ha     |      |         |     X    |   X  |
 351 +-------------------------+------+---------+----------+------+
 352 | os-odl-sfc-noha         |   X  |         |          |      |
 353 +-------------------------+------+---------+----------+------+
 354 | os-nosdn-ovs-ha         |      |         |          |   X  |
 355 +-------------------------+------+---------+----------+------+
 356 | k8-nosdn-nofeature-ha   |      |    X    |          |      |
 357 +-------------------------+------+---------+----------+------+
 358 | k8-nosdn-stor4nfv-noha  |      |    X    |          |      |
 359 +-------------------------+------+---------+----------+------+
 360 | k8-nosdn-stor4nfv-ha    |      |    X    |          |      |
 361 +-------------------------+------+---------+----------+------+
 362
 363
 364 Test results
 365 ============
 366
 367 Test results are available in:
 368
 369  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 370
 371
 372 Known Issues/Faults
 373 -------------------
 374
 375
 376 Corrected Faults
 377 ----------------
 378
 379 Gambia 7.2.0:
 380
 381 +--------------------+--------------------------------------------------------------------------+
 382 | **JIRA REFERENCE** |                             **DESCRIPTION**                              |
 383 +====================+==========================================================================+
 384 | YARDSTICK-1512     | [dovetail] split the sla check results into process recovery and service |
 385 |                    | recovery for HA test cases.                                              |
 386 +--------------------+--------------------------------------------------------------------------+
 387
 388 Gambia 7.2.0 known restrictions/issues
 389 ======================================
 390
 391
 392 Useful links
 393 ============
 394
 395  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 396
 397  - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia
 398
 399  - Yardstick repo: https://git.opnfv.org/yardstick
 400
 401  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 402
 403  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 404
 405  - Yardstick IRC channel: #opnfv-yardstick