docs/release/release-notes/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
   5  licensed under the terms of the Apache License, Version 2.0.
   6
   7 =======================
   8 Yardstick Release Notes
   9 =======================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/display/yardstick
  15
  16 .. _Dashboard: http://testresults.opnfv.org/grafana/
  17
  18 .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the Gambia release of OPNFV Yardstick.
  25
  26
  27 Version History
  28 ===============
  29 +-------------------+-----------+---------------------------------+
  30 | *Date*            | *Version* | *Comment*                       |
  31 |                   |           |                                 |
  32 +-------------------+-----------+---------------------------------+
  33 | November 9, 2018  | 7.0.0     | Yardstick for Gambia release    |
  34 |                   |           |                                 |
  35 +-------------------+-----------+---------------------------------+
  36 | December 14, 2018 | 7.1.0     | Yardstick for Gambia release    |
  37 |                   |           |                                 |
  38 +-------------------+-----------+---------------------------------+
  39
  40
  41 Important Notes
  42 ===============
  43
  44 The software delivered in the OPNFV Yardstick_ Project, comprising the
  45 *Yardstick framework*, and the *Yardstick test cases* is a realization of
  46 the methodology in ETSI-ISG NFV-TST001_.
  47
  48 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  49 independent.
  50
  51 OPNFV Gambia Release
  52 ====================
  53
  54 This Gambia release provides *Yardstick* as a framework for NFVI testing
  55 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  56
  57 * Documentation generated with Sphinx
  58
  59   * User Guide
  60   * Developer Guide
  61   * Release notes (this document)
  62   * Results
  63
  64 * Automated Yardstick test suite (daily, weekly)
  65
  66   * Jenkins Jobs for OPNFV community labs
  67
  68 * Automated Yardstick test results visualization
  69
  70   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  71     backend
  72
  73 * Yardstick framework source code
  74
  75 * Yardstick test cases yaml files
  76
  77 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  78
  79 For Gambia release, the *Yardstick framework* is used for the following
  80 testing:
  81
  82 * OPNFV platform testing - generic test cases to measure the categories:
  83
  84   * Compute
  85   * Network
  86   * Storage
  87
  88 * OPNFV platform network service benchmarking (NSB)
  89
  90   * NSB
  91
  92 * Test cases for the following OPNFV Projects:
  93
  94   * Container4NFV
  95   * High Availability
  96   * IPv6
  97   * KVM
  98   * Parser
  99   * StorPerf
 100   * VSperf
 101
 102 The *Yardstick framework* is developed in the OPNFV community, by the
 103 Yardstick_ team.
 104
 105 .. note:: The test case description template used for the Yardstick test cases
 106   is based on the document ETSI-ISG NFV-TST001_; the results report template
 107   used for the Yardstick results is based on the IEEE Std 829-2008.
 108
 109
 110 Release Data
 111 ============
 112
 113 +--------------------------------+-----------------------+
 114 | **Project**                    | Yardstick             |
 115 |                                |                       |
 116 +--------------------------------+-----------------------+
 117 | **Repo/tag**                   | yardstick/opnfv-7.1.0 |
 118 |                                |                       |
 119 +--------------------------------+-----------------------+
 120 | **Yardstick Docker image tag** | opnfv-7.1.0           |
 121 |                                |                       |
 122 +--------------------------------+-----------------------+
 123 | **Release designation**        | Gambia 7.1            |
 124 |                                |                       |
 125 +--------------------------------+-----------------------+
 126 | **Release date**               | December 14, 2018     |
 127 |                                |                       |
 128 +--------------------------------+-----------------------+
 129 | **Purpose of the delivery**    | OPNFV Gambia 7.1.0    |
 130 |                                |                       |
 131 +--------------------------------+-----------------------+
 132
 133
 134 Deliverables
 135 ============
 136
 137 Documents
 138 ---------
 139
 140  - User Guide: :ref:`<yardstick:userguide>`
 141
 142  - Developer Guide: :ref:`<yardstick:devguide>`
 143
 144
 145 Software Deliverables
 146 ---------------------
 147
 148  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)
 149
 150 List of Contexts
 151 ^^^^^^^^^^^^^^^^
 152
 153 +--------------+-------------------------------------------+
 154 | **Context**  | **Description**                           |
 155 |              |                                           |
 156 +--------------+-------------------------------------------+
 157 | *Heat*       | Models orchestration using OpenStack Heat |
 158 |              |                                           |
 159 +--------------+-------------------------------------------+
 160 | *Node*       | Models Baremetal, Controller, Compute     |
 161 |              |                                           |
 162 +--------------+-------------------------------------------+
 163 | *Standalone* | Models VM running on Non-Managed NFVi     |
 164 |              |                                           |
 165 +--------------+-------------------------------------------+
 166 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 167 |              |                                           |
 168 +--------------+-------------------------------------------+
 169
 170
 171 List of Runners
 172 ^^^^^^^^^^^^^^^
 173
 174 .. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC".
 175
 176 +----------------+-------------------------------------------------------+
 177 | **Runner**     | **Description**                                       |
 178 |                |                                                       |
 179 +----------------+-------------------------------------------------------+
 180 | *Arithmetic*   | Steps every run arithmetically according to specified |
 181 |                | input value                                           |
 182 |                |                                                       |
 183 +----------------+-------------------------------------------------------+
 184 | *Duration*     | Runs for a specified period of time                   |
 185 |                |                                                       |
 186 +----------------+-------------------------------------------------------+
 187 | *Iteration*    | Runs for a specified number of iterations             |
 188 |                |                                                       |
 189 +----------------+-------------------------------------------------------+
 190 | *IterationIPC* | Runs a configurable number of times before it         |
 191 |                | returns. Each iteration has a configurable timeout.   |
 192 |                |                                                       |
 193 +----------------+-------------------------------------------------------+
 194 | *Sequence*     | Selects input value to a scenario from an input file  |
 195 |                | and runs all entries sequentially                     |
 196 |                |                                                       |
 197 +----------------+-------------------------------------------------------+
 198 | *Dynamictp*    | A runner that searches for the max throughput with    |
 199 |                | binary search                                         |
 200 |                |                                                       |
 201 +----------------+-------------------------------------------------------+
 202 | *Search*       | A runner that runs a specific time before it returns  |
 203 |                |                                                       |
 204 +----------------+-------------------------------------------------------+
 205
 206
 207 List of Scenarios
 208 ^^^^^^^^^^^^^^^^^
 209
 210 +----------------+-----------------------------------------------------+
 211 | **Category**   | **Delivered**                                       |
 212 |                |                                                     |
 213 +----------------+-----------------------------------------------------+
 214 | *Availability* | Attacker:                                           |
 215 |                |                                                     |
 216 |                | * baremetal, process                                |
 217 |                |                                                     |
 218 |                | HA tools:                                           |
 219 |                |                                                     |
 220 |                | * check host, openstack, process, service           |
 221 |                | * kill process                                      |
 222 |                | * start/stop service                                |
 223 |                |                                                     |
 224 |                | Monitor:                                            |
 225 |                |                                                     |
 226 |                | * command, process                                  |
 227 |                |                                                     |
 228 +----------------+-----------------------------------------------------+
 229 | *Compute*      | * cpuload                                           |
 230 |                | * cyclictest                                        |
 231 |                | * lmbench                                           |
 232 |                | * lmbench_cache                                     |
 233 |                | * perf                                              |
 234 |                | * unixbench                                         |
 235 |                | * ramspeed                                          |
 236 |                | * cachestat                                         |
 237 |                | * memeoryload                                       |
 238 |                | * computecapacity                                   |
 239 |                | * SpecCPU2006                                       |
 240 |                |                                                     |
 241 +----------------+-----------------------------------------------------+
 242 | *Networking*   | * iperf3                                            |
 243 |                | * netperf                                           |
 244 |                | * netperf_node                                      |
 245 |                | * ping                                              |
 246 |                | * ping6                                             |
 247 |                | * pktgen                                            |
 248 |                | * sfc                                               |
 249 |                | * sfc with tacker                                   |
 250 |                | * networkcapacity                                   |
 251 |                | * netutilization                                    |
 252 |                | * nstat                                             |
 253 |                | * pktgenDPDK                                        |
 254 |                |                                                     |
 255 +----------------+-----------------------------------------------------+
 256 | *Parser*       | Tosca2Heat                                          |
 257 |                |                                                     |
 258 +----------------+-----------------------------------------------------+
 259 | *Storage*      | * fio                                               |
 260 |                | * bonnie++                                          |
 261 |                | * storagecapacity                                   |
 262 |                |                                                     |
 263 +----------------+-----------------------------------------------------+
 264 | *StorPerf*     | storperf                                            |
 265 |                |                                                     |
 266 +----------------+-----------------------------------------------------+
 267 | *NSB*          | vFW thoughput test case                             |
 268 |                |                                                     |
 269 +----------------+-----------------------------------------------------+
 270
 271
 272 New Test cases
 273 --------------
 274
 275 .. note:: Yardstick Gambia 7.1.0 adds no new test cases.
 276
 277 * Generic NFVI test cases
 278
 279  * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
 280
 281 * HA Test cases
 282
 283  * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
 284
 285
 286 Version Change
 287 ==============
 288
 289 Module Version Changes
 290 ----------------------
 291
 292 This is the seventh tracked release of Yardstick. It is based on following
 293 upstream versions:
 294
 295 - OpenStack Queens
 296 - OpenDayLight Oxygen
 297
 298
 299 Document Version Changes
 300 ------------------------
 301
 302 This is the seventh tracked version of the Yardstick framework in OPNFV.
 303 It includes the following documentation updates:
 304
 305 - Yardstick User Guide:
 306
 307   - Remove vTC chapter;
 308
 309 - Yardstick Developer Guide
 310 - Yardstick Release Notes for Yardstick: this document
 311
 312
 313 Feature additions
 314 -----------------
 315
 316 - Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)....
 317 - Spirent support.
 318 - vEPC testcases.
 319 - Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
 320 - PROX enhancements and the addition of Standalone test case using SRIOV and
 321   OVS-DPDK.
 322 - Ixia enhancements for vBNG and PPPoE traffic.
 323 - Improvements of unit tests and gating.
 324 - Add DPDK pktgen traffic generator.
 325 - Kubernetes context enhancements.
 326 - Kubernetes sample test cases of fio and lmbench added.
 327
 328
 329 Scenario Matrix
 330 ===============
 331
 332 For Gambia 7.1.0, Yardstick was tested on the following scenarios:
 333
 334 +-------------------------+------+---------+----------+------+
 335 |        Scenario         | Apex | Compass | Fuel-arm | Fuel |
 336 +=========================+======+=========+==========+======+
 337 | os-nosdn-nofeature-noha |   X  |         |          |      |
 338 +-------------------------+------+---------+----------+------+
 339 | os-nosdn-nofeature-ha   |   X  |         |          |      |
 340 +-------------------------+------+---------+----------+------+
 341 | os-odl-bgpvpn-noha      |   X  |         |          |      |
 342 +-------------------------+------+---------+----------+------+
 343 | os-nosdn-calipso-noha   |   X  |         |          |      |
 344 +-------------------------+------+---------+----------+------+
 345 | os-nosdn-kvm-ha         |      |    X    |          |      |
 346 +-------------------------+------+---------+----------+------+
 347 | os-odl-nofeature-ha     |      |         |     X    |   X  |
 348 +-------------------------+------+---------+----------+------+
 349 | os-odl-sfc-noha         |   X  |         |          |      |
 350 +-------------------------+------+---------+----------+------+
 351 | os-nosdn-ovs-ha         |      |         |          |   X  |
 352 +-------------------------+------+---------+----------+------+
 353 | k8-nosdn-nofeature-ha   |      |    X    |          |      |
 354 +-------------------------+------+---------+----------+------+
 355 | k8-nosdn-stor4nfv-noha  |      |    X    |          |      |
 356 +-------------------------+------+---------+----------+------+
 357 | k8-nosdn-stor4nfv-ha    |      |    X    |          |      |
 358 +-------------------------+------+---------+----------+------+
 359
 360
 361 Test results
 362 ============
 363
 364 Test results are available in:
 365
 366  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 367
 368
 369 Known Issues/Faults
 370 -------------------
 371
 372
 373 Corrected Faults
 374 ----------------
 375
 376 Gambia 7.1.0:
 377
 378 +--------------------+--------------------------------------------------------------------------+
 379 | **JIRA REFERENCE** |                             **DESCRIPTION**                              |
 380 +====================+==========================================================================+
 381 | YARDSTICK-1241     | Update NSB PROX devguide.                                                |
 382 +--------------------+--------------------------------------------------------------------------+
 383 | YARDSTICK-1458     | NSB NFVi PROX Should report realtime port activity not historical data.  |
 384 +--------------------+--------------------------------------------------------------------------+
 385 | YARDSTICK-1471     | Add Testcase Prox Standalone SRIOV.                                      |
 386 +--------------------+--------------------------------------------------------------------------+
 387 | YARDSTICK-1475     | Adding Testcase for Prox Stanalone OvS-DPDK.                             |
 388 +--------------------+--------------------------------------------------------------------------+
 389 | YARDSTICK-1500     | Adding Testcase for Prox L2FWD PktTouch Stanalone OvS-DPDK.              |
 390 +--------------------+--------------------------------------------------------------------------+
 391 | YARDSTICK-1517     | Missing opnfv "os-ovn-nofeature-ha" scenario test suite.                 |
 392 +--------------------+--------------------------------------------------------------------------+
 393 | YARDSTICK-l526     | Run testcase 074 result overridden by job status.                        |
 394 +--------------------+--------------------------------------------------------------------------+
 395 | YARDSTICK-1547     | Adding scale up test case for l3fwd OvS-DPDK.                            |
 396 +--------------------+--------------------------------------------------------------------------+
 397 | YARDSTICK-1560     | Fix pip environment.                                                     |
 398 +--------------------+--------------------------------------------------------------------------+
 399 | YARDSTICK-1561     | L3FWD Gradana Dashboards Out-of-date and incorrect.                      |
 400 +--------------------+--------------------------------------------------------------------------+
 401
 402 Gambia 7.1.0 known restrictions/issues
 403 ======================================
 404
 405
 406 Useful links
 407 ============
 408
 409  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 410
 411  - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia
 412
 413  - Yardstick repo: https://git.opnfv.org/yardstick
 414
 415  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 416
 417  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 418
 419  - Yardstick IRC channel: #opnfv-yardstick