docs/release/release-notes/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
   5  licensed under the terms of the Apache License, Version 2.0.
   6
   7 =======================
   8 Yardstick Release Notes
   9 =======================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/display/yardstick
  15
  16 .. _Dashboard: http://testresults.opnfv.org/grafana/
  17
  18 .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the Gambia release of OPNFV Yardstick.
  25
  26
  27 Version History
  28 ===============
  29 +-------------------+-----------+---------------------------------+
  30 | *Date*            | *Version* | *Comment*                       |
  31 |                   |           |                                 |
  32 +-------------------+-----------+---------------------------------+
  33 | November 9, 2018  | 7.0.0     | Yardstick for Gambia release    |
  34 |                   |           |                                 |
  35 +-------------------+-----------+---------------------------------+
  36
  37
  38 Important Notes
  39 ===============
  40
  41 The software delivered in the OPNFV Yardstick_ Project, comprising the
  42 *Yardstick framework*, and the *Yardstick test cases* is a realization of
  43 the methodology in ETSI-ISG NFV-TST001_.
  44
  45 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  46 independent.
  47
  48 OPNFV Gambia Release
  49 ====================
  50
  51 This Gambia release provides *Yardstick* as a framework for NFVI testing
  52 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  53
  54 * Documentation generated with Sphinx
  55
  56   * User Guide
  57   * Developer Guide
  58   * Release notes (this document)
  59   * Results
  60
  61 * Automated Yardstick test suite (daily, weekly)
  62
  63   * Jenkins Jobs for OPNFV community labs
  64
  65 * Automated Yardstick test results visualization
  66
  67   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  68     backend
  69
  70 * Yardstick framework source code
  71
  72 * Yardstick test cases yaml files
  73
  74 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  75
  76 For Gambia release, the *Yardstick framework* is used for the following
  77 testing:
  78
  79 * OPNFV platform testing - generic test cases to measure the categories:
  80
  81   * Compute
  82   * Network
  83   * Storage
  84
  85 * OPNFV platform network service benchmarking (NSB)
  86
  87   * NSB
  88
  89 * Test cases for the following OPNFV Projects:
  90
  91   * Container4NFV
  92   * High Availability
  93   * IPv6
  94   * KVM
  95   * Parser
  96   * StorPerf
  97   * VSperf
  98
  99 The *Yardstick framework* is developed in the OPNFV community, by the
 100 Yardstick_ team.
 101
 102 .. note:: The test case description template used for the Yardstick test cases
 103   is based on the document ETSI-ISG NFV-TST001_; the results report template
 104   used for the Yardstick results is based on the IEEE Std 829-2008.
 105
 106
 107 Release Data
 108 ============
 109
 110 +--------------------------------+-----------------------+
 111 | **Project**                    | Yardstick             |
 112 |                                |                       |
 113 +--------------------------------+-----------------------+
 114 | **Repo/tag**                   | yardstick/opnfv-7.0.0 |
 115 |                                |                       |
 116 +--------------------------------+-----------------------+
 117 | **Yardstick Docker image tag** | opnfv-7.0.0           |
 118 |                                |                       |
 119 +--------------------------------+-----------------------+
 120 | **Release designation**        | Gambia 7.0            |
 121 |                                |                       |
 122 +--------------------------------+-----------------------+
 123 | **Release date**               | November 9, 2018      |
 124 |                                |                       |
 125 +--------------------------------+-----------------------+
 126 | **Purpose of the delivery**    | OPNFV Gambia 7.0.0    |
 127 |                                |                       |
 128 +--------------------------------+-----------------------+
 129
 130
 131 Deliverables
 132 ============
 133
 134 Documents
 135 ---------
 136
 137  - User Guide: :ref:`<yardstick:userguide>`
 138
 139  - Developer Guide: :ref:`<yardstick:devguide>`
 140
 141
 142 Software Deliverables
 143 ---------------------
 144
 145  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-7.0.0)
 146
 147 List of Contexts
 148 ^^^^^^^^^^^^^^^^
 149
 150 +--------------+-------------------------------------------+
 151 | **Context**  | **Description**                           |
 152 |              |                                           |
 153 +--------------+-------------------------------------------+
 154 | *Heat*       | Models orchestration using OpenStack Heat |
 155 |              |                                           |
 156 +--------------+-------------------------------------------+
 157 | *Node*       | Models Baremetal, Controller, Compute     |
 158 |              |                                           |
 159 +--------------+-------------------------------------------+
 160 | *Standalone* | Models VM running on Non-Managed NFVi     |
 161 |              |                                           |
 162 +--------------+-------------------------------------------+
 163 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 164 |              |                                           |
 165 +--------------+-------------------------------------------+
 166
 167
 168 List of Runners
 169 ^^^^^^^^^^^^^^^
 170
 171 .. note:: Yardstick Gambia 7.0.0 adds 1 new Runner, "IterationIPC".
 172
 173 +----------------+-------------------------------------------------------+
 174 | **Runner**     | **Description**                                       |
 175 |                |                                                       |
 176 +----------------+-------------------------------------------------------+
 177 | *Arithmetic*   | Steps every run arithmetically according to specified |
 178 |                | input value                                           |
 179 |                |                                                       |
 180 +----------------+-------------------------------------------------------+
 181 | *Duration*     | Runs for a specified period of time                   |
 182 |                |                                                       |
 183 +----------------+-------------------------------------------------------+
 184 | *Iteration*    | Runs for a specified number of iterations             |
 185 |                |                                                       |
 186 +----------------+-------------------------------------------------------+
 187 | *IterationIPC* | Runs a configurable number of times before it         |
 188 |                | returns. Each iteration has a configurable timeout.   |
 189 |                |                                                       |
 190 +----------------+-------------------------------------------------------+
 191 | *Sequence*     | Selects input value to a scenario from an input file  |
 192 |                | and runs all entries sequentially                     |
 193 |                |                                                       |
 194 +----------------+-------------------------------------------------------+
 195 | *Dynamictp*    | A runner that searches for the max throughput with    |
 196 |                | binary search                                         |
 197 |                |                                                       |
 198 +----------------+-------------------------------------------------------+
 199 | *Search*       | A runner that runs a specific time before it returns  |
 200 |                |                                                       |
 201 +----------------+-------------------------------------------------------+
 202
 203
 204 List of Scenarios
 205 ^^^^^^^^^^^^^^^^^
 206
 207 +----------------+-----------------------------------------------------+
 208 | **Category**   | **Delivered**                                       |
 209 |                |                                                     |
 210 +----------------+-----------------------------------------------------+
 211 | *Availability* | Attacker:                                           |
 212 |                |                                                     |
 213 |                | * baremetal, process                                |
 214 |                |                                                     |
 215 |                | HA tools:                                           |
 216 |                |                                                     |
 217 |                | * check host, openstack, process, service           |
 218 |                | * kill process                                      |
 219 |                | * start/stop service                                |
 220 |                |                                                     |
 221 |                | Monitor:                                            |
 222 |                |                                                     |
 223 |                | * command, process                                  |
 224 |                |                                                     |
 225 +----------------+-----------------------------------------------------+
 226 | *Compute*      | * cpuload                                           |
 227 |                | * cyclictest                                        |
 228 |                | * lmbench                                           |
 229 |                | * lmbench_cache                                     |
 230 |                | * perf                                              |
 231 |                | * unixbench                                         |
 232 |                | * ramspeed                                          |
 233 |                | * cachestat                                         |
 234 |                | * memeoryload                                       |
 235 |                | * computecapacity                                   |
 236 |                | * SpecCPU2006                                       |
 237 |                |                                                     |
 238 +----------------+-----------------------------------------------------+
 239 | *Networking*   | * iperf3                                            |
 240 |                | * netperf                                           |
 241 |                | * netperf_node                                      |
 242 |                | * ping                                              |
 243 |                | * ping6                                             |
 244 |                | * pktgen                                            |
 245 |                | * sfc                                               |
 246 |                | * sfc with tacker                                   |
 247 |                | * networkcapacity                                   |
 248 |                | * netutilization                                    |
 249 |                | * nstat                                             |
 250 |                | * pktgenDPDK                                        |
 251 |                |                                                     |
 252 +----------------+-----------------------------------------------------+
 253 | *Parser*       | Tosca2Heat                                          |
 254 |                |                                                     |
 255 +----------------+-----------------------------------------------------+
 256 | *Storage*      | * fio                                               |
 257 |                | * bonnie++                                          |
 258 |                | * storagecapacity                                   |
 259 |                |                                                     |
 260 +----------------+-----------------------------------------------------+
 261 | *StorPerf*     | storperf                                            |
 262 |                |                                                     |
 263 +----------------+-----------------------------------------------------+
 264 | *NSB*          | vFW thoughput test case                             |
 265 |                |                                                     |
 266 +----------------+-----------------------------------------------------+
 267
 268
 269 New Test cases
 270 --------------
 271
 272 .. note:: Yardstick Gambia 7.0.0 adds no new test cases.
 273
 274 * Generic NFVI test cases
 275
 276  * (e.g.) OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
 277
 278 * HA Test cases
 279
 280  * (e.g.) OPNFV_YARDSTICK_TC093 - SDN Vswitch resilience in non-HA or HA configuration
 281
 282
 283 Version Change
 284 ==============
 285
 286 Module Version Changes
 287 ----------------------
 288
 289 This is the seventh tracked release of Yardstick. It is based on following
 290 upstream versions:
 291
 292 - OpenStack Queens
 293 - OpenDayLight Oxygen
 294
 295
 296 Document Version Changes
 297 ------------------------
 298
 299 This is the seventh tracked version of the Yardstick framework in OPNFV.
 300 It includes the following documentation updates:
 301
 302 - Yardstick User Guide:
 303
 304   - Remove vTC chapter;
 305
 306 - Yardstick Developer Guide
 307 - Yardstick Release Notes for Yardstick: this document
 308
 309
 310 Feature additions
 311 -----------------
 312
 313 - Simplify Yardstick installation to use a single ansible playbook (nsb_setup.yaml)....
 314 - Spirent support.
 315 - vEPC testcases.
 316 - Agnostic VNF tests cases for reuse of standard RFC-2544 test case.
 317 - PROX enhancements and the addition of Standalone test case using SRIOV and
 318   OVS-DPDK.
 319 - Ixia enhancements for vBNG and PPPoE traffic.
 320 - Improvements of unit tests and gating.
 321 - Add DPDK pktgen traffic generator.
 322 - Kubernetes context enhancements.
 323 - Kubernetes sample test cases of fio and lmbench added.
 324
 325
 326 Scenario Matrix
 327 ===============
 328
 329 For Gambia 7.0.0, Yardstick was tested on the following scenarios:
 330
 331 +-------------------------+------+---------+----------+------+
 332 |        Scenario         | Apex | Compass | Fuel-arm | Fuel |
 333 +=========================+======+=========+==========+======+
 334 | os-nosdn-nofeature-noha |   X  |         |          |      |
 335 +-------------------------+------+---------+----------+------+
 336 | os-nosdn-nofeature-ha   |   X  |         |          |      |
 337 +-------------------------+------+---------+----------+------+
 338 | os-odl-bgpvpn-noha      |   X  |         |          |      |
 339 +-------------------------+------+---------+----------+------+
 340 | os-nosdn-calipso-noha   |   X  |         |          |      |
 341 +-------------------------+------+---------+----------+------+
 342 | os-nosdn-kvm-ha         |      |    X    |          |      |
 343 +-------------------------+------+---------+----------+------+
 344 | os-odl-nofeature-ha     |      |         |     X    |   X  |
 345 +-------------------------+------+---------+----------+------+
 346 | os-odl-sfc-noha         |   X  |         |          |      |
 347 +-------------------------+------+---------+----------+------+
 348 | os-nosdn-ovs-ha         |      |         |          |   X  |
 349 +-------------------------+------+---------+----------+------+
 350 | k8-nosdn-nofeature-ha   |      |    X    |          |      |
 351 +-------------------------+------+---------+----------+------+
 352 | k8-nosdn-stor4nfv-noha  |      |    X    |          |      |
 353 +-------------------------+------+---------+----------+------+
 354 | k8-nosdn-stor4nfv-ha    |      |    X    |          |      |
 355 +-------------------------+------+---------+----------+------+
 356
 357
 358 Test results
 359 ============
 360
 361 Test results are available in:
 362
 363  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 364
 365
 366 Known Issues/Faults
 367 -------------------
 368
 369
 370 Corrected Faults
 371 ----------------
 372
 373 Gambia 7.0.0:
 374
 375 +--------------------+--------------------------------------------------------------------------+
 376 | **JIRA REFERENCE** |                             **DESCRIPTION**                              |
 377 +====================+==========================================================================+
 378 | YARDSTICK-1137     | Fix CLI argument handling in nsb_setup.sh                                |
 379 +--------------------+--------------------------------------------------------------------------+
 380 | YARDSTICK-1220     | Get stats for multiple port simultaneously                               |
 381 +--------------------+--------------------------------------------------------------------------+
 382 | YARDSTICK-1260     | Added missing functionality to start VM and access it using SSH keys     |
 383 |                    | in Standalone contexts.                                                  |
 384 +--------------------+--------------------------------------------------------------------------+
 385 | YARDSTICK-1298     | Allows for in-line overriding/modification of traffic profile variables  |
 386 |                    | from the testcase file.                                                  |
 387 +--------------------+--------------------------------------------------------------------------+
 388 | YARDSTICK-1368     | Updated existing test cases in Yardstick to minimize changes done        |
 389 |                    | manually to run standalone tests for Trex.                               |
 390 +--------------------+--------------------------------------------------------------------------+
 391 | YARDSTICK-1389     | Add status filed for RFC2544 TC iterations                               |
 392 +--------------------+--------------------------------------------------------------------------+
 393 | YARDSTICK-1395     | Update 'configure_uwsgi' role to work in baremetal/container modes.      |
 394 +--------------------+--------------------------------------------------------------------------+
 395 | YARDSTICK-1402     | Change IP assignment for VM to static for standalone context             |
 396 +--------------------+--------------------------------------------------------------------------+
 397 | YARDSTICK-1404     | CPU Utilization for VNF and traffic generator are now graphed on Grafana |
 398 +--------------------+--------------------------------------------------------------------------+
 399 | YARDSTICK-1411     | Fix Yardstick Docker image ARM build                                     |
 400 +--------------------+--------------------------------------------------------------------------+
 401 | YARDSTICK-1414     | Update the pinned sampleVNF version to use a commit instead of a branch  |
 402 +--------------------+--------------------------------------------------------------------------+
 403 | YARDSTICK-1418     | NSB PROX NFVi test now stops after reaching expected precision           |
 404 +--------------------+--------------------------------------------------------------------------+
 405 | YARDSTICK-1457     | Fix influxdb "field type conflict" error                                 |
 406 +--------------------+--------------------------------------------------------------------------+
 407 | YARDSTICK-1458     | Update Grafana to display "real-time" data instead of historical data.   |
 408 +--------------------+--------------------------------------------------------------------------+
 409 | YARDSTICK-1462     | NSB: Add OvS 2.8.1 support in SA context                                 |
 410 +--------------------+--------------------------------------------------------------------------+
 411 | YARDSTICK-1492     | Make OvS server to listen on TCP                                         |
 412 +--------------------+--------------------------------------------------------------------------+
 413 | YARDSTICK-1493     | The RX queues number is hard-codded and cannot be changed                |
 414 +--------------------+--------------------------------------------------------------------------+
 415
 416
 417 Gambia 7.0.0 known restrictions/issues
 418 ======================================
 419
 420 +-----------+-----------------------+------------------+
 421 | Installer |       Scenario        |      Issue       |
 422 +===========+=======================+==================+
 423 |   apex    |  os-ovn-nofeature-ha  |  YARDSTICK-1517  |
 424 +-----------+-----------------------+------------------+
 425
 426 Useful links
 427 ============
 428
 429  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 430
 431  - wiki Yardstick Gambia release planning page: https://wiki.opnfv.org/display/yardstick/Release+Gambia
 432
 433  - Yardstick repo: https://git.opnfv.org/yardstick
 434
 435  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 436
 437  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 438
 439  - Yardstick IRC channel: #opnfv-yardstick