docs/release/release-notes/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
   5  licensed under the terms of the Apache License, Version 2.0.
   6
   7 =======================
   8 Yardstick Release Notes
   9 =======================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/display/yardstick
  15
  16 .. _Dashboard: http://testresults.opnfv.org/grafana/
  17
  18 .. _NFV-TST001: https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the Hunter release of OPNFV Yardstick.
  25
  26
  27 Version History
  28 ===============
  29 +-------------------+-----------+---------------------------------+
  30 | *Date*            | *Version* | *Comment*                       |
  31 |                   |           |                                 |
  32 +-------------------+-----------+---------------------------------+
  33 | May 10, 2019      | 8.0.0     | Yardstick for Hunter release    |
  34 |                   |           |                                 |
  35 +-------------------+-----------+---------------------------------+
  36
  37
  38 Important Notes
  39 ===============
  40
  41 The software delivered in the OPNFV Yardstick_ Project, comprising the
  42 *Yardstick framework*, and the *Yardstick test cases* is a realization of
  43 the methodology in ETSI-ISG NFV-TST001_.
  44
  45 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  46 independent.
  47
  48 OPNFV Hunter Release
  49 ====================
  50
  51 This Hunter release provides *Yardstick* as a framework for NFVI testing
  52 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  53
  54 * Documentation generated with Sphinx
  55
  56   * User Guide
  57   * Developer Guide
  58   * Release notes (this document)
  59   * Results
  60
  61 * Automated Yardstick test suite (daily, weekly)
  62
  63   * Jenkins Jobs for OPNFV community labs
  64
  65 * Automated Yardstick test results visualization
  66
  67   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  68     backend
  69
  70 * Yardstick framework source code
  71
  72 * Yardstick test cases yaml files
  73
  74 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  75
  76 For Hunter release, the *Yardstick framework* is used for the following
  77 testing:
  78
  79 * OPNFV platform testing - generic test cases to measure the categories:
  80
  81   * Compute
  82   * Network
  83   * Storage
  84
  85 * OPNFV platform network service benchmarking (NSB)
  86
  87   * NSB
  88
  89 * Test cases for the following OPNFV Projects:
  90
  91   * Container4NFV
  92   * High Availability
  93   * IPv6
  94   * KVM
  95   * Parser
  96   * StorPerf
  97   * VSperf
  98
  99 The *Yardstick framework* is developed in the OPNFV community, by the
 100 Yardstick_ team.
 101
 102 .. note:: The test case description template used for the Yardstick test cases
 103   is based on the document ETSI-ISG NFV-TST001_; the results report template
 104   used for the Yardstick results is based on the IEEE Std 829-2008.
 105
 106
 107 Release Data
 108 ============
 109
 110 +--------------------------------+-----------------------+
 111 | **Project**                    | Yardstick             |
 112 |                                |                       |
 113 +--------------------------------+-----------------------+
 114 | **Repo/tag**                   | yardstick/opnfv-8.0.0 |
 115 |                                |                       |
 116 +--------------------------------+-----------------------+
 117 | **Yardstick Docker image tag** | opnfv-8.0.0           |
 118 |                                |                       |
 119 +--------------------------------+-----------------------+
 120 | **Release designation**        | Hunter 8.0            |
 121 |                                |                       |
 122 +--------------------------------+-----------------------+
 123 | **Release date**               | May 10, 2019          |
 124 |                                |                       |
 125 +--------------------------------+-----------------------+
 126 | **Purpose of the delivery**    | OPNFV Hunter 8.0.0    |
 127 |                                |                       |
 128 +--------------------------------+-----------------------+
 129
 130
 131 Deliverables
 132 ============
 133
 134 Documents
 135 ---------
 136
 137  - User Guide: :ref:`<yardstick:userguide>`
 138
 139  - Developer Guide: :ref:`<yardstick:devguide>`
 140
 141
 142 Software Deliverables
 143 ---------------------
 144
 145  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-8.0.0)
 146
 147 List of Contexts
 148 ^^^^^^^^^^^^^^^^
 149
 150 +--------------+-------------------------------------------+
 151 | **Context**  | **Description**                           |
 152 |              |                                           |
 153 +--------------+-------------------------------------------+
 154 | *Heat*       | Models orchestration using OpenStack Heat |
 155 |              |                                           |
 156 +--------------+-------------------------------------------+
 157 | *Node*       | Models Baremetal, Controller, Compute     |
 158 |              |                                           |
 159 +--------------+-------------------------------------------+
 160 | *Standalone* | Models VM running on Non-Managed NFVi     |
 161 |              |                                           |
 162 +--------------+-------------------------------------------+
 163 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 164 |              |                                           |
 165 +--------------+-------------------------------------------+
 166
 167
 168 List of Runners
 169 ^^^^^^^^^^^^^^^
 170
 171 +----------------+-------------------------------------------------------+
 172 | **Runner**     | **Description**                                       |
 173 |                |                                                       |
 174 +----------------+-------------------------------------------------------+
 175 | *Arithmetic*   | Steps every run arithmetically according to specified |
 176 |                | input value                                           |
 177 |                |                                                       |
 178 +----------------+-------------------------------------------------------+
 179 | *Duration*     | Runs for a specified period of time                   |
 180 |                |                                                       |
 181 +----------------+-------------------------------------------------------+
 182 | *Iteration*    | Runs for a specified number of iterations             |
 183 |                |                                                       |
 184 +----------------+-------------------------------------------------------+
 185 | *IterationIPC* | Runs a configurable number of times before it         |
 186 |                | returns. Each iteration has a configurable timeout.   |
 187 |                |                                                       |
 188 +----------------+-------------------------------------------------------+
 189 | *Sequence*     | Selects input value to a scenario from an input file  |
 190 |                | and runs all entries sequentially                     |
 191 |                |                                                       |
 192 +----------------+-------------------------------------------------------+
 193 | *Dynamictp*    | A runner that searches for the max throughput with    |
 194 |                | binary search                                         |
 195 |                |                                                       |
 196 +----------------+-------------------------------------------------------+
 197 | *Search*       | A runner that runs a specific time before it returns  |
 198 |                |                                                       |
 199 +----------------+-------------------------------------------------------+
 200
 201
 202 List of Scenarios
 203 ^^^^^^^^^^^^^^^^^
 204
 205 +----------------+-----------------------------------------------------+
 206 | **Category**   | **Delivered**                                       |
 207 |                |                                                     |
 208 +----------------+-----------------------------------------------------+
 209 | *Availability* | Attacker:                                           |
 210 |                |                                                     |
 211 |                | * baremetal, process                                |
 212 |                |                                                     |
 213 |                | HA tools:                                           |
 214 |                |                                                     |
 215 |                | * check host, openstack, process, service           |
 216 |                | * kill process                                      |
 217 |                | * start/stop service                                |
 218 |                |                                                     |
 219 |                | Monitor:                                            |
 220 |                |                                                     |
 221 |                | * command, process                                  |
 222 |                |                                                     |
 223 +----------------+-----------------------------------------------------+
 224 | *Compute*      | * cpuload                                           |
 225 |                | * cyclictest                                        |
 226 |                | * lmbench                                           |
 227 |                | * lmbench_cache                                     |
 228 |                | * perf                                              |
 229 |                | * unixbench                                         |
 230 |                | * ramspeed                                          |
 231 |                | * cachestat                                         |
 232 |                | * memeoryload                                       |
 233 |                | * computecapacity                                   |
 234 |                | * SpecCPU2006                                       |
 235 |                |                                                     |
 236 +----------------+-----------------------------------------------------+
 237 | *Networking*   | * iperf3                                            |
 238 |                | * netperf                                           |
 239 |                | * netperf_node                                      |
 240 |                | * ping                                              |
 241 |                | * ping6                                             |
 242 |                | * pktgen                                            |
 243 |                | * sfc                                               |
 244 |                | * sfc with tacker                                   |
 245 |                | * networkcapacity                                   |
 246 |                | * netutilization                                    |
 247 |                | * nstat                                             |
 248 |                | * pktgenDPDK                                        |
 249 |                |                                                     |
 250 +----------------+-----------------------------------------------------+
 251 | *Parser*       | Tosca2Heat                                          |
 252 |                |                                                     |
 253 +----------------+-----------------------------------------------------+
 254 | *Storage*      | * fio                                               |
 255 |                | * bonnie++                                          |
 256 |                | * storagecapacity                                   |
 257 |                |                                                     |
 258 +----------------+-----------------------------------------------------+
 259 | *StorPerf*     | storperf                                            |
 260 |                |                                                     |
 261 +----------------+-----------------------------------------------------+
 262 | *NSB*          | vFW thoughput test case                             |
 263 |                |                                                     |
 264 +----------------+-----------------------------------------------------+
 265
 266
 267 New Test cases
 268 --------------
 269
 270 .. note:: Yardstick Hunter 8.0.0 adds no new test cases.
 271
 272
 273
 274 Version Change
 275 ==============
 276
 277 Module Version Changes
 278 ----------------------
 279
 280 This is the seventh tracked release of Yardstick. It is based on following
 281 upstream versions:
 282
 283 - OpenStack Rocky
 284
 285
 286 Document Version Changes
 287 ------------------------
 288
 289 This is the seventh tracked version of the Yardstick framework in OPNFV.
 290 It includes the following documentation updates:
 291
 292 - Yardstick User Guide:
 293 - Yardstick Developer Guide
 294 - Yardstick Release Notes for Yardstick: this document
 295
 296
 297 Feature additions
 298 -----------------
 299
 300 - Add TRex Baremetal/SA scale up test cases.
 301 - Add possibility to distribute DPDK/vhost-user ports/queues between CPU cores.
 302 - Adding scale up test case for l3fwd OvS-DPDK.
 303 - Add Testcase Prox Standalone SRIOV.
 304 - Add vBNG PPPoE test cases functionality.
 305 - Add IXIA Baremetal scale up testcases.
 306 - Add vIPSEC testcases.
 307 - Add TRex Baremetal/SA scale up test cases.
 308 - Add OpenStack test cases with different framesize and IMIX.
 309 - Add new scenario NSPerf-RFC3511.
 310 - Add VPP traffic profiles for TRex traffic generator.
 311 - Add Trex traffic generator to initate Traffic for VPP IPSEC.
 312 - Add IxNextgen API for setting IP priority, flow tracking options, creating BGP protocol layer.
 313 - Add Pktgen traffic generator for vCMTS.
 314 - Add Trex traffic generator to initate Traffic for VPP IPSEC.
 315 - Add sipp trafficgen based SampleVNFTrafficGen.
 316 - Add ansible scripts to deploy Kubernetes.
 317 - Update NSB report to use Barometer metrics.
 318 - Support using existing private key when using external heat template file.
 319
 320 Scenario Matrix
 321 ===============
 322
 323
 324 Test results
 325 ============
 326
 327 Test results are available in:
 328
 329  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 330
 331
 332 Known Issues/Faults
 333 -------------------
 334
 335
 336 Corrected Faults
 337 ----------------
 338
 339
 340 Hunter 8.0.0 known restrictions/issues
 341 ======================================
 342
 343
 344 Useful links
 345 ============
 346
 347  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 348
 349  - wiki Yardstick Hunter release planning page: https://wiki.opnfv.org/display/yardstick/Release+Hunter
 350
 351  - Yardstick repo: https://git.opnfv.org/yardstick
 352
 353  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 354
 355  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 356
 357  - Yardstick IRC channel: #opnfv-yardstick