docs/release/release-notes/release-notes.rst

   1 =======
   2 License
   3 =======
   4
   5 OPNFV Fraser release note for Yardstick Docs
   6 are licensed under a Creative Commons Attribution 4.0 International License.
   7 You should have received a copy of the license along with this.
   8 If not, see <http://creativecommons.org/licenses/by/4.0/>.
   9
  10 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
  11  licensed under the terms of the Apache License, Version 2.0.
  12
  13 =======================================
  14 OPNFV Fraser Release Note for Yardstick
  15 =======================================
  16
  17 .. toctree::
  18    :maxdepth: 2
  19
  20 .. _Yardstick: https://wiki.opnfv.org/yardstick
  21
  22 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  23
  24 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  25
  26
  27 Abstract
  28 ========
  29
  30 This document describes the release note of Yardstick project.
  31
  32
  33 Version History
  34 ===============
  35 +-------------------+-----------+---------------------------------+
  36 | *Date*            | *Version* | *Comment*                       |
  37 |                   |           |                                 |
  38 +-------------------+-----------+---------------------------------+
  39 | April 27, 2018    | 6.0.0     | Yardstick for Fraser release    |
  40 |                   |           |                                 |
  41 +-------------------+-----------+---------------------------------+
  42
  43
  44 Important Notes
  45 ===============
  46
  47 The software delivered in the OPNFV Yardstick_ Project, comprising the
  48 *Yardstick framework*, and the *Yardstick test cases* is a realization of
  49 the methodology in ETSI-ISG NFV-TST001_.
  50
  51 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  52 independent.
  53
  54 OPNFV Fraser Release
  55 ====================
  56
  57 This Fraser release provides *Yardstick* as a framework for NFVI testing
  58 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  59
  60 * Documentation generated with Sphinx
  61
  62   * User Guide
  63   * Developer Guide
  64   * Release notes (this document)
  65   * Results
  66
  67 * Automated Yardstick test suite (daily, weekly)
  68
  69   * Jenkins Jobs for OPNFV community labs
  70
  71 * Automated Yardstick test results visualization
  72
  73   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  74     backend
  75
  76 * Yardstick framework source code
  77
  78 * Yardstick test cases yaml files
  79
  80 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  81
  82 For Fraser release, the *Yardstick framework* is used for the following
  83 testing:
  84
  85 * OPNFV platform testing - generic test cases to measure the categories:
  86
  87   * Compute
  88   * Network
  89   * Storage
  90
  91 * OPNFV platform network service benchmarking (NSB)
  92
  93   * NSB
  94
  95 * Test cases for the following OPNFV Projects:
  96
  97   * Container4NFV
  98   * High Availability
  99   * IPv6
 100   * KVM
 101   * Parser
 102   * StorPerf
 103   * VSperf
 104
 105 The *Yardstick framework* is developed in the OPNFV community, by the
 106 Yardstick_ team.
 107
 108 .. note:: The test case description template used for the Yardstick test cases
 109   is based on the document ETSI-ISG NFV-TST001_; the results report template
 110   used for the Yardstick results is based on the IEEE Std 829-2008.
 111
 112
 113 Release Data
 114 ============
 115
 116 +--------------------------------+-----------------------+
 117 | **Project**                    | Yardstick             |
 118 |                                |                       |
 119 +--------------------------------+-----------------------+
 120 | **Repo/tag**                   | yardstick/opnfv-6.0.0 |
 121 |                                |                       |
 122 +--------------------------------+-----------------------+
 123 | **Yardstick Docker image tag** | opnfv-6.0.0           |
 124 |                                |                       |
 125 +--------------------------------+-----------------------+
 126 | **Release designation**        | Fraser                |
 127 |                                |                       |
 128 +--------------------------------+-----------------------+
 129 | **Release date**               | April 27, 2018        |
 130 |                                |                       |
 131 +--------------------------------+-----------------------+
 132 | **Purpose of the delivery**    | OPNFV Fraser 6.0.0    |
 133 |                                |                       |
 134 +--------------------------------+-----------------------+
 135
 136
 137 Deliverables
 138 ============
 139
 140 Documents
 141 ---------
 142
 143  - User Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/user/userguide/index.html
 144
 145  - Developer Guide: http://docs.opnfv.org/en/stable-fraser/submodules/yardstick/docs/testing/developer/devguide/index.html
 146
 147
 148 Software Deliverables
 149 ---------------------
 150
 151  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-6.0.0)
 152
 153 List of Contexts
 154 ^^^^^^^^^^^^^^^^
 155
 156 +--------------+-------------------------------------------+
 157 | **Context**  | **Description**                           |
 158 |              |                                           |
 159 +--------------+-------------------------------------------+
 160 | *Heat*       | Models orchestration using OpenStack Heat |
 161 |              |                                           |
 162 +--------------+-------------------------------------------+
 163 | *Node*       | Models Baremetal, Controller, Compute     |
 164 |              |                                           |
 165 +--------------+-------------------------------------------+
 166 | *Standalone* | Models VM running on Non-Managed NFVi     |
 167 |              |                                           |
 168 +--------------+-------------------------------------------+
 169 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 170 |              |                                           |
 171 +--------------+-------------------------------------------+
 172
 173
 174 List of Runners
 175 ^^^^^^^^^^^^^^^
 176
 177 Note: Yardstick Fraser 6.0.0 add two new Runners, "Dynamictp" and "Search".
 178
 179 +---------------+-------------------------------------------------------+
 180 | **Runner**    | **Description**                                       |
 181 |               |                                                       |
 182 +---------------+-------------------------------------------------------+
 183 | *Arithmetic*  | Steps every run arithmetically according to specified |
 184 |               | input value                                           |
 185 |               |                                                       |
 186 +---------------+-------------------------------------------------------+
 187 | *Duration*    | Runs for a specified period of time                   |
 188 |               |                                                       |
 189 +---------------+-------------------------------------------------------+
 190 | *Iteration*   | Runs for a specified number of iterations             |
 191 |               |                                                       |
 192 +---------------+-------------------------------------------------------+
 193 | *Sequence*    | Selects input value to a scenario from an input file  |
 194 |               | and runs all entries sequentially                     |
 195 |               |                                                       |
 196 +---------------+-------------------------------------------------------+
 197 | **Dynamictp** | A runner that searches for the max throughput with    |
 198 |               | binary search                                         |
 199 |               |                                                       |
 200 +---------------+-------------------------------------------------------+
 201 | **Search**    | A runner that runs a specific time before it returns  |
 202 |               |                                                       |
 203 +---------------+-------------------------------------------------------+
 204
 205
 206 List of Scenarios
 207 ^^^^^^^^^^^^^^^^^
 208
 209 +----------------+-----------------------------------------------------+
 210 | **Category**   | **Delivered**                                       |
 211 |                |                                                     |
 212 +----------------+-----------------------------------------------------+
 213 | *Availability* | Attacker:                                           |
 214 |                |                                                     |
 215 |                | * baremetal, process                                |
 216 |                |                                                     |
 217 |                | HA tools:                                           |
 218 |                |                                                     |
 219 |                | * check host, openstack, process, service           |
 220 |                | * kill process                                      |
 221 |                | * start/stop service                                |
 222 |                |                                                     |
 223 |                | Monitor:                                            |
 224 |                |                                                     |
 225 |                | * command, process                                  |
 226 |                |                                                     |
 227 +----------------+-----------------------------------------------------+
 228 | *Compute*      | * cpuload                                           |
 229 |                | * cyclictest                                        |
 230 |                | * lmbench                                           |
 231 |                | * lmbench_cache                                     |
 232 |                | * perf                                              |
 233 |                | * unixbench                                         |
 234 |                | * ramspeed                                          |
 235 |                | * cachestat                                         |
 236 |                | * memeoryload                                       |
 237 |                | * computecapacity                                   |
 238 |                | * SpecCPU2006                                       |
 239 |                |                                                     |
 240 +----------------+-----------------------------------------------------+
 241 | *Networking*   | * iperf3                                            |
 242 |                | * netperf                                           |
 243 |                | * netperf_node                                      |
 244 |                | * ping                                              |
 245 |                | * ping6                                             |
 246 |                | * pktgen                                            |
 247 |                | * sfc                                               |
 248 |                | * sfc with tacker                                   |
 249 |                | * networkcapacity                                   |
 250 |                | * netutilization                                    |
 251 |                | * nstat                                             |
 252 |                | * pktgenDPDK                                        |
 253 |                |                                                     |
 254 +----------------+-----------------------------------------------------+
 255 | *Parser*       | Tosca2Heat                                          |
 256 |                |                                                     |
 257 +----------------+-----------------------------------------------------+
 258 | *Storage*      | * fio                                               |
 259 |                | * bonnie++                                          |
 260 |                | * storagecapacity                                   |
 261 |                |                                                     |
 262 +----------------+-----------------------------------------------------+
 263 | *StorPerf*     | storperf                                            |
 264 |                |                                                     |
 265 +----------------+-----------------------------------------------------+
 266 | *NSB*          | vFW thoughput test case                             |
 267 |                |                                                     |
 268 +----------------+-----------------------------------------------------+
 269
 270
 271 New Test cases
 272 --------------
 273
 274 * Generic NFVI test cases
 275
 276  * OPNFV_YARDSTICK_TCO84 - SPEC CPU 2006 for VM
 277
 278 * HA Test cases
 279
 280  * OPNFV_YARDSTICK_TC087 - SDN Controller resilience in non-HA configuration
 281  * OPNFV_YARDSTICK_TC090 - Control node Openstack service down - database instance
 282  * OPNFV_YARDSTICK_TC091 - Control node Openstack service down - heat-api
 283
 284
 285 Version Change
 286 ==============
 287
 288 Module Version Changes
 289 ----------------------
 290
 291 This is the sixth tracked release of Yardstick. It is based on following
 292 upstream versions:
 293
 294 - OpenStack Pike
 295 - OpenDayLight Oxygen
 296
 297
 298 Document Version Changes
 299 ------------------------
 300
 301 This is the sixth tracked version of the Yardstick framework in OPNFV.
 302 It includes the following documentation updates:
 303
 304 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
 305   add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
 306   add "Yardstick user interface" chapter; Update Yardstick installation chapter;
 307 - Yardstick Developer Guide
 308 - Yardstick Release Notes for Yardstick: this document
 309
 310
 311 Feature additions
 312 -----------------
 313
 314 - Plugin-based test cases support Heat context
 315 - SR-IOV support for the Heat context
 316 - Support using existing network in Heat context
 317 - Support running test cases with existing VNFs/without destroying VNF in Heat context
 318 - Add vFW scale-up template
 319 - Improvements of unit tests and gating
 320 - GUI improvement about passing parameters
 321
 322
 323 Scenario Matrix
 324 ===============
 325
 326 For Fraser 6.0.0, Yardstick was tested on the following scenarios:
 327
 328 +-------------------------+------+---------+----------+------+------+-------+
 329 |        Scenario         | Apex | Compass | Fuel-arm | Fuel | Joid | Daisy |
 330 +=========================+======+=========+==========+======+======+=======+
 331 | os-nosdn-nofeature-noha |  X   |    X    |          |      |  X   |       |
 332 +-------------------------+------+---------+----------+------+------+-------+
 333 | os-nosdn-nofeature-ha   |  X   |    X    |    X     |  X   |  X   |   X   |
 334 +-------------------------+------+---------+----------+------+------+-------+
 335 | os-nosdn-bar-noha       |  X   |    X    |          |      |      |       |
 336 +-------------------------+------+---------+----------+------+------+-------+
 337 | os-nosdn-bar-ha         |  X   |         |          |      |      |       |
 338 +-------------------------+------+---------+----------+------+------+-------+
 339 | os-odl-bgpvpn-ha        |  X   |         |          |      |      |       |
 340 +-------------------------+------+---------+----------+------+------+-------+
 341 | os-nosdn-calipso-noha   |  X   |         |          |      |      |       |
 342 +-------------------------+------+---------+----------+------+------+-------+
 343 | os-nosdn-kvm-ha         |      |    X    |          |      |      |       |
 344 +-------------------------+------+---------+----------+------+------+-------+
 345 | os-odl_l3-nofeature-ha  |      |    X    |          |      |      |       |
 346 +-------------------------+------+---------+----------+------+------+-------+
 347 | os-odl-sfc-ha           |      |    X    |          |      |      |       |
 348 +-------------------------+------+---------+----------+------+------+-------+
 349 | os-odl-nofeature-ha     |      |         |          |  X   |      |   X   |
 350 +-------------------------+------+---------+----------+------+------+-------+
 351 | os-nosdn-ovs-ha         |      |         |          |  X   |      |       |
 352 +-------------------------+------+---------+----------+------+------+-------+
 353 | k8-nosdn-nofeature-ha   |      |    X    |          |      |      |       |
 354 +-------------------------+------+---------+----------+------+------+-------+
 355 | k8-nosdn-stor4nfv-noha  |      |    X    |          |      |      |       |
 356 +-------------------------+------+---------+----------+------+------+-------+
 357
 358
 359 Test results
 360 ============
 361
 362 Test results are available in:
 363
 364  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 365
 366 The reporting pages can be found at:
 367
 368 +---------------+----------------------------------------------------------------------------------+
 369 | apex          | http://testresults.opnfv.org/reporting/fraser/yardstick/status-apex.html         |
 370 +---------------+----------------------------------------------------------------------------------+
 371 | compass       | http://testresults.opnfv.org/reporting/fraser/yardstick/status-compass.html      |
 372 +---------------+----------------------------------------------------------------------------------+
 373 | fuel\@x86     | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@x86.html     |
 374 +---------------+----------------------------------------------------------------------------------+
 375 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/fraser/yardstick/status-fuel@aarch64.html |
 376 +---------------+----------------------------------------------------------------------------------+
 377 | joid          | http://testresults.opnfv.org/reporting/fraser/yardstick/status-joid.html         |
 378 +---------------+----------------------------------------------------------------------------------+
 379
 380 Known Issues/Faults
 381 -------------------
 382
 383
 384 Corrected Faults
 385 ----------------
 386
 387 Fraser 6.0.0:
 388
 389 +--------------------+--------------------------------------------------------------------------+
 390 | **JIRA REFERENCE** |                             **DESCRIPTION**                              |
 391 +====================+==========================================================================+
 392 |   YARDSTICK-831    | tc053 kill haproxy wrong                                                 |
 393 +--------------------+--------------------------------------------------------------------------+
 394 |   YARDSTICK-842    | load image fails when there's cirros image exist                         |
 395 +--------------------+--------------------------------------------------------------------------+
 396 |   YARDSTICK-857    | tc006 failed due to volume attached to different location "/dev/vdc"     |
 397 +--------------------+--------------------------------------------------------------------------+
 398 |   YARDSTICK-874    | Specify supported architecture for Ubuntu backports repository           |
 399 +--------------------+--------------------------------------------------------------------------+
 400 |   YARDSTICK-875    | Check if multiverse repository is available in Ubuntu                    |
 401 +--------------------+--------------------------------------------------------------------------+
 402 |   YARDSTICK-893    | Fix proxy env handling and ansible multinode support                     |
 403 +--------------------+--------------------------------------------------------------------------+
 404 |   YARDSTICK-899    | Variable local_iface_name is read before it is set                       |
 405 +--------------------+--------------------------------------------------------------------------+
 406 |   YARDSTICK-900    | Section in "upload_yardstick_image.yml" invalid                          |
 407 +--------------------+--------------------------------------------------------------------------+
 408 |   YARDSTICK-911    | Remove 'inconsistent-return-statements' from Pylint checks               |
 409 +--------------------+--------------------------------------------------------------------------+
 410 |   YARDSTICK-989    | Yardstick real-time influxdb KPI reporting regressions                   |
 411 +--------------------+--------------------------------------------------------------------------+
 412 |   YARDSTICK-994    | NSB set-up build script for baremetal broken                             |
 413 +--------------------+--------------------------------------------------------------------------+
 414 |   YARDSTICK-996    | Error in address input format in "_ip_range_action_partial"              |
 415 +--------------------+--------------------------------------------------------------------------+
 416 |   YARDSTICK-1003   | Prox vnf descriptor cleanup for tg and vnf                               |
 417 +--------------------+--------------------------------------------------------------------------+
 418 |   YARDSTICK-1006   | Ansible destroy script will fail if vm has already been undefined        |
 419 +--------------------+--------------------------------------------------------------------------+
 420 |   YARDSTICK-1012   | constants: fix pylint warnings for OSError                               |
 421 +--------------------+--------------------------------------------------------------------------+
 422 |   YARDSTICK-1014   | Remove unused args in                                                    |
 423 |                    | network_services.traffic_profile.ixia_rfc2544.IXIARFC2544Profile         |
 424 +--------------------+--------------------------------------------------------------------------+
 425 |   YARDSTICK-1016   | Allow vm to access outside world through default gateway                 |
 426 +--------------------+--------------------------------------------------------------------------+
 427 |   YARDSTICK-1019   | For 'qemu-img version 2.10.1' unit 'MB' is not acceptable ansible script |
 428 +--------------------+--------------------------------------------------------------------------+
 429 |   YARDSTICK-1021   | NSB: All Sample VNF test cases timeout after 1 hour of execution         |
 430 +--------------------+--------------------------------------------------------------------------+
 431 |   YARDSTICK-1036   | Prox: Addition of storage of extra counters for Grafana                  |
 432 +--------------------+--------------------------------------------------------------------------+
 433 |   YARDSTICK-1038   | Missing file which is described in the operation_conf.yaml               |
 434 +--------------------+--------------------------------------------------------------------------+
 435 |   YARDSTICK-1047   | Error in string format in HeatTemplateError message                      |
 436 +--------------------+--------------------------------------------------------------------------+
 437 |   YARDSTICK-1056   | yardstick report command print error when run test case                  |
 438 +--------------------+--------------------------------------------------------------------------+
 439 |   YARDSTICK-1059   | Reduce the log level if TRex client is no connected                      |
 440 +--------------------+--------------------------------------------------------------------------+
 441 |   YARDSTICK-1073   | Error when retrieving "options" section in "scenario"                    |
 442 +--------------------+--------------------------------------------------------------------------+
 443 |   YARDSTICK-1080   | Running Test Case in Latest Yardstick Docker Image shows Error           |
 444 +--------------------+--------------------------------------------------------------------------+
 445 |   YARDSTICK-1082   | tc043,tc055, tc063, tc075,  pass wrong node name in the ci scenario yaml |
 446 +--------------------+--------------------------------------------------------------------------+
 447 |   YARDSTICK-1102   | Don't hide exception traceback from Task.start()                         |
 448 +--------------------+--------------------------------------------------------------------------+
 449 |   YARDSTICK-1107   | bad exception traceback print due to atexit_handler                      |
 450 +--------------------+--------------------------------------------------------------------------+
 451 |   YARDSTICK-1120   | HA test case tc050 should start monitor before attack                    |
 452 +--------------------+--------------------------------------------------------------------------+
 453
 454 Fraser 6.0.0 known restrictions/issues
 455 ======================================
 456
 457 +-----------+-----------+----------------------------------------------+
 458 | Installer | Scenario  | Issue                                        |
 459 +===========+===========+==============================================+
 460 |           |           |                                              |
 461 +-----------+-----------+----------------------------------------------+
 462
 463 Useful links
 464 ============
 465
 466  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 467
 468  - wiki Yardstick Fraser release planing page: https://wiki.opnfv.org/display/yardstick/Release+Fraser
 469
 470  - Yardstick repo: https://git.opnfv.org/cgit/yardstick
 471
 472  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 473
 474  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 475
 476  - Yardstick IRC chanel: #opnfv-yardstick