docs/release/release-notes/release-notes.rst

   1 License
   2 =======
   3
   4 OPNFV Euphrates release note for Yardstick Docs
   5 are licensed under a Creative Commons Attribution 4.0 International License.
   6 You should have received a copy of the license along with this.
   7 If not, see <http://creativecommons.org/licenses/by/4.0/>.
   8
   9 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
  10  licensed under the terms of the Apache License, Version 2.0.
  11
  12 OPNFV Euphrates Release Note for Yardstick
  13 ==========================================
  14
  15 .. toctree::
  16    :maxdepth: 2
  17
  18 .. _Yardstick: https://wiki.opnfv.org/yardstick
  19
  20 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  21
  22 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  23
  24
  25 Abstract
  26 --------
  27
  28 This document describes the release note of Yardstick project.
  29
  30
  31 Version History
  32 ---------------
  33 +------------------+-----------+---------------------------------+
  34 | *Date*           | *Version* | *Comment*                       |
  35 |                  |           |                                 |
  36 +------------------+-----------+---------------------------------+
  37 | October 20, 2017 | 5.0.0     | Yardstick for Euphrates release |
  38 |                  |           |                                 |
  39 +------------------+-----------+---------------------------------+
  40
  41
  42 Important Notes
  43 ---------------
  44
  45 The software delivered in the OPNFV Yardstick_ Project, comprising the
  46 *Yardstick framework*, the *Yardstick test cases* and the experimental
  47 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
  48 NFV-TST001_.
  49
  50 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  51 independent.
  52
  53 OPNFV Euphrates Release
  54 -----------------------
  55
  56 This Euphrates release provides *Yardstick* as a framework for NFVI testing
  57 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  58
  59 * Documentation generated with Sphinx
  60
  61   * User Guide
  62
  63   * Developer Guide
  64
  65   * Release notes (this document)
  66
  67   * Results
  68
  69 * Automated Yardstick test suite (daily, weekly)
  70
  71   * Jenkins Jobs for OPNFV community labs
  72
  73 * Automated Yardstick test results visualization
  74
  75   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  76     backend
  77
  78 * Yardstick framework source code
  79
  80 * Yardstick test cases yaml files
  81
  82 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  83
  84 For Euphrates release, the *Yardstick framework* is used for the following
  85 testing:
  86
  87 * OPNFV platform testing - generic test cases to measure the categories:
  88
  89   * Compute
  90
  91   * Network
  92
  93   * Storage
  94
  95 * OPNFV platform network service benchmarking(NSB)
  96
  97   * NSB
  98
  99 * Test cases for the following OPNFV Projects:
 100
 101   * Container4NFV
 102
 103   * High Availability
 104
 105   * IPv6
 106
 107   * KVM
 108
 109   * Parser
 110
 111   * StorPerf
 112
 113   * VSperf
 114
 115   * virtual Traffic Classifier
 116
 117 The *Yardstick framework* is developed in the OPNFV community, by the
 118 Yardstick_ team.
 119
 120 .. note:: The test case description template used for the Yardstick test cases
 121   is based on the document ETSI-ISG NFV-TST001_; the results report template
 122   used for the Yardstick results is based on the IEEE Std 829-2008.
 123
 124
 125 Release Data
 126 ------------
 127
 128 +--------------------------------+-----------------------+
 129 | **Project**                    | Yardstick             |
 130 |                                |                       |
 131 +--------------------------------+-----------------------+
 132 | **Repo/tag**                   | yardstick/opnfv-5.0.0 |
 133 |                                |                       |
 134 +--------------------------------+-----------------------+
 135 | **Yardstick Docker image tag** | opnfv-5.0.0           |
 136 |                                |                       |
 137 +--------------------------------+-----------------------+
 138 | **Release designation**        | Euphrates             |
 139 |                                |                       |
 140 +--------------------------------+-----------------------+
 141 | **Release date**               | October 20, 2017      |
 142 |                                |                       |
 143 +--------------------------------+-----------------------+
 144 | **Purpose of the delivery**    | OPNFV Euphrates 5.0.0 |
 145 |                                |                       |
 146 +--------------------------------+-----------------------+
 147
 148
 149 Deliverables
 150 ------------
 151
 152 Documents
 153 ^^^^^^^^^
 154
 155  - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html
 156
 157  - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html
 158
 159
 160 Software Deliverables
 161 ^^^^^^^^^^^^^^^^^^^^^
 162
 163
 164  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.0.0)
 165
 166
 167 New Contexts
 168 ############
 169
 170 +--------------+-------------------------------------------+
 171 | **Context**  | **Description**                           |
 172 |              |                                           |
 173 +--------------+-------------------------------------------+
 174 | *Heat*       | Models orchestration using OpenStack Heat |
 175 |              |                                           |
 176 +--------------+-------------------------------------------+
 177 | *Node*       | Models Baremetal, Controller, Compute     |
 178 |              |                                           |
 179 +--------------+-------------------------------------------+
 180 | *Standalone* | Models VM running on Non-Managed NFVi     |
 181 |              |                                           |
 182 +--------------+-------------------------------------------+
 183 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 184 |              |                                           |
 185 +--------------+-------------------------------------------+
 186
 187
 188 New Runners
 189 ###########
 190
 191 +--------------+-------------------------------------------------------+
 192 | **Runner**   | **Description**                                       |
 193 |              |                                                       |
 194 +--------------+-------------------------------------------------------+
 195 | *Arithmetic* | Steps every run arithmetically according to specified |
 196 |              | input value                                           |
 197 |              |                                                       |
 198 +--------------+-------------------------------------------------------+
 199 | *Duration*   | Runs for a specified period of time                   |
 200 |              |                                                       |
 201 +--------------+-------------------------------------------------------+
 202 | *Iteration*  | Runs for a specified number of iterations             |
 203 |              |                                                       |
 204 +--------------+-------------------------------------------------------+
 205 | *Sequence*   | Selects input value to a scenario from an input file  |
 206 |              | and runs all entries sequentially                     |
 207 |              |                                                       |
 208 +--------------+-------------------------------------------------------+
 209
 210
 211 New Scenarios
 212 #############
 213
 214 +----------------+-----------------------------------------------------+
 215 | **Category**   | **Delivered**                                       |
 216 |                |                                                     |
 217 +----------------+-----------------------------------------------------+
 218 | *Availability* | Attacker:                                           |
 219 |                |                                                     |
 220 |                | * baremetal, process                                |
 221 |                |                                                     |
 222 |                | HA tools:                                           |
 223 |                |                                                     |
 224 |                | * check host, openstack, process, service           |
 225 |                | * kill process                                      |
 226 |                | * start/stop service                                |
 227 |                |                                                     |
 228 |                | Monitor:                                            |
 229 |                |                                                     |
 230 |                | * command, process                                  |
 231 |                |                                                     |
 232 +----------------+-----------------------------------------------------+
 233 | *Compute*      | * cpuload                                           |
 234 |                |                                                     |
 235 |                | * cyclictest                                        |
 236 |                |                                                     |
 237 |                | * lmbench                                           |
 238 |                |                                                     |
 239 |                | * lmbench_cache                                     |
 240 |                |                                                     |
 241 |                | * perf                                              |
 242 |                |                                                     |
 243 |                | * unixbench                                         |
 244 |                |                                                     |
 245 |                | * ramspeed                                          |
 246 |                |                                                     |
 247 |                | * cachestat                                         |
 248 |                |                                                     |
 249 |                | * memeoryload                                       |
 250 |                |                                                     |
 251 |                | * computecapacity                                   |
 252 |                |                                                     |
 253 |                | * SpecCPU2006                                       |
 254 |                |                                                     |
 255 +----------------+-----------------------------------------------------+
 256 | *Networking*   | * iperf3                                            |
 257 |                |                                                     |
 258 |                | * netperf                                           |
 259 |                |                                                     |
 260 |                | * netperf_node                                      |
 261 |                |                                                     |
 262 |                | * ping                                              |
 263 |                |                                                     |
 264 |                | * ping6                                             |
 265 |                |                                                     |
 266 |                | * pktgen                                            |
 267 |                |                                                     |
 268 |                | * sfc                                               |
 269 |                |                                                     |
 270 |                | * sfc with tacker                                   |
 271 |                |                                                     |
 272 |                | * vtc instantion validation                         |
 273 |                |                                                     |
 274 |                | * vtc instantion validation with noisy neighbors    |
 275 |                |                                                     |
 276 |                | * vtc throughput                                    |
 277 |                |                                                     |
 278 |                | * vtc throughput in the presence of noisy neighbors |
 279 |                |                                                     |
 280 |                | * networkcapacity                                   |
 281 |                |                                                     |
 282 |                | * netutilization                                    |
 283 |                |                                                     |
 284 |                | * nstat                                             |
 285 |                |                                                     |
 286 |                | * pktgenDPDK                                        |
 287 |                |                                                     |
 288 +----------------+-----------------------------------------------------+
 289 | *Parser*       | Tosca2Heat                                          |
 290 |                |                                                     |
 291 +----------------+-----------------------------------------------------+
 292 | *Storage*      | fio                                                 |
 293 |                |                                                     |
 294 |                | bonnie++                                            |
 295 |                |                                                     |
 296 |                | storagecapacity                                     |
 297 |                |                                                     |
 298 +----------------+-----------------------------------------------------+
 299 | *StorPerf*     | storperf                                            |
 300 |                |                                                     |
 301 +----------------+-----------------------------------------------------+
 302 | *NSB*          | vPE thoughput test case                             |
 303 |                |                                                     |
 304 +----------------+-----------------------------------------------------+
 305
 306
 307
 308 New Test cases
 309 ^^^^^^^^^^^^^^
 310
 311 * Generic NFVI test cases
 312
 313  * OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006
 314
 315  * OPNFV_YARDSTICK_TCO79 - Bonnie++
 316
 317 * Kubernetes Test cases
 318
 319  * OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER
 320
 321  * OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM
 322
 323
 324 Version Change
 325 --------------
 326
 327 Module Version Changes
 328 ^^^^^^^^^^^^^^^^^^^^^^
 329
 330 This is the fifth tracked release of Yardstick. It is based on following
 331 upstream versions:
 332
 333 - OpenStack Ocata
 334
 335 - OpenDayLight Nitrogen
 336
 337 - ONOS Junco
 338
 339
 340 Document Version Changes
 341 ^^^^^^^^^^^^^^^^^^^^^^^^
 342
 343 This is the fifth tracked version of the Yardstick framework in OPNFV.
 344 It includes the following documentation updates:
 345
 346 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
 347   add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
 348   add "Yardstick user interface" chapter; Update Yardstick installation chapter;
 349
 350 - Yardstick Developer Guide
 351
 352 - Yardstick Release Notes for Yardstick: this document
 353
 354
 355 Feature additions
 356 ^^^^^^^^^^^^^^^^^
 357
 358 - Yardstick RESTful API support
 359
 360 - Network service benchmarking
 361
 362 - Stress testing with Bottlenecks team
 363
 364 - Yardstick framework improvement:
 365
 366   - yardstick report CLI
 367
 368   - Node context support OpenStack configuration via Ansible
 369
 370   - Https support
 371
 372   - Kubernetes context type
 373
 374 - Yardstick container local GUI
 375
 376 - Python 3 support
 377
 378
 379 Scenario Matrix
 380 ---------------
 381
 382 For Euphrates 5.0.0, Yardstick was tested on the following scenarios:
 383
 384 +--------------------------+------+---------+------+------+
 385 | Scenario                 | Apex | Compass | Fuel | Joid |
 386 +==========================+======+=========+======+======+
 387 | os-nosdn-nofeature-noha  |      |         | X    | X    |
 388 +--------------------------+------+---------+------+------+
 389 | os-nosdn-nofeature-ha    | X    | X       | X    | X    |
 390 +--------------------------+------+---------+------+------+
 391 | os-odl_l2-nofeature-ha   |      | X       | X    | X    |
 392 +--------------------------+------+---------+------+------+
 393 | os-odl_l2-nofeature-noha |      |         | X    |      |
 394 +--------------------------+------+---------+------+------+
 395 | os-odl_l3-nofeature-ha   | X    | X       | X    |      |
 396 +--------------------------+------+---------+------+------+
 397 | os-odl_l3-nofeature-noha |      |         | X    |      |
 398 +--------------------------+------+---------+------+------+
 399 | os-onos-sfc-ha           |      |         |      |      |
 400 +--------------------------+------+---------+------+------+
 401 | os-onos-nofeature-ha     |      | X       |      | X    |
 402 +--------------------------+------+---------+------+------+
 403 | os-onos-nofeature-noha   |      |         |      |      |
 404 +--------------------------+------+---------+------+------+
 405 | os-odl_l2-sfc-ha         |      |         | X    |      |
 406 +--------------------------+------+---------+------+------+
 407 | os-odl_l2-sfc-noha       |      |         | X    |      |
 408 +--------------------------+------+---------+------+------+
 409 | os-odl_l2-bgpvpn-ha      | X    |         | X    |      |
 410 +--------------------------+------+---------+------+------+
 411 | os-odl_l2-bgpvpn-noha    |      |         | X    |      |
 412 +--------------------------+------+---------+------+------+
 413 | os-nosdn-kvm-ha          | X    |         | X    |      |
 414 +--------------------------+------+---------+------+------+
 415 | os-nosdn-kvm-noha        |      |         | X    |      |
 416 +--------------------------+------+---------+------+------+
 417 | os-nosdn-ovs-ha          |      |         | X    |      |
 418 +--------------------------+------+---------+------+------+
 419 | os-nosdn-ovs-noha        |      |         | X    |      |
 420 +--------------------------+------+---------+------+------+
 421 | os-ocl-nofeature-ha      |      | X       |      |      |
 422 +--------------------------+------+---------+------+------+
 423 | os-nosdn-lxd-ha          |      |         |      | X    |
 424 +--------------------------+------+---------+------+------+
 425 | os-nosdn-lxd-noha        |      |         |      | X    |
 426 +--------------------------+------+---------+------+------+
 427 | os-nosdn-fdio-ha         | X    |         |      |      |
 428 +--------------------------+------+---------+------+------+
 429 | os-odl_l2-fdio-noha      | X    |         |      |      |
 430 +--------------------------+------+---------+------+------+
 431 | os-odl-gluon-noha        | X    |         |      |      |
 432 +--------------------------+------+---------+------+------+
 433 | os-nosdn-openo-ha        |      | X       |      |      |
 434 +--------------------------+------+---------+------+------+
 435 | os-nosdn-kvm_ovs_dpdk    |      |         | X    |      |
 436 | -noha                    |      |         |      |      |
 437 +--------------------------+------+---------+------+------+
 438 | os-nosdn-kvm_ovs_dpdk-ha |      |         | X    |      |
 439 +--------------------------+------+---------+------+------+
 440 | os-nosdn-kvm_ovs_dpdk    |      |         | X    |      |
 441 | _bar-ha                  |      |         |      |      |
 442 +--------------------------+------+---------+------+------+
 443 | os-nosdn-kvm_ovs_dpdk    |      |         | X    |      |
 444 | _bar-noha                |      |         |      |      |
 445 +--------------------------+------+---------+------+------+
 446 | opnfv_os-ovn-nofeature-  | X    |         |      |      |
 447 | noha_daily               |      |         |      |      |
 448 +--------------------------+------+---------+------+------+
 449
 450 Test results
 451 ------------
 452
 453 Test results are available in:
 454
 455  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 456
 457 The reporting pages can be found at:
 458
 459 +---------------+-------------------------------------------------------------------------------------+
 460 | apex          | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html         |
 461 +---------------+-------------------------------------------------------------------------------------+
 462 | compass       | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html      |
 463 +---------------+-------------------------------------------------------------------------------------+
 464 | fuel\@x86     | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html     |
 465 +---------------+-------------------------------------------------------------------------------------+
 466 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html |
 467 +---------------+-------------------------------------------------------------------------------------+
 468 | joid          | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html         |
 469 +---------------+-------------------------------------------------------------------------------------+
 470
 471 Known Issues/Faults
 472 ^^^^^^^^^^^^^^^^^^^
 473
 474
 475 Corrected Faults
 476 ^^^^^^^^^^^^^^^^
 477
 478 Euphrates 5.0.0:
 479
 480 +---------------------+--------------------------------------------+
 481 | **JIRA REFERENCE**  | **DESCRIPTION**                            |
 482 |                     |                                            |
 483 +---------------------+--------------------------------------------+
 484 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
 485 |                     | 'openstack -h'                             |
 486 +---------------------+--------------------------------------------+
 487 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they   |
 488 |                     | are dynamic                                |
 489 +---------------------+--------------------------------------------+
 490
 491
 492 Euphratess 5.0.0 known restrictions/issues
 493 ------------------------------------------
 494 +-----------+-----------+----------------------------------------------+
 495 | Installer | Scenario  | Issue                                        |
 496 +===========+===========+==============================================+
 497 | any       | \*-bgpvpn | Floating ips not supported. Some Test cases  |
 498 |           |           | related to floating ips are excluded.        |
 499 +-----------+-----------+----------------------------------------------+
 500 | any       | odl_l3-\* | Some test cases related to using floating IP |
 501 |           |           | addresses fail because of a known ODL bug.   |
 502 |           |           |                                              |
 503 +-----------+-----------+----------------------------------------------+
 504 | compass   | odl_l2-\* | In some test cases, VM instance will failed  |
 505 |           |           | raising network interfaces.                  |
 506 |           |           |                                              |
 507 +-----------+-----------+----------------------------------------------+
 508
 509
 510 Open JIRA tickets
 511 -----------------
 512
 513 +---------------------+----------------------------------------------+
 514 | **JIRA REFERENCE**  | **DESCRIPTION**                              |
 515 |                     |                                              |
 516 +---------------------+----------------------------------------------+
 517 | JIRA: YARDSTICK-412 | IPv6 test case should add support for newton |
 518 |                     |                                              |
 519 +---------------------+----------------------------------------------+
 520
 521
 522 Useful links
 523 ------------
 524
 525  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 526
 527  - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning
 528
 529  - Yardstick repo: https://git.opnfv.org/cgit/yardstick
 530
 531  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 532
 533  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 534
 535  - Yardstick IRC chanel: #opnfv-yardstick