docs/release/release-notes/release-notes.rst

   1 License
   2 =======
   3
   4 OPNFV Euphrates release note for Yardstick Docs
   5 are licensed under a Creative Commons Attribution 4.0 International License.
   6 You should have received a copy of the license along with this.
   7 If not, see <http://creativecommons.org/licenses/by/4.0/>.
   8
   9 The *Yardstick framework*, the *Yardstick test cases* are open-source software,
  10  licensed under the terms of the Apache License, Version 2.0.
  11
  12 OPNFV Euphrates Release Note for Yardstick
  13 ==========================================
  14
  15 .. toctree::
  16    :maxdepth: 2
  17
  18 .. _Yardstick: https://wiki.opnfv.org/yardstick
  19
  20 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  21
  22 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  23
  24
  25 Abstract
  26 --------
  27
  28 This document describes the release note of Yardstick project.
  29
  30
  31 Version History
  32 ---------------
  33 +-------------------+-----------+---------------------------------+
  34 | *Date*            | *Version* | *Comment*                       |
  35 |                   |           |                                 |
  36 +-------------------+-----------+---------------------------------+
  37 | December 15, 2017 | 5.1.0     | Yardstick for Euphrates release |
  38 |                   |           |                                 |
  39 +-------------------+-----------+---------------------------------+
  40 | October 20, 2017  | 5.0.0     | Yardstick for Euphrates release |
  41 |                   |           |                                 |
  42 +-------------------+-----------+---------------------------------+
  43
  44
  45 Important Notes
  46 ---------------
  47
  48 The software delivered in the OPNFV Yardstick_ Project, comprising the
  49 *Yardstick framework*, the *Yardstick test cases* and the experimental
  50 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
  51 NFV-TST001_.
  52
  53 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  54 independent.
  55
  56 OPNFV Euphrates Release
  57 -----------------------
  58
  59 This Euphrates release provides *Yardstick* as a framework for NFVI testing
  60 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  61
  62 * Documentation generated with Sphinx
  63
  64   * User Guide
  65
  66   * Developer Guide
  67
  68   * Release notes (this document)
  69
  70   * Results
  71
  72 * Automated Yardstick test suite (daily, weekly)
  73
  74   * Jenkins Jobs for OPNFV community labs
  75
  76 * Automated Yardstick test results visualization
  77
  78   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  79     backend
  80
  81 * Yardstick framework source code
  82
  83 * Yardstick test cases yaml files
  84
  85 * Yardstick plug-in configuration yaml files, plug-in install/remove scripts
  86
  87 For Euphrates release, the *Yardstick framework* is used for the following
  88 testing:
  89
  90 * OPNFV platform testing - generic test cases to measure the categories:
  91
  92   * Compute
  93
  94   * Network
  95
  96   * Storage
  97
  98 * OPNFV platform network service benchmarking(NSB)
  99
 100   * NSB
 101
 102 * Test cases for the following OPNFV Projects:
 103
 104   * Container4NFV
 105
 106   * High Availability
 107
 108   * IPv6
 109
 110   * KVM
 111
 112   * Parser
 113
 114   * StorPerf
 115
 116   * VSperf
 117
 118   * virtual Traffic Classifier
 119
 120 The *Yardstick framework* is developed in the OPNFV community, by the
 121 Yardstick_ team.
 122
 123 .. note:: The test case description template used for the Yardstick test cases
 124   is based on the document ETSI-ISG NFV-TST001_; the results report template
 125   used for the Yardstick results is based on the IEEE Std 829-2008.
 126
 127
 128 Release Data
 129 ------------
 130
 131 +--------------------------------+-----------------------+
 132 | **Project**                    | Yardstick             |
 133 |                                |                       |
 134 +--------------------------------+-----------------------+
 135 | **Repo/tag**                   | yardstick/opnfv-5.1.0 |
 136 |                                |                       |
 137 +--------------------------------+-----------------------+
 138 | **Yardstick Docker image tag** | opnfv-5.1.0           |
 139 |                                |                       |
 140 +--------------------------------+-----------------------+
 141 | **Release designation**        | Euphrates             |
 142 |                                |                       |
 143 +--------------------------------+-----------------------+
 144 | **Release date**               | December 15, 2017     |
 145 |                                |                       |
 146 +--------------------------------+-----------------------+
 147 | **Purpose of the delivery**    | OPNFV Euphrates 5.1.0 |
 148 |                                |                       |
 149 +--------------------------------+-----------------------+
 150
 151
 152 Deliverables
 153 ------------
 154
 155 Documents
 156 ^^^^^^^^^
 157
 158  - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/user/userguide/index.html
 159
 160  - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/yardstick/docs/testing/developer/devguide/index.html
 161
 162
 163 Software Deliverables
 164 ^^^^^^^^^^^^^^^^^^^^^
 165
 166
 167  - The Yardstick Docker image: https://hub.docker.com/r/opnfv/yardstick (tag: opnfv-5.1.0)
 168
 169
 170 New Contexts
 171 ############
 172
 173 +--------------+-------------------------------------------+
 174 | **Context**  | **Description**                           |
 175 |              |                                           |
 176 +--------------+-------------------------------------------+
 177 | *Heat*       | Models orchestration using OpenStack Heat |
 178 |              |                                           |
 179 +--------------+-------------------------------------------+
 180 | *Node*       | Models Baremetal, Controller, Compute     |
 181 |              |                                           |
 182 +--------------+-------------------------------------------+
 183 | *Standalone* | Models VM running on Non-Managed NFVi     |
 184 |              |                                           |
 185 +--------------+-------------------------------------------+
 186 | *Kubernetes* | Models VM running on Non-Managed NFVi     |
 187 |              |                                           |
 188 +--------------+-------------------------------------------+
 189
 190
 191 New Runners
 192 ###########
 193
 194 +--------------+-------------------------------------------------------+
 195 | **Runner**   | **Description**                                       |
 196 |              |                                                       |
 197 +--------------+-------------------------------------------------------+
 198 | *Arithmetic* | Steps every run arithmetically according to specified |
 199 |              | input value                                           |
 200 |              |                                                       |
 201 +--------------+-------------------------------------------------------+
 202 | *Duration*   | Runs for a specified period of time                   |
 203 |              |                                                       |
 204 +--------------+-------------------------------------------------------+
 205 | *Iteration*  | Runs for a specified number of iterations             |
 206 |              |                                                       |
 207 +--------------+-------------------------------------------------------+
 208 | *Sequence*   | Selects input value to a scenario from an input file  |
 209 |              | and runs all entries sequentially                     |
 210 |              |                                                       |
 211 +--------------+-------------------------------------------------------+
 212
 213
 214 New Scenarios
 215 #############
 216
 217 +----------------+-----------------------------------------------------+
 218 | **Category**   | **Delivered**                                       |
 219 |                |                                                     |
 220 +----------------+-----------------------------------------------------+
 221 | *Availability* | Attacker:                                           |
 222 |                |                                                     |
 223 |                | * baremetal, process                                |
 224 |                |                                                     |
 225 |                | HA tools:                                           |
 226 |                |                                                     |
 227 |                | * check host, openstack, process, service           |
 228 |                | * kill process                                      |
 229 |                | * start/stop service                                |
 230 |                |                                                     |
 231 |                | Monitor:                                            |
 232 |                |                                                     |
 233 |                | * command, process                                  |
 234 |                |                                                     |
 235 +----------------+-----------------------------------------------------+
 236 | *Compute*      | * cpuload                                           |
 237 |                |                                                     |
 238 |                | * cyclictest                                        |
 239 |                |                                                     |
 240 |                | * lmbench                                           |
 241 |                |                                                     |
 242 |                | * lmbench_cache                                     |
 243 |                |                                                     |
 244 |                | * perf                                              |
 245 |                |                                                     |
 246 |                | * unixbench                                         |
 247 |                |                                                     |
 248 |                | * ramspeed                                          |
 249 |                |                                                     |
 250 |                | * cachestat                                         |
 251 |                |                                                     |
 252 |                | * memeoryload                                       |
 253 |                |                                                     |
 254 |                | * computecapacity                                   |
 255 |                |                                                     |
 256 |                | * SpecCPU2006                                       |
 257 |                |                                                     |
 258 +----------------+-----------------------------------------------------+
 259 | *Networking*   | * iperf3                                            |
 260 |                |                                                     |
 261 |                | * netperf                                           |
 262 |                |                                                     |
 263 |                | * netperf_node                                      |
 264 |                |                                                     |
 265 |                | * ping                                              |
 266 |                |                                                     |
 267 |                | * ping6                                             |
 268 |                |                                                     |
 269 |                | * pktgen                                            |
 270 |                |                                                     |
 271 |                | * sfc                                               |
 272 |                |                                                     |
 273 |                | * sfc with tacker                                   |
 274 |                |                                                     |
 275 |                | * vtc instantion validation                         |
 276 |                |                                                     |
 277 |                | * vtc instantion validation with noisy neighbors    |
 278 |                |                                                     |
 279 |                | * vtc throughput                                    |
 280 |                |                                                     |
 281 |                | * vtc throughput in the presence of noisy neighbors |
 282 |                |                                                     |
 283 |                | * networkcapacity                                   |
 284 |                |                                                     |
 285 |                | * netutilization                                    |
 286 |                |                                                     |
 287 |                | * nstat                                             |
 288 |                |                                                     |
 289 |                | * pktgenDPDK                                        |
 290 |                |                                                     |
 291 +----------------+-----------------------------------------------------+
 292 | *Parser*       | Tosca2Heat                                          |
 293 |                |                                                     |
 294 +----------------+-----------------------------------------------------+
 295 | *Storage*      | fio                                                 |
 296 |                |                                                     |
 297 |                | bonnie++                                            |
 298 |                |                                                     |
 299 |                | storagecapacity                                     |
 300 |                |                                                     |
 301 +----------------+-----------------------------------------------------+
 302 | *StorPerf*     | storperf                                            |
 303 |                |                                                     |
 304 +----------------+-----------------------------------------------------+
 305 | *NSB*          | vPE thoughput test case                             |
 306 |                |                                                     |
 307 +----------------+-----------------------------------------------------+
 308
 309
 310
 311 New Test cases
 312 ^^^^^^^^^^^^^^
 313
 314 * Generic NFVI test cases
 315
 316  * OPNFV_YARDSTICK_TCO78 - SPEC CPU 2006
 317
 318  * OPNFV_YARDSTICK_TCO79 - Bonnie++
 319
 320 * Kubernetes Test cases
 321
 322  * OPNFV_YARDSTICK_TCO80 - NETWORK LATENCY BETWEEN CONTAINER
 323
 324  * OPNFV_YARDSTICK_TCO81 - NETWORK LATENCY BETWEEN CONTAINER AND VM
 325
 326
 327 Version Change
 328 --------------
 329
 330 Module Version Changes
 331 ^^^^^^^^^^^^^^^^^^^^^^
 332
 333 This is the fifth tracked release of Yardstick. It is based on following
 334 upstream versions:
 335
 336 - OpenStack Ocata
 337
 338 - OpenDayLight Nitrogen
 339
 340 - ONOS Junco
 341
 342
 343 Document Version Changes
 344 ^^^^^^^^^^^^^^^^^^^^^^^^
 345
 346 This is the fifth tracked version of the Yardstick framework in OPNFV.
 347 It includes the following documentation updates:
 348
 349 - Yardstick User Guide: add "network service benchmarking(NSB)" chapter;
 350   add "Yardstick - NSB Testing -Installation" chapter; add "Yardstick API" chapter;
 351   add "Yardstick user interface" chapter; Update Yardstick installation chapter;
 352
 353 - Yardstick Developer Guide
 354
 355 - Yardstick Release Notes for Yardstick: this document
 356
 357
 358 Feature additions
 359 ^^^^^^^^^^^^^^^^^
 360
 361 - Yardstick RESTful API support
 362
 363 - Network service benchmarking
 364
 365 - Stress testing with Bottlenecks team
 366
 367 - Yardstick framework improvement:
 368
 369   - yardstick report CLI
 370
 371   - Node context support OpenStack configuration via Ansible
 372
 373   - Https support
 374
 375   - Kubernetes context type
 376
 377 - Yardstick container local GUI
 378
 379 - Python 3 support
 380
 381
 382 Scenario Matrix
 383 ---------------
 384
 385 For Euphrates 5.0.0, Yardstick was tested on the following scenarios:
 386
 387 +--------------------------+------+---------+------+------+
 388 | Scenario                 | Apex | Compass | Fuel | Joid |
 389 +==========================+======+=========+======+======+
 390 | os-nosdn-nofeature-noha  |      |         | X    | X    |
 391 +--------------------------+------+---------+------+------+
 392 | os-nosdn-nofeature-ha    | X    | X       | X    | X    |
 393 +--------------------------+------+---------+------+------+
 394 | os-odl_l2-nofeature-ha   |      | X       | X    | X    |
 395 +--------------------------+------+---------+------+------+
 396 | os-odl_l2-nofeature-noha |      |         | X    |      |
 397 +--------------------------+------+---------+------+------+
 398 | os-odl_l3-nofeature-ha   | X    | X       | X    |      |
 399 +--------------------------+------+---------+------+------+
 400 | os-odl_l3-nofeature-noha |      |         | X    |      |
 401 +--------------------------+------+---------+------+------+
 402 | os-onos-sfc-ha           |      |         |      |      |
 403 +--------------------------+------+---------+------+------+
 404 | os-onos-nofeature-ha     |      | X       |      | X    |
 405 +--------------------------+------+---------+------+------+
 406 | os-onos-nofeature-noha   |      |         |      |      |
 407 +--------------------------+------+---------+------+------+
 408 | os-odl_l2-sfc-ha         |      |         | X    |      |
 409 +--------------------------+------+---------+------+------+
 410 | os-odl_l2-sfc-noha       |      |         | X    |      |
 411 +--------------------------+------+---------+------+------+
 412 | os-odl_l2-bgpvpn-ha      | X    |         | X    |      |
 413 +--------------------------+------+---------+------+------+
 414 | os-odl_l2-bgpvpn-noha    |      |         | X    |      |
 415 +--------------------------+------+---------+------+------+
 416 | os-nosdn-kvm-ha          | X    |         | X    |      |
 417 +--------------------------+------+---------+------+------+
 418 | os-nosdn-kvm-noha        |      |         | X    |      |
 419 +--------------------------+------+---------+------+------+
 420 | os-nosdn-ovs-ha          |      |         | X    |      |
 421 +--------------------------+------+---------+------+------+
 422 | os-nosdn-ovs-noha        |      |         | X    |      |
 423 +--------------------------+------+---------+------+------+
 424 | os-ocl-nofeature-ha      |      | X       |      |      |
 425 +--------------------------+------+---------+------+------+
 426 | os-nosdn-lxd-ha          |      |         |      | X    |
 427 +--------------------------+------+---------+------+------+
 428 | os-nosdn-lxd-noha        |      |         |      | X    |
 429 +--------------------------+------+---------+------+------+
 430 | os-nosdn-fdio-ha         | X    |         |      |      |
 431 +--------------------------+------+---------+------+------+
 432 | os-odl_l2-fdio-noha      | X    |         |      |      |
 433 +--------------------------+------+---------+------+------+
 434 | os-odl-gluon-noha        | X    |         |      |      |
 435 +--------------------------+------+---------+------+------+
 436 | os-nosdn-openo-ha        |      | X       |      |      |
 437 +--------------------------+------+---------+------+------+
 438 | os-nosdn-kvm_ovs_dpdk    |      |         | X    |      |
 439 | -noha                    |      |         |      |      |
 440 +--------------------------+------+---------+------+------+
 441 | os-nosdn-kvm_ovs_dpdk-ha |      |         | X    |      |
 442 +--------------------------+------+---------+------+------+
 443 | os-nosdn-kvm_ovs_dpdk    |      |         | X    |      |
 444 | _bar-ha                  |      |         |      |      |
 445 +--------------------------+------+---------+------+------+
 446 | os-nosdn-kvm_ovs_dpdk    |      |         | X    |      |
 447 | _bar-noha                |      |         |      |      |
 448 +--------------------------+------+---------+------+------+
 449 | opnfv_os-ovn-nofeature-  | X    |         |      |      |
 450 | noha_daily               |      |         |      |      |
 451 +--------------------------+------+---------+------+------+
 452
 453 Test results
 454 ------------
 455
 456 Test results are available in:
 457
 458  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 459
 460 The reporting pages can be found at:
 461
 462 +---------------+-------------------------------------------------------------------------------------+
 463 | apex          | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-apex.html         |
 464 +---------------+-------------------------------------------------------------------------------------+
 465 | compass       | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-compass.html      |
 466 +---------------+-------------------------------------------------------------------------------------+
 467 | fuel\@x86     | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@x86.html     |
 468 +---------------+-------------------------------------------------------------------------------------+
 469 | fuel\@aarch64 | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-fuel@aarch64.html |
 470 +---------------+-------------------------------------------------------------------------------------+
 471 | joid          | http://testresults.opnfv.org/reporting/euphrates/yardstick/status-joid.html         |
 472 +---------------+-------------------------------------------------------------------------------------+
 473
 474 Known Issues/Faults
 475 ^^^^^^^^^^^^^^^^^^^
 476
 477
 478 Corrected Faults
 479 ^^^^^^^^^^^^^^^^
 480
 481 Euphrates 5.1.0:
 482
 483 +---------------------+-------------------------------------------------------------------------+
 484 | **JIRA REFERENCE**  | **DESCRIPTION**                                                         |
 485 |                     |                                                                         |
 486 +---------------------+-------------------------------------------------------------------------+
 487 | JIRA: YARDSTICK-841 | Fix various NSB license issues                                          |
 488 +---------------------+-------------------------------------------------------------------------+
 489 | JIRA: YARDSTICK-73  | How To Work with Test Cases                                             |
 490 +---------------------+-------------------------------------------------------------------------+
 491 | JIRA: YARDSTICK-500 | VNF testing documentation                                               |
 492 +---------------------+-------------------------------------------------------------------------+
 493 | JIRA: YARDSTICK-826 | Allow overriding Heat IP addresses to match traffic generator profile   |
 494 +---------------------+-------------------------------------------------------------------------+
 495 | JIRA: YARDSTICK-828 | Refactor doc/testing/user/userguide "Yardstick Installation"            |
 496 +---------------------+-------------------------------------------------------------------------+
 497 | JIRA: YARDSTICK-830 | build_yardstick_image Ansible mount module doesn't work on Ubuntu 14.04 |
 498 +---------------------+-------------------------------------------------------------------------+
 499 | JIRA: YARDSTICK-833 | ansible_common transform password into lower case                       |
 500 +---------------------+-------------------------------------------------------------------------+
 501 | JIRA: YARDSTICK-847 | tc006, tc079, tc082 miss grafana dashboard in local deployment          |
 502 +---------------------+-------------------------------------------------------------------------+
 503 | JIRA: YARDSTICK-849 | kill process do not accurately kill the process like "nova-api"         |
 504 +---------------------+-------------------------------------------------------------------------+
 505 | JIRA: YARDSTICK-850 | tc023 miss description and tc050-58 wrong description                   |
 506 +---------------------+-------------------------------------------------------------------------+
 507 | JIRA: YARDSTICK-852 | tc078 cpu2006 fails in some situation                                   |
 508 +---------------------+-------------------------------------------------------------------------+
 509 | JIRA: YARDSTICK-854 | yardstick docker lack of trex_client                                    |
 510 +---------------------+-------------------------------------------------------------------------+
 511 | JIRA: YARDSTICK-867 | testcase tc078 have no data stored or dashboard to show results         |
 512 +---------------------+-------------------------------------------------------------------------+
 513 | JIRA: YARDSTICK-871 | Remove img_modify_playbook assignation in build_yardstick_image.yml     |
 514 +---------------------+-------------------------------------------------------------------------+
 515 | JIRA: YARDSTICK-829 | "nsb_setup.sh" doesn't parse the controller IP correctly                |
 516 +---------------------+-------------------------------------------------------------------------+
 517 | JIRA: YARDSTICK-839 | NSB Prox BM test cases to be fixed for incorporating scale-up           |
 518 +---------------------+-------------------------------------------------------------------------+
 519 | JIRA: YARDSTICK-840 | NSB Prox test documentation of vPE and LW-AFTR test cases               |
 520 +---------------------+-------------------------------------------------------------------------+
 521 | JIRA: YARDSTICK-848 | NSB "Prox" : Cleanup duplicated traffic profile                         |
 522 +---------------------+-------------------------------------------------------------------------+
 523
 524
 525
 526
 527 Euphrates 5.0.0:
 528
 529 +---------------------+--------------------------------------------+
 530 | **JIRA REFERENCE**  | **DESCRIPTION**                            |
 531 |                     |                                            |
 532 +---------------------+--------------------------------------------+
 533 | JIRA: YARDSTICK-599 | Could not load EntryPoint.parse when using |
 534 |                     | 'openstack -h'                             |
 535 +---------------------+--------------------------------------------+
 536 | JIRA: YARDSTICK-602 | Don't rely on staic ip addresses as they   |
 537 |                     | are dynamic                                |
 538 +---------------------+--------------------------------------------+
 539
 540
 541 Euphratess 5.0.0 known restrictions/issues
 542 ------------------------------------------
 543 +-----------+-----------+----------------------------------------------+
 544 | Installer | Scenario  | Issue                                        |
 545 +===========+===========+==============================================+
 546 | any       | \*-bgpvpn | Floating ips not supported. Some Test cases  |
 547 |           |           | related to floating ips are excluded.        |
 548 +-----------+-----------+----------------------------------------------+
 549 | any       | odl_l3-\* | Some test cases related to using floating IP |
 550 |           |           | addresses fail because of a known ODL bug.   |
 551 |           |           |                                              |
 552 +-----------+-----------+----------------------------------------------+
 553 | compass   | odl_l2-\* | In some test cases, VM instance will failed  |
 554 |           |           | raising network interfaces.                  |
 555 |           |           |                                              |
 556 +-----------+-----------+----------------------------------------------+
 557
 558
 559 Useful links
 560 ------------
 561
 562  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 563
 564  - wiki Yardstick Euphrates release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Euphrates+Release+Planning
 565
 566  - Yardstick repo: https://git.opnfv.org/cgit/yardstick
 567
 568  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 569
 570  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 571
 572  - Yardstick IRC chanel: #opnfv-yardstick