docs/release/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6
   7 ============================================
   8 OPNFV Colorado Release Note for Yardstick
   9 ============================================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/yardstick
  15
  16 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  17
  18 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  19
  20
  21 Abstract
  22 ========
  23
  24 This document describes the release note of Yardstick project.
  25
  26
  27 License
  28 =======
  29
  30 OPNFV Colorado release note for Yardstick Docs
  31 are licensed under a Creative Commons Attribution 4.0 International License.
  32 You should have received a copy of the license along with this.
  33 If not, see <http://creativecommons.org/licenses/by/4.0/>.
  34
  35 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
  36 experimental framework are opensource software, licensed under the terms of the
  37 Apache License, Version 2.0.
  38
  39
  40 Version History
  41 ===============
  42
  43 +----------------+--------------------+---------------------------------+
  44 | *Date*         | *Version*          | *Comment*                       |
  45 |                |                    |                                 |
  46 +----------------+--------------------+---------------------------------+
  47 | Aug 22nd, 2016 |  1.0               | Yardstick for Colorado release  |
  48 |                |                    |                                 |
  49 +----------------+--------------------+---------------------------------+
  50
  51
  52 Important Notes
  53 ===============
  54
  55 The software delivered in the OPNFV Yardstick_ Project, comprising the
  56 *Yardstick framework*, the *Yardstick test cases* and the experimental
  57 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
  58 NFV-TST001_.
  59
  60 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  61 independent.
  62
  63
  64 Summary
  65 =======
  66
  67 This Colorado release provides *Yardstick* as a framework for NFVI testing
  68 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  69
  70 * Documentation generated with Sphinx
  71
  72   * User Guide
  73
  74   * Code Documentation
  75
  76   * Release notes (this document)
  77
  78   * Results
  79
  80 * Automated Yardstick test suite (daily, weekly)
  81
  82   * Jenkins Jobs for OPNFV community labs
  83
  84 * Automated Yardstick test results visualization
  85
  86   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB is used as
  87     backend
  88
  89 * Yardstick framework source code
  90
  91 * Yardstick test cases yaml files
  92
  93 * Yardstick pliug-in configration yaml files, plug-in install/remove scripts
  94
  95 For Colorado release, the *Yardstick framework* is used for the following
  96 testing:
  97
  98 * OPNFV platform testing - generic test cases to measure the categories:
  99
 100   * Compute
 101
 102   * Network
 103
 104   * Storage
 105
 106 * Test cases for the following OPNFV Projects:
 107
 108   * High Availability
 109
 110   * IPv6
 111
 112   * KVM
 113
 114   * Parser
 115
 116   * StorPerf
 117
 118   * VSperf
 119
 120   * virtual Traffic Classifier
 121
 122 The *Yardstick framework* is developed in the OPNFV community, by the
 123 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
 124 Project.
 125
 126 .. note:: The test case description template used for the Yardstick test cases
 127   is based on the document ETSI-ISG NFV-TST001_; the results report template
 128   used for the Yardstick results is based on the IEEE Std 829-2008.
 129
 130
 131 Release Data
 132 ============
 133
 134 +--------------------------------------+--------------------------------------+
 135 | **Project**                          | Yardstick                            |
 136 |                                      |                                      |
 137 +--------------------------------------+--------------------------------------+
 138 | **Repo/tag**                         | yardstick/colorado.1.0               |
 139 |                                      |                                      |
 140 +--------------------------------------+--------------------------------------+
 141 | **Yardstick Docker image tag**       | colorado.1.0                         |
 142 |                                      |                                      |
 143 +--------------------------------------+--------------------------------------+
 144 | **Release designation**              | Colorado base release                |
 145 |                                      |                                      |
 146 +--------------------------------------+--------------------------------------+
 147 | **Release date**                     | September 22 2016                    |
 148 |                                      |                                      |
 149 +--------------------------------------+--------------------------------------+
 150 | **Purpose of the delivery**          | Colorado base release                |
 151 |                                      |                                      |
 152 +--------------------------------------+--------------------------------------+
 153
 154
 155 Deliverables
 156 ============
 157
 158 Software Deliverables
 159 ---------------------
 160
 161 **Yardstick framework source code <colorado.1.0>**
 162
 163 +--------------------------------------+--------------------------------------+
 164 | **Project**                          | Yardstick                            |
 165 |                                      |                                      |
 166 +--------------------------------------+--------------------------------------+
 167 | **Repo/tag**                         | yardstick/colorado.1.0               |
 168 |                                      |                                      |
 169 +--------------------------------------+--------------------------------------+
 170 | **Yardstick Docker image tag**       | colorado.1.0                         |
 171 |                                      |                                      |
 172 +--------------------------------------+--------------------------------------+
 173 | **Release designation**              | colorado                             |
 174 |                                      |                                      |
 175 +--------------------------------------+--------------------------------------+
 176 | **Release date**                     | September 22th, 2016                 |
 177 |                                      |                                      |
 178 +--------------------------------------+--------------------------------------+
 179 | **Purpose of the delivery**          | OPNFV Colorado release               |
 180 |                                      |                                      |
 181 +--------------------------------------+--------------------------------------+
 182
 183
 184 **Contexts**
 185
 186 +---------------------+-------------------------------------------------------+
 187 | **Context**         | **Description**                                       |
 188 |                     |                                                       |
 189 +---------------------+-------------------------------------------------------+
 190 | *Heat*              | Models orchestration using OpenStack Heat             |
 191 |                     |                                                       |
 192 +---------------------+-------------------------------------------------------+
 193 | *Node*              | Models Baremetal, Controller, Compute                 |
 194 |                     |                                                       |
 195 +---------------------+-------------------------------------------------------+
 196
 197
 198 **Runners**
 199
 200 +---------------------+-------------------------------------------------------+
 201 | **Runner**          | **Description**                                       |
 202 |                     |                                                       |
 203 +---------------------+-------------------------------------------------------+
 204 | *Arithmetic*        | Steps every run arithmetically according to specified |
 205 |                     | input value                                           |
 206 |                     |                                                       |
 207 +---------------------+-------------------------------------------------------+
 208 | *Duration*          | Runs for a specified period of time                   |
 209 |                     |                                                       |
 210 +---------------------+-------------------------------------------------------+
 211 | *Iteration*         | Runs for a specified number of iterations             |
 212 |                     |                                                       |
 213 +---------------------+-------------------------------------------------------+
 214 | *Sequence*          | Selects input value to a scenario from an input file  |
 215 |                     | and runs all entries sequentially                     |
 216 |                     |                                                       |
 217 +---------------------+-------------------------------------------------------+
 218
 219
 220 **Scenarios**
 221
 222 +---------------------+-------------------------------------------------------+
 223 | **Category**        | **Delivered**                                         |
 224 |                     |                                                       |
 225 +---------------------+-------------------------------------------------------+
 226 | *Availability*      | Attacker:                                             |
 227 |                     |                                                       |
 228 |                     | * baremetal, process                                  |
 229 |                     |                                                       |
 230 |                     | HA tools:                                             |
 231 |                     |                                                       |
 232 |                     | * check host, openstack, process, service             |
 233 |                     | * kill process                                        |
 234 |                     | * start/stop service                                  |
 235 |                     |                                                       |
 236 |                     | Monitor:                                              |
 237 |                     |                                                       |
 238 |                     | * command, process                                    |
 239 |                     |                                                       |
 240 +---------------------+-------------------------------------------------------+
 241 | *Compute*           | * cpuload                                             |
 242 |                     |                                                       |
 243 |                     | * cyclictest                                          |
 244 |                     |                                                       |
 245 |                     | * lmbench                                             |
 246 |                     |                                                       |
 247 |                     | * lmbench_cache                                       |
 248 |                     |                                                       |
 249 |                     | * perf                                                |
 250 |                     |                                                       |
 251 |                     | * unixbench                                           |
 252 |                     |                                                       |
 253 |                     | * ramspeed                                            |
 254 |                     |                                                       |
 255 |                     | * cachestat                                           |
 256 |                     |                                                       |
 257 |                     | * memeoryload                                         |
 258 |                     |                                                       |
 259 |                     | * computecapacity                                     |
 260 |                     |                                                       |
 261 +---------------------+-------------------------------------------------------+
 262 | *Networking*        | * iperf3                                              |
 263 |                     |                                                       |
 264 |                     | * netperf                                             |
 265 |                     |                                                       |
 266 |                     | * netperf_node                                        |
 267 |                     |                                                       |
 268 |                     | * ping                                                |
 269 |                     |                                                       |
 270 |                     | * ping6                                               |
 271 |                     |                                                       |
 272 |                     | * pktgen                                              |
 273 |                     |                                                       |
 274 |                     | * sfc                                                 |
 275 |                     |                                                       |
 276 |                     | * sfc with tacker                                     |
 277 |                     |                                                       |
 278 |                     | * vtc instantion validation                           |
 279 |                     |                                                       |
 280 |                     | * vtc instantion validation with noisy neighbors      |
 281 |                     |                                                       |
 282 |                     | * vtc throughput                                      |
 283 |                     |                                                       |
 284 |                     | * vtc throughput in the presence of noisy neighbors   |
 285 |                     |                                                       |
 286 |                     | * networkcapacity                                     |
 287 |                     |                                                       |
 288 |                     | * netutilization                                      |
 289 |                     |                                                       |
 290 +---------------------+-------------------------------------------------------+
 291 | *Parser*            | Tosca2Heat                                            |
 292 |                     |                                                       |
 293 +---------------------+-------------------------------------------------------+
 294 | *Storage*           | fio                                                   |
 295 |                     |                                                       |
 296 |                     | storagecapacity                                       |
 297 |                     |                                                       |
 298 +---------------------+-------------------------------------------------------+
 299 | *StorPerf*          | storperf                                              |
 300 |                     |                                                       |
 301 +---------------------+-------------------------------------------------------+
 302
 303
 304 **API to Other Frameworks**
 305
 306 +---------------------+-------------------------------------------------------+
 307 | **Framework**       | **Description**                                       |
 308 |                     |                                                       |
 309 +---------------------+-------------------------------------------------------+
 310 | *ApexLake*          | Experimental framework that enables the user to       |
 311 |                     | validate NFVI from the perspective of a VNF.          |
 312 |                     | A virtual Traffic Classifier is utilized as VNF.      |
 313 |                     | Enables experiments with SR-IOV on Compute Node.      |
 314 |                     |                                                       |
 315 +---------------------+-------------------------------------------------------+
 316
 317
 318 **Test Results Output**
 319
 320 +-----------------------------+-----------------------------------------------+
 321 | **Dispatcher**              | **Description**                               |
 322 |                             |                                               |
 323 +-----------------------------+-----------------------------------------------+
 324 |  file                       | Log to a file.                                |
 325 |                             |                                               |
 326 +-----------------------------+-----------------------------------------------+
 327 |  http                       | Post data to html.                            |
 328 |                             |                                               |
 329 +-----------------------------+-----------------------------------------------+
 330 |  influxdb                   | Post data to influxDB.                        |
 331 |                             |                                               |
 332 +-----------------------------+-----------------------------------------------+
 333
 334
 335 Delivered Test cases
 336 --------------------
 337
 338 * Generic NFVI test cases
 339
 340   * OPNFV_YARDSTICK_TCOO1 - NW Performance
 341
 342   * OPNFV_YARDSTICK_TCOO2 - NW Latency
 343
 344   * OPNFV_YARDSTICK_TCOO4 - Cache Utilization
 345
 346   * OPNFV_YARDSTICK_TCOO5 - Storage Performance
 347
 348   * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
 349
 350   * OPNFV_YARDSTICK_TCOO9 - Packet Loss
 351
 352   * OPNFV_YARDSTICK_TCO10 - Memory Latency
 353
 354   * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
 355
 356   * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
 357
 358   * OPNFV_YARDSTICK_TCO14 - Processing Speed
 359
 360   * OPNFV_YARDSTICK_TCO24 - CPU Load
 361
 362   * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
 363
 364   * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
 365     Test
 366
 367   * OPNFV_YARDSTICK_TCO42 - Network Performance
 368
 369   * OPNFV_YARDSTICK_TCO43 - Network Latency Between NFVI Nodes
 370
 371   * OPNFV_YARDSTICK_TCO44 - Memory Utilization
 372
 373   * OPNFV_YARDSTICK_TCO55 - Compute Capacity
 374
 375   * OPNFV_YARDSTICK_TCO61 - Network Utilization
 376
 377   * OPNFV_YARDSTICK_TCO63 - Storage Capacity
 378
 379   * OPNFV_YARDSTICK_TCO69 - Memory Bandwidth
 380
 381   * OPNFV_YARDSTICK_TCO70 - Latency, Memory Utilization, Throughput, Packet
 382     Loss
 383
 384   * OPNFV_YARDSTICK_TCO71 - Latency, Cache Utilization, Throughput, Packet Loss
 385
 386   * OPNFV_YARDSTICK_TCO72 - Latency, Network Utilization, Throughput, Packet
 387     Loss
 388
 389   * OPNFV_YARDSTICK_TC073 - Network Latency and Throughput Between Nodes
 390
 391   * OPNFV_YARDSTICK_TCO75 - Network Capacity and Scale
 392
 393 * Test Cases for OPNFV HA Project:
 394
 395   * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
 396
 397   * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
 398
 399   * OPNFV_YARDSTICK_TCO45 - HA: Control node Openstack service down - neutron
 400     server
 401
 402   * OPNFV_YARDSTICK_TC046 - HA: Control node Openstack service down - keystone
 403
 404   * OPNFV_YARDSTICK_TCO47 - HA: Control node Openstack service down - glance
 405     api
 406
 407   * OPNFV_YARDSTICK_TC048 - HA: Control node Openstack service down - cinder
 408     api
 409
 410   * OPNFV_YARDSTICK_TCO49 - HA: Control node Openstack service down - swift
 411     proxy
 412
 413   * OPNFV_YARDSTICK_TC050 - HA: OpenStack Controller Node Network High
 414     Availability
 415
 416   * OPNFV_YARDSTICK_TCO51 - HA: OpenStack Controller Node CPU Overload High
 417     Availability
 418
 419   * OPNFV_YARDSTICK_TC052 - HA: OpenStack Controller Node Disk I/O Block High
 420     Availability
 421
 422   * OPNFV_YARDSTICK_TCO53 - HA: OpenStack Controller Load Balance Service High
 423     Availability
 424
 425   * OPNFV_YARDSTICK_TC054 - HA: OpenStack Virtual IP High Availability
 426
 427 * Test Case for OPNFV IPv6 Project:
 428
 429   * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
 430
 431 * Test Case for OPNFV KVM Project:
 432
 433   * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
 434
 435 * Test Case for OPNFV Parser Project:
 436
 437   * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
 438
 439 * Test Case for OPNFV StorPerf Project:
 440
 441   * OPNFV_YARDSTICK_TCO74 - Storperf
 442
 443 * Test Cases for Virtual Traffic Classifier:
 444
 445   * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
 446 Benchmarking Test
 447
 448   * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
 449 Benchmarking in presence of noisy neighbors Test
 450
 451   * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
 452
 453   * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
 454 presence of noisy neighbors Test
 455
 456
 457 Version Change
 458 ==============
 459
 460 Module Version Changes
 461 ----------------------
 462
 463 This is the third tracked release of Yardstick. It is based on following
 464 upstream versions:
 465
 466 - OpenStack Mitaka
 467
 468 - OpenDaylight Beryllium
 469
 470
 471 Document Version Changes
 472 ------------------------
 473
 474 This is the first tracked version of the Yardstick framework in OPNFV.
 475 It includes the following documentation updates:
 476
 477 - Yardstick User Guide: added yardstick plugin chapter; added Store Other
 478 Project's Test Results in InfluxDB chapter; Refine yardstick instantion chapter.
 479
 480 - Yardstick Code Documentation: no changes
 481
 482 - Yardstick Release Notes for Yardstick: this document
 483
 484 - Test Results report for Colorado testing with Yardstick: updated listed of
 485 verified scenarios and limitations
 486
 487
 488 Reason for Version
 489 ==================
 490 * TODO *
 491
 492 Feature additions
 493 -----------------
 494 * TODO *
 495
 496 Corrected Faults
 497 ----------------
 498 * TODO *
 499
 500 Known Issues/Faults
 501 ------------
 502  - IPv6 support
 503  - Boot up VM failed in joid-os-nosdn-lxd-ha and joid-os-nosdn-lxd-noha scenarios
 504  - Yardstick CI job timeout in fuel-os-onos-nofeature-ha scenario
 505  - SSH timeout in apex-os-onos-sfc-ha, apex-os-onos-nofeature-ha and apex-os-odl_l3-nofeature-ha scenarios
 506  - Scp /home/stack/overcloudrc failed in apex-os-nosdn-ovs-noha and apex-os-odl_l2-sfc-noha scenarios
 507
 508 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
 509   which were not fully verified, are listed in the OPNFV installer's release
 510   notes.
 511
 512
 513 Colorado known restrictions/issues
 514 ==================================
 515 +-----------+-----------+----------------------------------------------+
 516 | Installer | Scenario  |  Issue                                       |
 517 +===========+===========+==============================================+
 518 | any       | *-bgpvpn  | floating ips not supported. Some Test cases  |
 519 |           |           | related to floating ips are excluded.        |
 520 +-----------+-----------+----------------------------------------------+
 521 * TODO *
 522
 523
 524 Test results
 525 ============
 526
 527 Test results are available in:
 528
 529  - jenkins logs on CI: https://build.opnfv.org/ci/view/yardstick/
 530
 531
 532 Open JIRA tickets
 533 =================
 534
 535 +------------------+-----------------------------------------------+
 536 |   JIRA           |         Description                           |
 537 +==================+===============================================+
 538 +------------------+-----------------------------------------------+
 539 +------------------+-----------------------------------------------+
 540 +------------------+-----------------------------------------------+
 541 +------------------+-----------------------------------------------+
 542 +------------------+-----------------------------------------------+
 543
 544
 545 Useful links
 546 ============
 547
 548  - wiki project page: https://wiki.opnfv.org/display/yardstick/Yardstick
 549
 550  - wiki Yardstick Colorado release planing page: https://wiki.opnfv.org/display/yardstick/Yardstick+Colorado+Release+Planning
 551
 552  - wiki Yardstick Colorado release jira page: https://wiki.opnfv.org/display/yardstick/Jira+Yardstick-Colorado
 553
 554  - Yardstick repo: https://git.opnfv.org/cgit/yardstick
 555
 556  - Yardstick CI dashboard: https://build.opnfv.org/ci/view/yardstick
 557
 558  - Yardstick grafana dashboard: http://testresults.opnfv.org/grafana/
 559
 560  - Yardstick IRC chanel: #opnfv-yardstick
 561