docs/release/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6
   7 ============================================
   8 OPNFV Brahmaputra Release Note for Yardstick
   9 ============================================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/yardstick
  15
  16 .. _Dashboard: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
  17
  18 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the OPNFV Brahmaputra release
  25 for Yardstick framework as well as Yardstick_ Project deliverables.
  26
  27 License
  28 =======
  29
  30 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
  31 experimental framework are opensource software, licensed under the terms of the
  32 Apache License, Version 2.0.
  33
  34
  35 Version History
  36 ===============
  37
  38 +---------------+--------------------+---------------------------------+
  39 | *Date*        | *Version*          | *Comment*                       |
  40 |               |                    |                                 |
  41 +---------------+--------------------+---------------------------------+
  42 | Apr 27th,2016 |  3.0               | Brahmaputra release             |
  43 |               |                    |                                 |
  44 +---------------+--------------------+---------------------------------+
  45 | Mar 30th,2016 |  2.0               | Brahmaputra release             |
  46 |               |                    |                                 |
  47 +---------------+--------------------+---------------------------------+
  48 | Feb 25th,2016 |  1.0               | Brahmaputra release             |
  49 |               |                    |                                 |
  50 +---------------+--------------------+---------------------------------+
  51
  52
  53 Important Notes
  54 ===============
  55
  56 The software delivered in the OPNFV Yardstick_ Project, comprising the
  57 *Yardstick framework*, the *Yardstick test cases* and the experimental
  58 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
  59 NFV-TST001_.
  60
  61 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  62 independent.
  63
  64
  65 Summary
  66 =======
  67
  68 This Brahmaputra release provides *Yardstick* as a framework for NFVI testing
  69 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  70
  71 * Documentation generated with Sphinx
  72
  73   * User Guide
  74
  75   * Code Documentation
  76
  77   * Release notes (this document)
  78
  79   * Results
  80
  81 * Automated Yardstick test suite (daily, weekly)
  82
  83   * Jenkins Jobs for OPNFV community labs
  84
  85 * Automated Yardstick test results visualization
  86
  87   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB used as
  88     backend
  89
  90 * Yardstick framework source code
  91
  92 * Yardstick test cases yaml files
  93
  94 For Brahmaputra release, the *Yardstick framework* is used for the following
  95 testing:
  96
  97 * OPNFV platform testing - generic test cases to measure the categories:
  98
  99   * Compute
 100
 101   * Network
 102
 103   * Storage
 104
 105 * Test cases for the following OPNFV Projects:
 106
 107   * High Availability
 108
 109   * IPv6
 110
 111   * KVM
 112
 113   * Parser
 114
 115 * Test cases added in Brahmaputra2.0:
 116
 117   * virtual Traffic Classifier
 118
 119 The *Yardstick framework* is developed in the OPNFV community, by the
 120 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
 121 Project.
 122
 123 .. note:: The test case description template used for the Yardstick test cases
 124   is based on the document ETSI-ISG NFV-TST001_; the results report template
 125   used for the Yardstick results is based on the IEEE Std 829-2008.
 126
 127
 128 Release Data
 129 ============
 130
 131 +--------------------------------------+--------------------------------------+
 132 | **Project**                          | Yardstick                            |
 133 |                                      |                                      |
 134 +--------------------------------------+--------------------------------------+
 135 | **Repo/tag**                         | yardstick/brahmaputra.3.0            |
 136 |                                      |                                      |
 137 +--------------------------------------+--------------------------------------+
 138 | **Yardstick Docker image tag**       | brahmaputra.3.0                      |
 139 |                                      |                                      |
 140 +--------------------------------------+--------------------------------------+
 141 | **Release designation**              | Brahmaputra                          |
 142 |                                      |                                      |
 143 +--------------------------------------+--------------------------------------+
 144 | **Release date**                     | Apr 27th, 2016                       |
 145 |                                      |                                      |
 146 +--------------------------------------+--------------------------------------+
 147 | **Purpose of the delivery**          | OPNFV Brahmaputra release            |
 148 |                                      |                                      |
 149 +--------------------------------------+--------------------------------------+
 150
 151
 152 Version Change
 153 --------------
 154
 155 Module Version Changes
 156 ~~~~~~~~~~~~~~~~~~~~~~
 157
 158 This is the third tracked release of Yardstick. It is based on following
 159 upstream versions:
 160
 161 - OpenStack Liberty
 162
 163 - OpenDaylight Beryllium
 164
 165
 166 Document Version Changes
 167 ~~~~~~~~~~~~~~~~~~~~~~~~
 168
 169 This is the third tracked version of the Yardstick framework in OPNFV.
 170 It includes the following documentation updates:
 171
 172 - Yardstick User Guide: corrected faulty links
 173
 174 - Yardstick Code Documentation: no changes
 175
 176 - Yardstick Release Notes for Yardstick: this document
 177
 178 - Test Results report for Brahmaputra testing with Yardstick: updated listed of
 179 verified scenarios and limitations
 180
 181 Documentation updates on the second tracked version:
 182
 183 - Yardstick User Guide: added software architecture chapter
 184
 185 - Yardstick Code Documentation: no changes
 186
 187 - Yardstick Release Notes for Yardstick: this document
 188
 189 - Test Results report for Brahmaputra testing with Yardstick: added test cases
 190 and results for virtual Traffic Classifier
 191
 192
 193 Reason for Version
 194 ------------------
 195
 196 Feature additions
 197 ~~~~~~~~~~~~~~~~~
 198
 199 No new features.
 200
 201 Brahmaputra.2.0:
 202
 203 +----------------------------+------------------------------------------------+
 204 | **JIRA REFERENCE**         | **SLOGAN**                                     |
 205 |                            |                                                |
 206 +----------------------------+------------------------------------------------+
 207 | JIRA: YARDSTICK-227        | Heat HTTPS SSL support.                        |
 208 |                            |                                                |
 209 +----------------------------+------------------------------------------------+
 210
 211
 212 Corrected Faults
 213 ~~~~~~~~~~~~~~~~
 214
 215 No corrected faults.
 216
 217 Brahmaputra.2.0:
 218
 219 +----------------------------+------------------------------------------------+
 220 | **JIRA REFERENCE**         | **SLOGAN**                                     |
 221 |                            |                                                |
 222 +----------------------------+------------------------------------------------+
 223 | JIRA: -                    | Change copyrights for base scenario, runners,  |
 224 |                            | dispatchers, cover.                            |
 225 |                            |                                                |
 226 +----------------------------+------------------------------------------------+
 227 | JIRA: -                    | Update setup.py and dependencies               |
 228 |                            |                                                |
 229 +----------------------------+------------------------------------------------+
 230 | JIRA: -                    | Add missing dependencies to docker file        |
 231 |                            |                                                |
 232 +----------------------------+------------------------------------------------+
 233 | JIRA: -                    | Fix Heat template for noisy neighbors deploy   |
 234 |                            |                                                |
 235 +----------------------------+------------------------------------------------+
 236
 237 Known Faults
 238 ~~~~~~~~~~~~
 239
 240
 241 +----------------------------+------------------------------------------------+
 242 | **JIRA REFERENCE**         | **SLOGAN**                                     |
 243 |                            |                                                |
 244 +----------------------------+------------------------------------------------+
 245 | JIRA: YARDSTICK-175        | Running test suite, if a test cases running    |
 246 |                            | failed, the test is stopped.                   |
 247 |                            |                                                |
 248 +----------------------------+------------------------------------------------+
 249 | JIRA: YARDSTICK-176        | Fix plotter bug since Output format has been   |
 250 |                            | changed.                                       |
 251 |                            |                                                |
 252 +----------------------------+------------------------------------------------+
 253 | JIRA: YARDSTICK-216        | ArgsAlreadyParsedError: arguments already      |
 254 |                            | parsed: cannot register CLI option.            |
 255 |                            |                                                |
 256 +----------------------------+------------------------------------------------+
 257 | JIRA: YARDSTICK-231        | Installation instructions on Wiki not accurate |
 258 |                            |                                                |
 259 +----------------------------+------------------------------------------------+
 260
 261 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
 262   which were not fully verified, are listed in the OPNFV installer's release
 263   notes.
 264
 265
 266 Deliverables
 267 ------------
 268
 269 Software Deliverables
 270 ~~~~~~~~~~~~~~~~~~~~~
 271
 272 **Yardstick framework source code <brahmaputra.3.0>**
 273
 274 +--------------------------------------+--------------------------------------+
 275 | **Project**                          | Yardstick                            |
 276 |                                      |                                      |
 277 +--------------------------------------+--------------------------------------+
 278 | **Repo/tag**                         | yardstick/brahmaputra.3.0            |
 279 |                                      |                                      |
 280 +--------------------------------------+--------------------------------------+
 281 | **Yardstick Docker image tag**       | brahmaputra.3.0                      |
 282 |                                      |                                      |
 283 +--------------------------------------+--------------------------------------+
 284 | **Release designation**              | Brahmaputra                          |
 285 |                                      |                                      |
 286 +--------------------------------------+--------------------------------------+
 287 | **Release date**                     | Apr 27th, 2016                       |
 288 |                                      |                                      |
 289 +--------------------------------------+--------------------------------------+
 290 | **Purpose of the delivery**          | OPNFV Brahmaputra release            |
 291 |                                      |                                      |
 292 +--------------------------------------+--------------------------------------+
 293
 294 **Contexts**
 295
 296 +---------------------+-------------------------------------------------------+
 297 | **Context**         | **Description**                                       |
 298 |                     |                                                       |
 299 +---------------------+-------------------------------------------------------+
 300 | *Heat*              | Models orchestration using OpenStack Heat             |
 301 |                     |                                                       |
 302 +---------------------+-------------------------------------------------------+
 303 | *Node*              | Models Baremetal, Controller, Compute                 |
 304 |                     |                                                       |
 305 +---------------------+-------------------------------------------------------+
 306
 307
 308 **Runners**
 309
 310 +---------------------+-------------------------------------------------------+
 311 | **Runner**          | **Description**                                       |
 312 |                     |                                                       |
 313 +---------------------+-------------------------------------------------------+
 314 | *Arithmetic*        | Steps every run arithmetically according to specified |
 315 |                     | input value                                           |
 316 |                     |                                                       |
 317 +---------------------+-------------------------------------------------------+
 318 | *Duration*          | Runs for a specified period of time                   |
 319 |                     |                                                       |
 320 +---------------------+-------------------------------------------------------+
 321 | *Iteration*         | Runs for a specified number of iterations             |
 322 |                     |                                                       |
 323 +---------------------+-------------------------------------------------------+
 324 | *Sequence*          | Selects input value to a scenario from an input file  |
 325 |                     | and runs all entries sequentially                     |
 326 |                     |                                                       |
 327 +---------------------+-------------------------------------------------------+
 328
 329 **Scenarios**
 330
 331 +---------------------+-------------------------------------------------------+
 332 | **Category**        | **Delivered**                                         |
 333 |                     |                                                       |
 334 +---------------------+-------------------------------------------------------+
 335 | *Availability*      | Attacker:                                             |
 336 |                     |                                                       |
 337 |                     | * baremetal, process                                  |
 338 |                     |                                                       |
 339 |                     | HA tools:                                             |
 340 |                     |                                                       |
 341 |                     | * check host, openstack, process, service             |
 342 |                     | * kill process                                        |
 343 |                     | * start/stop service                                  |
 344 |                     |                                                       |
 345 |                     | Monitor:                                              |
 346 |                     |                                                       |
 347 |                     | * command, process                                    |
 348 |                     |                                                       |
 349 +---------------------+-------------------------------------------------------+
 350 | *Compute*           | * cpuload                                             |
 351 |                     |                                                       |
 352 |                     | * cyclictest                                          |
 353 |                     |                                                       |
 354 |                     | * lmbench                                             |
 355 |                     |                                                       |
 356 |                     | * perf                                                |
 357 |                     |                                                       |
 358 |                     | * unixbench                                           |
 359 |                     |                                                       |
 360 +---------------------+-------------------------------------------------------+
 361 | *Networking*        | * iperf3                                              |
 362 |                     |                                                       |
 363 |                     | * netperf                                             |
 364 |                     |                                                       |
 365 |                     | * ping                                                |
 366 |                     |                                                       |
 367 |                     | * ping6                                               |
 368 |                     |                                                       |
 369 |                     | * pktgen                                              |
 370 |                     |                                                       |
 371 |                     | * sfc                                                 |
 372 |                     |                                                       |
 373 |                     | * sfc with tacker                                     |
 374 |                     |                                                       |
 375 |                     | * vtc instantion validation                           |
 376 |                     |                                                       |
 377 |                     | * vtc instantion validation with noisy neighbors      |
 378 |                     |                                                       |
 379 |                     | * vtc throughput                                      |
 380 |                     |                                                       |
 381 |                     | * vtc throughput in the presence of noisy neighbors   |
 382 |                     |                                                       |
 383 +---------------------+-------------------------------------------------------+
 384 | *Parser*            | Tosca2Heat                                            |
 385 |                     |                                                       |
 386 +---------------------+-------------------------------------------------------+
 387 | *Storage*           | fio                                                   |
 388 |                     |                                                       |
 389 +---------------------+-------------------------------------------------------+
 390
 391 **API to Other Frameworks**
 392
 393 +---------------------+-------------------------------------------------------+
 394 | **Framework**       | **Description**                                       |
 395 |                     |                                                       |
 396 +---------------------+-------------------------------------------------------+
 397 | *ApexLake*          | Experimental framework that enables the user to       |
 398 |                     | validate NFVI from the perspective of a VNF.          |
 399 |                     | A virtual Traffic Classifier is utilized as VNF.      |
 400 |                     | Enables experiments with SR-IOV on Compute Node.      |
 401 |                     |                                                       |
 402 +---------------------+-------------------------------------------------------+
 403
 404 **Test Results Output**
 405
 406 +-----------------------------+-----------------------------------------------+
 407 | **Dispatcher**              | **Description**                               |
 408 |                             |                                               |
 409 +-----------------------------+-----------------------------------------------+
 410 |  file                       | Log to a file.                                |
 411 |                             |                                               |
 412 +-----------------------------+-----------------------------------------------+
 413 |  http                       | Post data to html.                            |
 414 |                             |                                               |
 415 +-----------------------------+-----------------------------------------------+
 416 |  influxdb                   | Post data to influxdB.                        |
 417 |                             |                                               |
 418 +-----------------------------+-----------------------------------------------+
 419
 420
 421 Delivered Test cases
 422 ~~~~~~~~~~~~~~~~~~~~
 423
 424 * Generic NFVI test cases
 425
 426   * OPNFV_YARDSTICK_TCOO1 - NW Performance
 427
 428   * OPNFV_YARDSTICK_TCOO2 - NW Latency
 429
 430   * OPNFV_YARDSTICK_TCOO5 - Storage Performance
 431
 432   * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
 433
 434   * OPNFV_YARDSTICK_TCOO9 - Packet Loss
 435
 436   * OPNFV_YARDSTICK_TCO10 - Memory Latency
 437
 438   * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
 439
 440   * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
 441
 442   * OPNFV_YARDSTICK_TCO14 - Processing Speed
 443
 444   * OPNFV_YARDSTICK_TCO24 - CPU Load
 445
 446   * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
 447
 448   * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
 449     Test
 450
 451
 452 * Test Cases for OPNFV HA Project:
 453
 454   * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
 455
 456   * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
 457
 458 * Test Case for OPNFV IPv6 Project:
 459
 460   * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
 461
 462 * Test Case for OPNFV KVM Project:
 463
 464   * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
 465
 466 * Test Case for OPNFV Parser Project:
 467
 468   * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
 469
 470 * Test Cases for Virtual Traffic Classifier:
 471
 472   * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
 473 Benchmarking Test
 474
 475   * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
 476 Benchmarking in presence of noisy neighbors Test
 477
 478   * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
 479
 480   * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
 481 presence of noisy neighbors Test