docs/release/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6
   7 ============================================
   8 OPNFV Brahmaputra Release Note for Yardstick
   9 ============================================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/yardstick
  15
  16 .. _Dashboard: http://130.211.154.108/grafana/dashboard/db/yardstick-main
  17
  18 .. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the OPNFV Brahmaputra release
  25 for Yardstick framework as well as Yardstick_ Project deliverables.
  26
  27 License
  28 =======
  29
  30 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
  31 experimental framework are opensource software, licensed under the terms of the
  32 Apache License, Version 2.0.
  33
  34
  35 Version History
  36 ===============
  37
  38 +---------------+--------------------+---------------------------------+
  39 | *Date*        | *Version*          | *Comment*                       |
  40 |               |                    |                                 |
  41 +---------------+--------------------+---------------------------------+
  42 | Mar 30th,2016 |  2.0               | Brahmaputra release             |
  43 |               |                    |                                 |
  44 +---------------+--------------------+---------------------------------+
  45 | Feb 25th,2016 |  1.0               | Brahmaputra release             |
  46 |               |                    |                                 |
  47 +---------------+--------------------+---------------------------------+
  48
  49
  50 Important Notes
  51 ===============
  52
  53 The software delivered in the OPNFV Yardstick_ Project, comprising the
  54 *Yardstick framework*, the *Yardstick test cases* and the experimental
  55 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
  56 NFV-TST001_.
  57
  58 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  59 independent.
  60
  61
  62 Summary
  63 =======
  64
  65 This Brahmaputra release provides *Yardstick* as a framework for NFVI testing
  66 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  67
  68 * Documentation generated with Sphinx
  69
  70   * User Guide
  71
  72   * Code Documentation
  73
  74   * Release notes (this document)
  75
  76   * Results
  77
  78 * Automated Yardstick test suite (daily, weekly)
  79
  80   * Jenkins Jobs for OPNFV community labs
  81
  82 * Automated Yardstick test results visualization
  83
  84   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB used as
  85     backend
  86
  87 * Yardstick framework source code
  88
  89 * Yardstick test cases yaml files
  90
  91 For Brahmaputra release, the *Yardstick framework* is used for the following
  92 testing:
  93
  94 * OPNFV platform testing - generic test cases to measure the categories:
  95
  96   * Compute
  97
  98   * Network
  99
 100   * Storage
 101
 102 * Test cases for the following OPNFV Projects:
 103
 104   * High Availability
 105
 106   * IPv6
 107
 108   * KVM
 109
 110   * Parser
 111
 112 * Test cases added in Brahmaputra2.0:
 113
 114   * virtual Traffic Classifier
 115
 116 The *Yardstick framework* is developed in the OPNFV community, by the
 117 Yardstick_ team. The *virtual Traffic Classifier* is a part of the Yardstick
 118 Project.
 119
 120 .. note:: The test case description template used for the Yardstick test cases
 121   is based on the document ETSI-ISG NFV-TST001_; the results report template
 122   used for the Yardstick results is based on the IEEE Std 829-2008.
 123
 124
 125 Release Data
 126 ============
 127
 128 +--------------------------------------+--------------------------------------+
 129 | **Project**                          | Yardstick                            |
 130 |                                      |                                      |
 131 +--------------------------------------+--------------------------------------+
 132 | **Repo/tag**                         | yardstick/brahmaputra.2.0            |
 133 |                                      |                                      |
 134 +--------------------------------------+--------------------------------------+
 135 | **Yardstick Docker image tag**       | brahmaputra.2.0                      |
 136 |                                      |                                      |
 137 +--------------------------------------+--------------------------------------+
 138 | **Release designation**              | Brahmaputra                          |
 139 |                                      |                                      |
 140 +--------------------------------------+--------------------------------------+
 141 | **Release date**                     | Mar 30th, 2016                       |
 142 |                                      |                                      |
 143 +--------------------------------------+--------------------------------------+
 144 | **Purpose of the delivery**          | OPNFV Brahmaputra release            |
 145 |                                      |                                      |
 146 +--------------------------------------+--------------------------------------+
 147
 148
 149 Version Change
 150 --------------
 151
 152 Module Version Changes
 153 ~~~~~~~~~~~~~~~~~~~~~~
 154
 155 This is the second tracked release of Yardstick. It is based on following
 156 upstream versions:
 157
 158 - OpenStack Liberty
 159
 160 - OpenDaylight Beryllium
 161
 162
 163 Document Version Changes
 164 ~~~~~~~~~~~~~~~~~~~~~~~~
 165
 166 This is the second tracked version of the Yardstick framework in OPNFV.
 167 It includes the following documentation updates:
 168
 169 - Yardstick User Guide: added software architecture chapter
 170
 171 - Yardstick Code Documentation: no changes
 172
 173 - Yardstick Release Notes for Yardstick: this document
 174
 175 - Test Results report for Brahmaputra testing with Yardstick: added test cases
 176 and results for virtual Traffic Classifier
 177
 178
 179 Reason for Version
 180 ------------------
 181
 182 Feature additions
 183 ~~~~~~~~~~~~~~~~~
 184
 185 +----------------------------+------------------------------------------------+
 186 | **JIRA REFERENCE**         | **SLOGAN**                                     |
 187 |                            |                                                |
 188 +----------------------------+------------------------------------------------+
 189 | JIRA: YARDSTICK-227        | Heat HTTPS SSL support.                        |
 190 |                            |                                                |
 191 +----------------------------+------------------------------------------------+
 192
 193
 194 Corrected Faults
 195 ~~~~~~~~~~~~~~~~
 196
 197 +----------------------------+------------------------------------------------+
 198 | **JIRA REFERENCE**         | **SLOGAN**                                     |
 199 |                            |                                                |
 200 +----------------------------+------------------------------------------------+
 201 | JIRA: -                    | Change copyrights for base scenario, runners,  |
 202 |                            | dispatchers, cover.                            |
 203 |                            |                                                |
 204 +----------------------------+------------------------------------------------+
 205 | JIRA: -                    | Update setup.py and dependencies               |
 206 |                            |                                                |
 207 +----------------------------+------------------------------------------------+
 208 | JIRA: -                    | Add missing dependencies to docker file        |
 209 |                            |                                                |
 210 +----------------------------+------------------------------------------------+
 211 | JIRA: -                    | Fix Heat template for noisy neighbors deploy   |
 212 |                            |                                                |
 213 +----------------------------+------------------------------------------------+
 214
 215 Known Faults
 216 ~~~~~~~~~~~~
 217
 218
 219 +----------------------------+------------------------------------------------+
 220 | **JIRA REFERENCE**         | **SLOGAN**                                     |
 221 |                            |                                                |
 222 +----------------------------+------------------------------------------------+
 223 | JIRA: YARDSTICK-175        | Running test suite, if a test cases running    |
 224 |                            | failed, the test is stopped.                   |
 225 |                            |                                                |
 226 +----------------------------+------------------------------------------------+
 227 | JIRA: YARDSTICK-176        | Fix plotter bug since Output format has been   |
 228 |                            | changed.                                       |
 229 |                            |                                                |
 230 +----------------------------+------------------------------------------------+
 231 | JIRA: YARDSTICK-216        | ArgsAlreadyParsedError: arguments already      |
 232 |                            | parsed: cannot register CLI option.            |
 233 |                            |                                                |
 234 +----------------------------+------------------------------------------------+
 235 | JIRA: YARDSTICK-231        | Installation instructions on Wiki not accurate |
 236 |                            |                                                |
 237 +----------------------------+------------------------------------------------+
 238
 239 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
 240   which were not fully verified, are listed in the OPNFV installer's release
 241   notes.
 242
 243
 244 Deliverables
 245 ------------
 246
 247 Software Deliverables
 248 ~~~~~~~~~~~~~~~~~~~~~
 249
 250 **Yardstick framework source code <brahmaputra.2.0>**
 251
 252 +--------------------------------------+--------------------------------------+
 253 | **Project**                          | Yardstick                            |
 254 |                                      |                                      |
 255 +--------------------------------------+--------------------------------------+
 256 | **Repo/tag**                         | yardstick/brahmaputra.2.0            |
 257 |                                      |                                      |
 258 +--------------------------------------+--------------------------------------+
 259 | **Yardstick Docker image tag**       | brahmaputra.2.0                      |
 260 |                                      |                                      |
 261 +--------------------------------------+--------------------------------------+
 262 | **Release designation**              | Brahmaputra                          |
 263 |                                      |                                      |
 264 +--------------------------------------+--------------------------------------+
 265 | **Release date**                     | Mar 30th, 2016                       |
 266 |                                      |                                      |
 267 +--------------------------------------+--------------------------------------+
 268 | **Purpose of the delivery**          | OPNFV Brahmaputra release            |
 269 |                                      |                                      |
 270 +--------------------------------------+--------------------------------------+
 271
 272 **Contexts**
 273
 274 +---------------------+-------------------------------------------------------+
 275 | **Context**         | **Description**                                       |
 276 |                     |                                                       |
 277 +---------------------+-------------------------------------------------------+
 278 | *Heat*              | Models orchestration using OpenStack Heat             |
 279 |                     |                                                       |
 280 +---------------------+-------------------------------------------------------+
 281 | *Node*              | Models Baremetal, Controller, Compute                 |
 282 |                     |                                                       |
 283 +---------------------+-------------------------------------------------------+
 284
 285
 286 **Runners**
 287
 288 +---------------------+-------------------------------------------------------+
 289 | **Runner**          | **Description**                                       |
 290 |                     |                                                       |
 291 +---------------------+-------------------------------------------------------+
 292 | *Arithmetic*        | Steps every run arithmetically according to specified |
 293 |                     | input value                                           |
 294 |                     |                                                       |
 295 +---------------------+-------------------------------------------------------+
 296 | *Duration*          | Runs for a specified period of time                   |
 297 |                     |                                                       |
 298 +---------------------+-------------------------------------------------------+
 299 | *Iteration*         | Runs for a specified number of iterations             |
 300 |                     |                                                       |
 301 +---------------------+-------------------------------------------------------+
 302 | *Sequence*          | Selects input value to a scenario from an input file  |
 303 |                     | and runs all entries sequentially                     |
 304 |                     |                                                       |
 305 +---------------------+-------------------------------------------------------+
 306
 307 **Scenarios**
 308
 309 +---------------------+-------------------------------------------------------+
 310 | **Category**        | **Delivered**                                         |
 311 |                     |                                                       |
 312 +---------------------+-------------------------------------------------------+
 313 | *Availability*      | Attacker:                                             |
 314 |                     |                                                       |
 315 |                     | * baremetal, process                                  |
 316 |                     |                                                       |
 317 |                     | HA tools:                                             |
 318 |                     |                                                       |
 319 |                     | * check host, openstack, process, service             |
 320 |                     | * kill process                                        |
 321 |                     | * start/stop service                                  |
 322 |                     |                                                       |
 323 |                     | Monitor:                                              |
 324 |                     |                                                       |
 325 |                     | * command, process                                    |
 326 |                     |                                                       |
 327 +---------------------+-------------------------------------------------------+
 328 | *Compute*           | * cpuload                                             |
 329 |                     |                                                       |
 330 |                     | * cyclictest                                          |
 331 |                     |                                                       |
 332 |                     | * lmbench                                             |
 333 |                     |                                                       |
 334 |                     | * perf                                                |
 335 |                     |                                                       |
 336 |                     | * unixbench                                           |
 337 |                     |                                                       |
 338 +---------------------+-------------------------------------------------------+
 339 | *Networking*        | * iperf3                                              |
 340 |                     |                                                       |
 341 |                     | * netperf                                             |
 342 |                     |                                                       |
 343 |                     | * ping                                                |
 344 |                     |                                                       |
 345 |                     | * ping6                                               |
 346 |                     |                                                       |
 347 |                     | * pktgen                                              |
 348 |                     |                                                       |
 349 |                     | * sfc                                                 |
 350 |                     |                                                       |
 351 |                     | * sfc with tacker                                     |
 352 |                     |                                                       |
 353 |                     | * vtc instantion validation                           |
 354 |                     |                                                       |
 355 |                     | * vtc instantion validation with noisy neighbors      |
 356 |                     |                                                       |
 357 |                     | * vtc throughput                                      |
 358 |                     |                                                       |
 359 |                     | * vtc throughput in the presence of noisy neighbors   |
 360 |                     |                                                       |
 361 +---------------------+-------------------------------------------------------+
 362 | *Parser*            | Tosca2Heat                                            |
 363 |                     |                                                       |
 364 +---------------------+-------------------------------------------------------+
 365 | *Storage*           | fio                                                   |
 366 |                     |                                                       |
 367 +---------------------+-------------------------------------------------------+
 368
 369 **API to Other Frameworks**
 370
 371 +---------------------+-------------------------------------------------------+
 372 | **Framework**       | **Description**                                       |
 373 |                     |                                                       |
 374 +---------------------+-------------------------------------------------------+
 375 | *ApexLake*          | Experimental framework that enables the user to       |
 376 |                     | validate NFVI from the perspective of a VNF.          |
 377 |                     | A virtual Traffic Classifier is utilized as VNF.      |
 378 |                     | Enables experiments with SR-IOV on Compute Node.      |
 379 |                     |                                                       |
 380 +---------------------+-------------------------------------------------------+
 381
 382 **Test Results Output**
 383
 384 +-----------------------------+-----------------------------------------------+
 385 | **Dispatcher**              | **Description**                               |
 386 |                             |                                               |
 387 +-----------------------------+-----------------------------------------------+
 388 |  file                       | Log to a file.                                |
 389 |                             |                                               |
 390 +-----------------------------+-----------------------------------------------+
 391 |  http                       | Post data to html.                            |
 392 |                             |                                               |
 393 +-----------------------------+-----------------------------------------------+
 394 |  influxdb                   | Post data to influxdB.                        |
 395 |                             |                                               |
 396 +-----------------------------+-----------------------------------------------+
 397
 398
 399 Delivered Test cases
 400 ~~~~~~~~~~~~~~~~~~~~
 401
 402 * Generic NFVI test cases
 403
 404   * OPNFV_YARDSTICK_TCOO1 - NW Performance
 405
 406   * OPNFV_YARDSTICK_TCOO2 - NW Latency
 407
 408   * OPNFV_YARDSTICK_TCOO5 - Storage Performance
 409
 410   * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
 411
 412   * OPNFV_YARDSTICK_TCOO9 - Packet Loss
 413
 414   * OPNFV_YARDSTICK_TCO10 - Memory Latency
 415
 416   * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
 417
 418   * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
 419
 420   * OPNFV_YARDSTICK_TCO14 - Processing Speed
 421
 422   * OPNFV_YARDSTICK_TCO24 - CPU Load
 423
 424   * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
 425
 426   * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
 427     Test
 428
 429
 430 * Test Cases for OPNFV HA Project:
 431
 432   * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
 433
 434   * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
 435
 436 * Test Case for OPNFV IPv6 Project:
 437
 438   * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
 439
 440 * Test Case for OPNFV KVM Project:
 441
 442   * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
 443
 444 * Test Case for OPNFV Parser Project:
 445
 446   * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca
 447
 448 * Test Cases for Virtual Traffic Classifier:
 449
 450   * OPNFV_YARDSTICK_TC006 - Virtual Traffic Classifier Data Plane Throughput
 451 Benchmarking Test
 452
 453   * OPNFV_YARDSTICK_TC007 - Virtual Traffic Classifier Data Plane Throughput
 454 Benchmarking in presence of noisy neighbors Test
 455
 456   * OPNFV_YARDSTICK_TC020 - Virtual Traffic Classifier Instantiation Test
 457
 458   * OPNFV_YARDSTICK_TC021 - Virtual Traffic Classifier Instantiation in
 459 presence of noisy neighbors Test