docs/release/release-notes.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2 .. License.
   3 .. http://creativecommons.org/licenses/by/4.0
   4 .. (c) OPNFV, Ericsson AB and others.
   5
   6
   7 ============================================
   8 OPNFV Brahmaputra Release Note for Yardstick
   9 ============================================
  10
  11 .. toctree::
  12    :maxdepth: 2
  13
  14 .. _Yardstick: https://wiki.opnfv.org/yardstick
  15
  16 .. _Dashboard: http://130.211.154.108/grafana/dashboard/db/yardstick-main
  17
  18 .. _NFV-TST001: https://docbox.etsi.org/ISG/NFV/Open/Drafts/TST001_-_Pre-deployment_Validation/
  19
  20
  21 Abstract
  22 ========
  23
  24 This document compiles the release notes for the OPNFV Brahmaputra release
  25 for Yardstick framework as well as Yardstick_ Project deliverables.
  26
  27 License
  28 =======
  29
  30 The *Yardstick framework*, the *Yardstick test cases* and the *ApexLake*
  31 experimental framework are opensource software, licensed under the terms of the
  32 Apache License, Version 2.0.
  33
  34
  35 Version History
  36 ===============
  37
  38 +---------------+--------------------+---------------------------------+
  39 | *Date*        | *Version*          | *Comment*                       |
  40 |               |                    |                                 |
  41 +---------------+--------------------+---------------------------------+
  42 | Feb 25th,2016 |  1.0               | Brahmaputra release             |
  43 |               |                    |                                 |
  44 +---------------+--------------------+---------------------------------+
  45
  46
  47 Important Notes
  48 ===============
  49
  50 The software delivered in the OPNFV Yardstick_ Project, comprising the
  51 *Yardstick framework*, the *Yardstick test cases* and the experimental
  52 framework *Apex Lake* is a realization of the methodology in ETSI-ISG
  53 NFV-TST001_.
  54
  55 The *Yardstick* framework is *installer*, *infrastructure* and *application*
  56 independent.
  57
  58
  59 Summary
  60 =======
  61
  62 This Brahmaputra release provides *Yardstick* as a framework for NFVI testing
  63 and OPNFV feature testing, automated in the OPNFV CI pipeline, including:
  64
  65 * Documentation generated with Sphinx
  66
  67   * User Guide
  68
  69   * Code Documentation
  70
  71   * Release notes (this document)
  72
  73   * Results
  74
  75 * Automated Yardstick test suite (daily, weekly)
  76
  77   * Jenkins Jobs for OPNFV community labs
  78
  79 * Automated Yardstick test results visualization
  80
  81   * Dashboard_ using Grafana (user:opnfv/password: opnfv), influxDB used as
  82     backend
  83
  84 * Yardstick framework source code
  85
  86 * Yardstick test cases yaml files
  87
  88 For Brahmaputra release, the *Yardstick framework* is used for the following
  89 testing:
  90
  91 * OPNFV platform testing - generic test cases to measure the categories:
  92
  93   * Compute
  94
  95   * Network
  96
  97   * Storage
  98
  99 * Test cases for the following OPNFV Projects:
 100
 101   * High Availability
 102
 103   * IPv6
 104
 105   * KVM
 106
 107   * Parser
 108
 109 The *Yardstick framework* is developed in the OPNFV community, by the
 110 Yardstick_ team.
 111
 112 .. note:: The test case description template used for the Yardstick test cases
 113   is based on the document ETSI-ISG NFV-TST001_; the results report template
 114   used for the Yardstick results is based on the IEEE Std 829-2008.
 115
 116
 117 Release Data
 118 ============
 119
 120 +--------------------------------------+--------------------------------------+
 121 | **Project**                          | Yardstick                            |
 122 |                                      |                                      |
 123 +--------------------------------------+--------------------------------------+
 124 | **Repo/tag**                         | yardstick/brahmaputra.1.0            |
 125 |                                      |                                      |
 126 +--------------------------------------+--------------------------------------+
 127 | **Yardstick Docker image tag**       | brahmaputra.1.0                      |
 128 |                                      |                                      |
 129 +--------------------------------------+--------------------------------------+
 130 | **Release designation**              | Brahmaputra                          |
 131 |                                      |                                      |
 132 +--------------------------------------+--------------------------------------+
 133 | **Release date**                     | Feb 25th, 2016                       |
 134 |                                      |                                      |
 135 +--------------------------------------+--------------------------------------+
 136 | **Purpose of the delivery**          | OPNFV Brahmaputra release            |
 137 |                                      |                                      |
 138 +--------------------------------------+--------------------------------------+
 139
 140
 141 Version Change
 142 --------------
 143
 144 Module Version Changes
 145 ~~~~~~~~~~~~~~~~~~~~~~
 146
 147 This is the first tracked release of Yardstick. It is based on following
 148 upstream versions:
 149
 150 - OpenStack Liberty
 151
 152 - OpenDaylight Beryllium
 153
 154
 155 Document Version Changes
 156 ~~~~~~~~~~~~~~~~~~~~~~~~
 157
 158 This is the first tracked version of the Yardstick framework in OPNFV.
 159 It includes the following documentation:
 160
 161 - Yardstick User Guide
 162
 163 - Yardstick Code Documentation
 164
 165 - Yardstick Release Notes for Yardstick
 166
 167 - Test Results report for Brahmaputra testing with Yardstick
 168
 169
 170 Reason for Version
 171 ------------------
 172
 173 Feature additions
 174 ~~~~~~~~~~~~~~~~~
 175
 176 This is the first tracked version of OPNFV Yardstick.
 177
 178
 179 Corrected Faults
 180 ~~~~~~~~~~~~~~~~
 181
 182 This is the first tracked version of OPNFV Yardstick.
 183
 184
 185 Known Faults
 186 ~~~~~~~~~~~~
 187
 188
 189 +----------------------------+------------------------------------------------+
 190 | **JIRA REFERENCE**         | **SLOGAN**                                     |
 191 |                            |                                                |
 192 +----------------------------+------------------------------------------------+
 193 | JIRA: YARDSTICK-175        | Running test suite, if a test cases running    |
 194 |                            | failed, the test is stopped.                   |
 195 |                            |                                                |
 196 +----------------------------+------------------------------------------------+
 197 | JIRA: YARDSTICK-176        | Fix plotter bug since Output format has been   |
 198 |                            | changed.                                       |
 199 |                            |                                                |
 200 +----------------------------+------------------------------------------------+
 201 | JIRA: YARDSTICK-216        | ArgsAlreadyParsedError: arguments already      |
 202 |                            | parsed: cannot register CLI option.            |
 203 |                            |                                                |
 204 +----------------------------+------------------------------------------------+
 205
 206 .. note:: The faults not related to *Yardstick* framework, addressing scenarios
 207   which were not fully verified, are listed in the OPNFV installer's release
 208   notes.
 209
 210
 211 Deliverables
 212 ------------
 213
 214 Software Deliverables
 215 ~~~~~~~~~~~~~~~~~~~~~
 216
 217 **Yardstick framework source code <brahmaputra.1.0>**
 218
 219 +--------------------------------------+--------------------------------------+
 220 | **Project**                          | Yardstick                            |
 221 |                                      |                                      |
 222 +--------------------------------------+--------------------------------------+
 223 | **Repo/tag**                         | yardstick/brahmaputra.1.0            |
 224 |                                      |                                      |
 225 +--------------------------------------+--------------------------------------+
 226 | **Yardstick Docker image tag**       | brahmaputra.1.0                      |
 227 |                                      |                                      |
 228 +--------------------------------------+--------------------------------------+
 229 | **Release designation**              | Brahmaputra                          |
 230 |                                      |                                      |
 231 +--------------------------------------+--------------------------------------+
 232 | **Release date**                     | Feb 25th, 2016                       |
 233 |                                      |                                      |
 234 +--------------------------------------+--------------------------------------+
 235 | **Purpose of the delivery**          | OPNFV Brahmaputra release            |
 236 |                                      |                                      |
 237 +--------------------------------------+--------------------------------------+
 238
 239 **Contexts**
 240
 241 +---------------------+-------------------------------------------------------+
 242 | **Context**         | **Description**                                       |
 243 |                     |                                                       |
 244 +---------------------+-------------------------------------------------------+
 245 | *Heat*              | Models orchestration using OpenStack Heat             |
 246 |                     |                                                       |
 247 +---------------------+-------------------------------------------------------+
 248 | *Node*              | Models Baremetal, Controller, Compute                 |
 249 |                     |                                                       |
 250 +---------------------+-------------------------------------------------------+
 251
 252
 253 **Runners**
 254
 255 +---------------------+-------------------------------------------------------+
 256 | **Runner**          | **Description**                                       |
 257 |                     |                                                       |
 258 +---------------------+-------------------------------------------------------+
 259 | *Arithmetic*        | Steps every run arithmetically according to specified |
 260 |                     | input value                                           |
 261 |                     |                                                       |
 262 +---------------------+-------------------------------------------------------+
 263 | *Duration*          | Runs for a specified period of time                   |
 264 |                     |                                                       |
 265 +---------------------+-------------------------------------------------------+
 266 | *Iteration*         | Runs for a specified number of iterations             |
 267 |                     |                                                       |
 268 +---------------------+-------------------------------------------------------+
 269 | *Sequence*          | Selects input value to a scenario from an input file  |
 270 |                     | and runs all entries sequentially                     |
 271 |                     |                                                       |
 272 +---------------------+-------------------------------------------------------+
 273
 274 **Scenarios**
 275
 276 +---------------------+-------------------------------------------------------+
 277 | **Category**        | **Delivered**                                         |
 278 |                     |                                                       |
 279 +---------------------+-------------------------------------------------------+
 280 | *Availability*      | Attacker:                                             |
 281 |                     |                                                       |
 282 |                     | * baremetal, process                                  |
 283 |                     |                                                       |
 284 |                     | HA tools:                                             |
 285 |                     |                                                       |
 286 |                     | * check host, openstack, process, service             |
 287 |                     | * kill process                                        |
 288 |                     | * start/stop service                                  |
 289 |                     |                                                       |
 290 |                     | Monitor:                                              |
 291 |                     |                                                       |
 292 |                     | * command, process                                    |
 293 |                     |                                                       |
 294 +---------------------+-------------------------------------------------------+
 295 | *Compute*           | * cpuload                                             |
 296 |                     |                                                       |
 297 |                     | * cyclictest                                          |
 298 |                     |                                                       |
 299 |                     | * lmbench                                             |
 300 |                     |                                                       |
 301 |                     | * perf                                                |
 302 |                     |                                                       |
 303 |                     | * unixbench                                           |
 304 |                     |                                                       |
 305 +---------------------+-------------------------------------------------------+
 306 | *Networking*        | * iperf3                                              |
 307 |                     |                                                       |
 308 |                     | * netperf                                             |
 309 |                     |                                                       |
 310 |                     | * ping                                                |
 311 |                     |                                                       |
 312 |                     | * ping6                                               |
 313 |                     |                                                       |
 314 |                     | * pktgen                                              |
 315 |                     |                                                       |
 316 |                     | * sfc                                                 |
 317 |                     |                                                       |
 318 |                     | * sfc with tacker                                     |
 319 |                     |                                                       |
 320 |                     | * vtc instantion validation                           |
 321 |                     |                                                       |
 322 |                     | * vtc instantion validation with noisy neighbors      |
 323 |                     |                                                       |
 324 |                     | * vtc throughput                                      |
 325 |                     |                                                       |
 326 |                     | * vtc throughput in the presence of noisy neighbors   |
 327 |                     |                                                       |
 328 +---------------------+-------------------------------------------------------+
 329 | *Parser*            | Tosca2Heat                                            |
 330 |                     |                                                       |
 331 +---------------------+-------------------------------------------------------+
 332 | *Storage*           | fio                                                   |
 333 |                     |                                                       |
 334 +---------------------+-------------------------------------------------------+
 335
 336 **API to Other Frameworks**
 337
 338 +---------------------+-------------------------------------------------------+
 339 | **Framework**       | **Description**                                       |
 340 |                     |                                                       |
 341 +---------------------+-------------------------------------------------------+
 342 | *ApexLake*          | Experimental framework that enables the user to       |
 343 |                     | validate NFVI from the perspective of a VNF.          |
 344 |                     | A virtual Traffic Classifier is utilized as VNF.      |
 345 |                     | Enables experiments with SR-IOV on Compute Node.      |
 346 |                     |                                                       |
 347 +---------------------+-------------------------------------------------------+
 348
 349 **Test Results Output**
 350
 351 +-----------------------------+-----------------------------------------------+
 352 | **Dispatcher**              | **Description**                               |
 353 |                             |                                               |
 354 +-----------------------------+-----------------------------------------------+
 355 |  file                       | Log to a file.                                |
 356 |                             |                                               |
 357 +-----------------------------+-----------------------------------------------+
 358 |  http                       | Post data to html.                            |
 359 |                             |                                               |
 360 +-----------------------------+-----------------------------------------------+
 361 |  influxdb                   | Post data to influxdB.                        |
 362 |                             |                                               |
 363 +-----------------------------+-----------------------------------------------+
 364
 365
 366 Delivered Test cases
 367 ~~~~~~~~~~~~~~~~~~~~
 368
 369 * Generic NFVI test cases
 370
 371   * OPNFV_YARDSTICK_TCOO1 - NW Performance
 372
 373   * OPNFV_YARDSTICK_TCOO2 - NW Latency
 374
 375   * OPNFV_YARDSTICK_TCOO5 - Storage Performance
 376
 377   * OPNFV_YARDSTICK_TCOO8 - Packet Loss Extended Test
 378
 379   * OPNFV_YARDSTICK_TCOO9 - Packet Loss
 380
 381   * OPNFV_YARDSTICK_TCO10 - Memory Latency
 382
 383   * OPNFV_YARDSTICK_TCO11 - Packet Delay Variation Between VMs
 384
 385   * OPNFV_YARDSTICK_TCO12 - Memory Bandwidth
 386
 387   * OPNFV_YARDSTICK_TCO14 - Processing Speed
 388
 389   * OPNFV_YARDSTICK_TCO24 - CPU Load
 390
 391   * OPNFV_YARDSTICK_TCO37 - Latency, CPU Load, Throughput, Packet Loss
 392
 393   * OPNFV_YARDSTICK_TCO38 - Latency, CPU Load, Throughput, Packet Loss Extended
 394     Test
 395
 396
 397 * Test Cases for OPNFV HA Project:
 398
 399   * OPNFV_YARDSTICK_TCO19 - HA: Control node Openstack service down
 400
 401   * OPNFV_YARDSTICK_TC025 - HA: OpenStacK Controller Node abnormally down
 402
 403 * Test Case for OPNFV IPv6 Project:
 404
 405   * OPNFV_YARDSTICK_TCO27 - IPv6 connectivity
 406
 407 * Test Case for OPNFV KVM Project:
 408
 409   * OPNFV_YARDSTICK_TCO28 - KVM Latency measurements
 410
 411 * Test Case for OPNFV Parser Project:
 412
 413   * OPNFV_YARDSTICK_TCO40 - Verify Parser Yang-to-Tosca