docs/testing/user/userguide/02-methodology.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International
   2    License.
   3    http://creativecommons.org/licenses/by/4.0
   4    (c) OPNFV, Ericsson AB and others.
   5
   6    Convention for heading levels in Yardstick documentation:
   7
   8    =======  Heading 0 (reserved for the title in a document)
   9    -------  Heading 1
  10    ^^^^^^^  Heading 2
  11    +++++++  Heading 3
  12    '''''''  Heading 4
  13
  14    Avoid deeper levels because they do not render well.
  15
  16 ===========
  17 Methodology
  18 ===========
  19
  20 Abstract
  21 --------
  22
  23 This chapter describes the methodology implemented by the Yardstick project for
  24 verifying the :term:`NFVI` from the perspective of a :term:`VNF`.
  25
  26 ETSI-NFV
  27 --------
  28
  29 .. _NFV-TST001: http://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/001/01.01.01_60/gs_NFV-TST001v010101p.pdf
  30 .. _Yardsticktst: https://wiki.opnfv.org/download/attachments/2925202/opnfv_summit_-_bridging_opnfv_and_etsi.pdf?version=1&modificationDate=1458848320000&api=v2
  31
  32
  33 The document ETSI GS NFV-TST001_, "Pre-deployment Testing; Report on Validation
  34 of NFV Environments and Services", recommends methods for pre-deployment
  35 testing of the functional components of an NFV environment.
  36
  37 The Yardstick project implements the methodology described in chapter 6, "Pre-
  38 deployment validation of NFV infrastructure".
  39
  40 The methodology consists in decomposing the typical :term:`VNF` work-load
  41 performance metrics into a number of characteristics/performance vectors, which
  42 each can be represented by distinct test-cases.
  43
  44 The methodology includes five steps:
  45
  46 * *Step1:* Define Infrastruture - the Hardware, Software  and corresponding
  47    configuration target for validation; the OPNFV infrastructure, in OPNFV
  48    community labs.
  49
  50 * *Step2:* Identify :term:`VNF` type - the application for which the
  51    infrastructure is to be validated, and its requirements on the underlying
  52    infrastructure.
  53
  54 * *Step3:* Select test cases - depending on the workload that represents the
  55    application for which the infrastruture is to be validated, the relevant
  56    test cases amongst the list of available Yardstick test cases.
  57
  58 * *Step4:* Execute tests - define the duration and number of iterations for the
  59    selected test cases, tests runs are automated via OPNFV Jenkins Jobs.
  60
  61 * *Step5:* Collect results - using the common API for result collection.
  62
  63 .. seealso:: Yardsticktst_ for material on alignment ETSI TST001 and Yardstick.
  64
  65 Metrics
  66 -------
  67
  68 The metrics, as defined by ETSI GS NFV-TST001, are shown in
  69 :ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and
  70 :ref:`Table3 <table2_3>`.
  71
  72 In OPNFV Colorado release, generic test cases covering aspects of the listed
  73 metrics are available; further OPNFV releases will provide extended testing of
  74 these metrics.
  75 The view of available Yardstick test cases cross ETSI definitions in
  76 :ref:`Table1 <table2_1>`, :ref:`Table2 <table2_2>` and :ref:`Table3 <table2_3>`
  77 is shown in :ref:`Table4 <table2_4>`.
  78 It shall be noticed that the Yardstick test cases are examples, the test
  79 duration and number of iterations are configurable, as are the System Under
  80 Test (SUT) and the attributes (or, in Yardstick nomemclature, the scenario
  81 options).
  82
  83 .. _table2_1:
  84
  85 **Table 1 - Performance/Speed Metrics**
  86
  87 +---------+-------------------------------------------------------------------+
  88 | Category| Performance/Speed                                                 |
  89 |         |                                                                   |
  90 +---------+-------------------------------------------------------------------+
  91 | Compute | * Latency for random memory access                                |
  92 |         | * Latency for cache read/write operations                         |
  93 |         | * Processing speed (instructions per second)                      |
  94 |         | * Throughput for random memory access (bytes per second)          |
  95 |         |                                                                   |
  96 +---------+-------------------------------------------------------------------+
  97 | Network | * Throughput per NFVI node (frames/byte per second)               |
  98 |         | * Throughput provided to a VM (frames/byte per second)            |
  99 |         | * Latency per traffic flow                                        |
 100 |         | * Latency between VMs                                             |
 101 |         | * Latency between NFVI nodes                                      |
 102 |         | * Packet delay variation (jitter) between VMs                     |
 103 |         | * Packet delay variation (jitter) between NFVI nodes              |
 104 |         |                                                                   |
 105 +---------+-------------------------------------------------------------------+
 106 | Storage | * Sequential read/write IOPS                                      |
 107 |         | * Random read/write IOPS                                          |
 108 |         | * Latency for storage read/write operations                       |
 109 |         | * Throughput for storage read/write operations                    |
 110 |         |                                                                   |
 111 +---------+-------------------------------------------------------------------+
 112
 113 .. _table2_2:
 114
 115 **Table 2 - Capacity/Scale Metrics**
 116
 117 +---------+-------------------------------------------------------------------+
 118 | Category| Capacity/Scale                                                    |
 119 |         |                                                                   |
 120 +---------+-------------------------------------------------------------------+
 121 | Compute | * Number of cores and threads- Available memory size              |
 122 |         | * Cache size                                                      |
 123 |         | * Processor utilization (max, average, standard deviation)        |
 124 |         | * Memory utilization (max, average, standard deviation)           |
 125 |         | * Cache utilization (max, average, standard deviation)            |
 126 |         |                                                                   |
 127 +---------+-------------------------------------------------------------------+
 128 | Network | * Number of connections                                           |
 129 |         | * Number of frames sent/received                                  |
 130 |         | * Maximum throughput between VMs (frames/byte per second)         |
 131 |         | * Maximum throughput between NFVI nodes (frames/byte per second)  |
 132 |         | * Network utilization (max, average, standard deviation)          |
 133 |         | * Number of traffic flows                                         |
 134 |         |                                                                   |
 135 +---------+-------------------------------------------------------------------+
 136 | Storage | * Storage/Disk size                                               |
 137 |         | * Capacity allocation (block-based, object-based)                 |
 138 |         | * Block size                                                      |
 139 |         | * Maximum sequential read/write IOPS                              |
 140 |         | * Maximum random read/write IOPS                                  |
 141 |         | * Disk utilization (max, average, standard deviation)             |
 142 |         |                                                                   |
 143 +---------+-------------------------------------------------------------------+
 144
 145 .. _table2_3:
 146
 147 **Table 3 - Availability/Reliability Metrics**
 148
 149 +---------+-------------------------------------------------------------------+
 150 | Category| Availability/Reliability                                          |
 151 |         |                                                                   |
 152 +---------+-------------------------------------------------------------------+
 153 | Compute | * Processor availability (Error free processing time)             |
 154 |         | * Memory availability (Error free memory time)                    |
 155 |         | * Processor mean-time-to-failure                                  |
 156 |         | * Memory mean-time-to-failure                                     |
 157 |         | * Number of processing faults per second                          |
 158 |         |                                                                   |
 159 +---------+-------------------------------------------------------------------+
 160 | Network | * NIC availability (Error free connection time)                   |
 161 |         | * Link availability (Error free transmission time)                |
 162 |         | * NIC mean-time-to-failure                                        |
 163 |         | * Network timeout duration due to link failure                    |
 164 |         | * Frame loss rate                                                 |
 165 |         |                                                                   |
 166 +---------+-------------------------------------------------------------------+
 167 | Storage | * Disk availability (Error free disk access time)                 |
 168 |         | * Disk mean-time-to-failure                                       |
 169 |         | * Number of failed storage read/write operations per second       |
 170 |         |                                                                   |
 171 +---------+-------------------------------------------------------------------+
 172
 173 .. _table2_4:
 174
 175 **Table 4 - Yardstick Generic Test Cases**
 176
 177 +---------+-------------------+----------------+------------------------------+
 178 | Category| Performance/Speed | Capacity/Scale | Availability/Reliability     |
 179 |         |                   |                |                              |
 180 +---------+-------------------+----------------+------------------------------+
 181 | Compute | TC003 [1]_        | TC003 [1]_     |  TC013 [1]_                  |
 182 |         | TC004             | TC004          |  TC015 [1]_                  |
 183 |         | TC010             | TC024          |                              |
 184 |         | TC012             | TC055          |                              |
 185 |         | TC014             |                |                              |
 186 |         | TC069             |                |                              |
 187 +---------+-------------------+----------------+------------------------------+
 188 | Network | TC001             | TC044          |  TC016 [1]_                  |
 189 |         | TC002             | TC073          |  TC018 [1]_                  |
 190 |         | TC009             | TC075          |                              |
 191 |         | TC011             |                |                              |
 192 |         | TC042             |                |                              |
 193 |         | TC043             |                |                              |
 194 +---------+-------------------+----------------+------------------------------+
 195 | Storage | TC005             | TC063          |  TC017 [1]_                  |
 196 +---------+-------------------+----------------+------------------------------+
 197
 198 .. note:: The description in this OPNFV document is intended as a reference for
 199   users to understand the scope of the Yardstick Project and the
 200   deliverables of the Yardstick framework. For complete description of
 201   the methodology, please refer to the ETSI document.
 202
 203 .. rubric:: Footnotes
 204 .. [1] To be included in future deliveries.
 205