1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
6 =======================================
7 Test Results for os-odl_l2-nofeature-ha
8 =======================================
17 .. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
18 .. _POD2: https://wiki.opnfv.org/pharos?&#community_test_labs
20 Overview of test results
21 ------------------------
23 See Grafana_ for viewing test result metrics for each respective test case. It
24 is possible to chose which specific scenarios to look at, and then to zoom in
25 on the details of each run test scenario as well.
27 All of the test case results below are based on 4 scenario test
28 runs, each run on the Ericsson POD2_ or LF POD2_ between August 25 and 29 in
33 The round-trip-time (RTT) between 2 VMs on different blades is measured using
34 ping. Most test run measurements result on average between 0.5 and 0.6 ms.
35 A few runs start with a 1 - 1.5 ms RTT spike (This could be because of normal ARP
36 handling). One test run has a greater RTT spike of 1.9 ms, which is the same
37 one with the 0.7 ms average. The other runs have no similar spike at all.
38 To be able to draw conclusions more runs should be made.
39 SLA set to 10 ms. The SLA value is used as a reference, it has not
40 been defined by OPNFV.
44 The IO read bandwidth looks similar between different dates, with an
45 average between approx. 170 and 200 MB/s. Within each test run the results
46 vary, with a minimum 2 MB/s and maximum 838 MB/s on the totality. Most runs
47 have a minimum BW of 3 MB/s (two runs at 2 MB/s). The maximum BW varies more in
48 absolute numbers between the dates, between 617 and 838 MB/s.
49 SLA set to 400 MB/s. The SLA value is used as a reference, it has not been
54 The measurements for memory latency are similar between test dates and result
55 in approx. 1.2 ns. The variations within each test run are similar, between
56 1.215 and 1.219 ns. One exception is February 16, where the average is 1.222
57 and varies between 1.22 and 1.28 ns.
58 SLA set to 30 ns. The SLA value is used as a reference, it has not been defined
63 Packet delay variation between 2 VMs on different blades is measured using
64 Iperf3. On the first date the reported packet delay variation varies between
65 0.0025 and 0.011 ms, with an average delay variation of 0.0067 ms.
66 On the second date the delay variation varies between 0.002 and 0.006 ms, with
67 an average delay variation of 0.004 ms.
71 Between test dates, the average measurements for memory bandwidth vary between
72 17.4 and 17.9 GB/s. Within each test run the results vary more, with a minimal
73 BW of 16.4 GB/s and maximum of 18.2 GB/s on the totality.
74 SLA set to 15 GB/s. The SLA value is used as a reference, it has not been
79 The Unixbench processor test run results vary between scores 3080 and 3240,
80 one result each date. The average score on the total is 3150.
85 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
86 on different blades are measured when increasing the amount of UDP flows sent
87 between the VMs using pktgen as packet generator tool.
89 Round trip times and packet throughput between VMs can typically be affected by
90 the amount of flows set up and result in higher RTT and less PPS throughput.
92 The RTT results are similar throughout the different test dates and runs at
93 approx. 15 ms. Some test runs show an increase with many flows, in the range
94 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
95 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
97 In some test runs when running with less than approx. 10000 flows the PPS
98 throughput is normally flatter compared to when running with more flows, after
99 which the PPS throughput decreases. Around 20 percent decrease in the worst
100 case. For the other test runs there is however no significant change to the PPS
101 throughput when the number of flows are increased. In some test runs the PPS
102 is also greater with 1000000 flows compared to other test runs where the PPS
103 result is less with only 2 flows.
105 The average PPS throughput in the different runs varies between 414000 and
106 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
107 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
108 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
111 There are lost packets reported in most of the test runs. There is no observed
112 correlation between the amount of flows and the amount of lost packets.
113 The lost amount of packets normally range between 100 and 1000 per test run,
114 but there are spikes in the range of 10000 lost packets as well, and even
115 more in a rare cases.
117 CPU utilization statistics are collected during UDP flows sent between the VMs
118 using pktgen as packet generator tool. The average measurements for CPU
119 utilization ratio vary between 1% to 2%. The peak of CPU utilization ratio
124 Between test dates, the average measurements for memory bandwidth vary between
125 15.5 and 25.4 GB/s. Within each test run the results vary more, with a minimal
126 BW of 9.7 GB/s and maximum of 29.5 GB/s on the totality.
127 SLA set to 6 GB/s. The SLA value is used as a reference, it has not been
132 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
133 on different blades are measured when increasing the amount of UDP flows sent
134 between the VMs using pktgen as packet generator tool.
136 Round trip times and packet throughput between VMs can typically be affected by
137 the amount of flows set up and result in higher RTT and less PPS throughput.
139 The RTT results are similar throughout the different test dates and runs at
140 approx. 15 ms. Some test runs show an increase with many flows, in the range
141 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
142 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
144 In some test runs when running with less than approx. 10000 flows the PPS
145 throughput is normally flatter compared to when running with more flows, after
146 which the PPS throughput decreases. Around 20 percent decrease in the worst
147 case. For the other test runs there is however no significant change to the PPS
148 throughput when the number of flows are increased. In some test runs the PPS
149 is also greater with 1000000 flows compared to other test runs where the PPS
150 result is less with only 2 flows.
152 The average PPS throughput in the different runs varies between 414000 and
153 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
154 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
155 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
158 There are lost packets reported in most of the test runs. There is no observed
159 correlation between the amount of flows and the amount of lost packets.
160 The lost amount of packets normally range between 100 and 1000 per test run,
161 but there are spikes in the range of 10000 lost packets as well, and even
162 more in a rare cases.
164 Memory utilization statistics are collected during UDP flows sent between the
165 VMs using pktgen as packet generator tool. The average measurements for memory
166 utilization vary between 225MB to 246MB. The peak of memory utilization appears
171 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
172 on different blades are measured when increasing the amount of UDP flows sent
173 between the VMs using pktgen as packet generator tool.
175 Round trip times and packet throughput between VMs can typically be affected by
176 the amount of flows set up and result in higher RTT and less PPS throughput.
178 The RTT results are similar throughout the different test dates and runs at
179 approx. 15 ms. Some test runs show an increase with many flows, in the range
180 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
181 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
183 In some test runs when running with less than approx. 10000 flows the PPS
184 throughput is normally flatter compared to when running with more flows, after
185 which the PPS throughput decreases. Around 20 percent decrease in the worst
186 case. For the other test runs there is however no significant change to the PPS
187 throughput when the number of flows are increased. In some test runs the PPS
188 is also greater with 1000000 flows compared to other test runs where the PPS
189 result is less with only 2 flows.
191 The average PPS throughput in the different runs varies between 414000 and
192 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
193 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
194 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
197 There are lost packets reported in most of the test runs. There is no observed
198 correlation between the amount of flows and the amount of lost packets.
199 The lost amount of packets normally range between 100 and 1000 per test run,
200 but there are spikes in the range of 10000 lost packets as well, and even
201 more in a rare cases.
203 Cache utilization statistics are collected during UDP flows sent between the
204 VMs using pktgen as packet generator tool. The average measurements for cache
205 utilization vary between 205MB to 212MB.
209 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
210 on different blades are measured when increasing the amount of UDP flows sent
211 between the VMs using pktgen as packet generator tool.
213 Round trip times and packet throughput between VMs can typically be affected by
214 the amount of flows set up and result in higher RTT and less PPS throughput.
216 The RTT results are similar throughout the different test dates and runs at
217 approx. 15 ms. Some test runs show an increase with many flows, in the range
218 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
219 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
221 In some test runs when running with less than approx. 10000 flows the PPS
222 throughput is normally flatter compared to when running with more flows, after
223 which the PPS throughput decreases. Around 20 percent decrease in the worst
224 case. For the other test runs there is however no significant change to the PPS
225 throughput when the number of flows are increased. In some test runs the PPS
226 is also greater with 1000000 flows compared to other test runs where the PPS
227 result is less with only 2 flows.
229 The average PPS throughput in the different runs varies between 414000 and
230 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
231 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
232 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
235 There are lost packets reported in most of the test runs. There is no observed
236 correlation between the amount of flows and the amount of lost packets.
237 The lost amount of packets normally range between 100 and 1000 per test run,
238 but there are spikes in the range of 10000 lost packets as well, and even
239 more in a rare cases.
241 Network utilization statistics are collected during UDP flows sent between the
242 VMs using pktgen as packet generator tool. Total number of packets received per
243 second was average on 200 kpps and total number of packets transmitted per
244 second was average on 600 kpps.
246 Detailed test results
247 ---------------------
248 The scenario was run on Ericsson POD2_ and LF POD2_ with:
251 OpenVirtualSwitch 2.5.90
252 OpenDayLight Beryllium
254 Rationale for decisions
255 -----------------------
258 Tests were successfully executed and metrics collected.
259 No SLA was verified. To be decided on in next release of OPNFV.
261 Conclusions and recommendations
262 -------------------------------
263 The pktgen test configuration has a relatively large base effect on RTT in
264 TC037 compared to TC002, where there is no background load at all. Approx.
265 15 ms compared to approx. 0.5 ms, which is more than a 3000 percentage
266 difference in RTT results.
267 Especially RTT and throughput come out with better results than for instance
268 the *fuel-os-nosdn-nofeature-ha* scenario does. The reason for this should
269 probably be further analyzed and understood. Also of interest could be
270 to make further analyzes to find patterns and reasons for lost traffic.
271 Also of interest could be to see if there are continuous variations where
272 some test cases stand out with better or worse results than the general test
280 .. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
281 .. _POD6: https://wiki.opnfv.org/pharos?&#community_test_labs
283 Overview of test results
284 ------------------------
286 See Grafana_ for viewing test result metrics for each respective test case. It
287 is possible to chose which specific scenarios to look at, and then to zoom in
288 on the details of each run test scenario as well.
290 All of the test case results below are based on 4 scenario test runs, each run
291 on the Intel POD6_ between September 1 and 8 in 2016.
295 The round-trip-time (RTT) between 2 VMs on different blades is measured using
296 ping. Most test run measurements result on average between 1.01 ms and 1.88 ms.
297 Only one test run has reached greatest RTT spike of 1.88 ms. Meanwhile, the
298 smallest network latency is 1.01 ms, which is obtained on Sep. 1st. In general,
299 the average of network latency of the four test runs are between 1.29 ms and
300 1.34 ms. SLA set to be 10 ms. The SLA value is used as a reference, it has not
301 been defined by OPNFV.
305 The IO read bandwidth actually refers to the storage throughput, which is
306 measured by fio and the greatest IO read bandwidth of the four runs is 183.65
307 MB/s. The IO read bandwidth of the three runs looks similar, with an average
308 between 62.9 and 64.3 MB/s, except one on Sep. 1, for its maximum storage
309 throughput is only 159.1 MB/s. One of the runs has a minimum BW of 685 KM/s and
310 other has a maximum BW of 183.6 MB/s. The SLA of read bandwidth sets to be
311 400 MB/s, which is used as a reference, and it has not been defined by OPNFV.
313 The results of storage IOPS for the four runs look similar with each other. The
314 IO read times per second of the four test runs have an average value between
315 1.41k per second and 1.64k per second, and meanwhile, the minimum result is
316 only 55 times per second.
320 The tool we use to measure memory read latency is lmbench, which is a series of
321 micro benchmarks intended to measure basic operating system and hardware system
322 metrics. The memory read latency of the four runs is between 1.152 ns and 1.179
323 ns on average. The variations within each test run are quite different, some
324 vary from a large range and others have a small change. For example, the
325 largest change is on September 8, the memory read latency of which is ranging
326 from 1.120 ns to 1.221 ns. However, the results on September 7 change very
327 little. The SLA sets to be 30 ns. The SLA value is used as a reference, it has
328 not been defined by OPNFV.
332 Iperf3 is a tool for evaluating the packet delay variation between 2 VMs on
333 different blades. The reported packet delay variations of the four test runs
334 differ from each other. In general, the packet delay of the first two runs look
335 similar, for they both stay stable within each run. And the mean packet delay
336 of them are 0.0087 ms and 0.0127 ms respectively. Of the four runs, the fourth
337 has the worst result, because the packet delay reaches 0.0187 ms. The SLA value
338 sets to be 10 ms. The SLA value is used as a reference, it has not been defined
343 Lmbench is also used to measure the memory read and write bandwidth, in which
344 we use bw_mem to obtain the results. Among the four test runs, the trend of
345 three memory bandwidth almost look similar, which all have a narrow range, and
346 the average result is 11.78 GB/s. Here SLA set to be 15 GB/s. The SLA value is
347 used as a reference, it has not been defined by OPNFV.
351 The Unixbench is used to evaluate the IaaS processing speed with regards to
352 score of single cpu running and parallel running. It can be seen from the
353 dashboard that the processing test results vary from scores 3260k to 3328k, and
354 there is only one result one date. No SLA set.
358 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
359 on different blades are measured when increasing the amount of UDP flows sent
360 between the VMs using pktgen as packet generator tool.
362 Round trip times and packet throughput between VMs can typically be affected by
363 the amount of flows set up and result in higher RTT and less PPS throughput.
365 The mean packet throughput of the four test runs is between 307.3 kpps and
366 447.1 kpps, of which the result of the third run is the highest. The RTT
367 results of all the test runs keep flat at approx. 15 ms. It is obvious that the
368 PPS results are not as consistent as the RTT results.
370 The No. flows of the four test runs are 240 k on average and the PPS results
371 look a little waved since the largest packet throughput is 418.1 kpps and the
372 minimum throughput is 326.5 kpps respectively.
374 There are no errors of packets received in the four runs, but there are still
375 lost packets in all the test runs. The RTT values obtained by ping of the four
376 runs have the similar average vaue, that is approx. 15 ms.
378 CPU load is measured by mpstat, and CPU load of the four test runs seem a
379 little similar, since the minimum value and the peak of CPU load is between 0
380 percent and nine percent respectively. And the best result is obtained on Sep.
381 1, with an CPU load of nine percent. But on the whole, the CPU load is very
382 poor, since the average value is quite small.
386 With the block size changing from 1 kb to 512 kb, the memory write bandwidth
387 tends to become larger first and then smaller within every run test, which
388 rangs from 21.9 GB/s to 25.9 GB/s and then to 17.8 GB/s on average. Since the
389 test id is one, it is that only the INT memory write bandwidth is tested. On
390 the whole, when the block size is 2 kb or 16 kb, the memory write bandwidth
391 look similar with a minimal BW of 24.8 GB/s and peak value of 27.8 GB/s. And
392 then with the block size becoming larger, the memory write bandwidth tends to
393 decrease. SLA sets to be 7 GB/s. The SLA value is used as a reference, it has
394 not been defined by OPNFV.
398 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
399 on different blades are measured when increasing the amount of UDP flows sent
400 between the VMs using pktgen as packet generator tool.
402 Round trip times and packet throughput between VMs can typically be affected by
403 the amount of flows set up and result in higher RTT and less PPS throughput.
405 The network latency is measured by ping, and the results of the four test runs
406 look similar with each other, and within these test runs, the maximum RTT can
407 reach 39 ms and the average RTT is usually approx. 15 ms. The network latency
408 tested on Sep. 1 and Sep. 8 have a peak latency of 39 ms. But on the whole,
409 the average RTTs of the five runs keep flat and the network latency is
412 Memory utilization is measured by free, which can display amount of free and
413 used memory in the system. The largest amount of used memory is 267 MiB for the
414 four runs. In general, the four test runs have very large memory utilization,
415 which can reach 257 MiB on average. On the other hand, for the mean free memory,
416 the four test runs have the similar trend with that of the mean used memory.
417 In general, the mean free memory change from 233 MiB to 241 MiB.
419 Packet throughput and packet loss can be measured by pktgen, which is a tool
420 in the network for generating traffic loads for network experiments. The mean
421 packet throughput of the four test runs seem quite different, ranging from
422 305.3 kpps to 447.1 kpps. The average number of flows in these tests is
423 240000, and each run has a minimum number of flows of 2 and a maximum number
424 of flows of 1.001 Mil. At the same time, the corresponding average packet
425 throughput is between 354.4 kpps and 381.8 kpps. In summary, the PPS results
426 seem consistent. Within each test run of the four runs, when number of flows
427 becomes larger, the packet throughput seems not larger at the same time.
431 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
432 on different blades are measured when increasing the amount of UDP flows sent
433 between the VMs using pktgen as packet generator tool.
435 Round trip times and packet throughput between VMs can typically be affected by
436 the amount of flows set up and result in higher RTT and less PPS throughput.
438 The network latency is measured by ping, and the results of the four test runs
439 look similar with each other. Within each test run, the maximum RTT is only 42
440 ms and the average RTT is usually approx. 15 ms. On the whole, the average
441 RTTs of the four runs keep stable and the network latency is relatively small.
443 Cache utilization is measured by cachestat, which can display size of cache and
444 buffer in the system. Cache utilization statistics are collected during UDP
445 flows sent between the VMs using pktgen as packet generator tool. The largest
446 cache size is 212 MiB, which is same for the four runs, and the smallest cache
447 size is 75 MiB. On the whole, the average cache size of the four runs look the
448 same and is between 197 MiB and 211 MiB. Meanwhile, the tread of the buffer
449 size keep flat, since they have a minimum value of 7 MiB and a maximum value of
450 8 MiB, with an average value of about 7.9 MiB.
452 Packet throughput can be measured by pktgen, which is a tool in the network for
453 generating traffic loads for network experiments. The mean packet throughput of
454 the four test runs differ from 354.4 kpps to 381.8 kpps. The average number of
455 flows in these tests is 240k, and each run has a minimum number of flows of 2
456 and a maximum number of flows of 1.001 Mil. At the same time, the corresponding
457 packet throughput differ between 305.3 kpps to 447.1 kpps. Within each test run
458 of the four runs, when number of flows becomes larger, the packet throughput
459 seems not larger in the meantime.
463 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
464 on different blades are measured when increasing the amount of UDP flows sent
465 between the VMs using pktgen as packet generator tool.
467 Round trip times and packet throughput between VMs can typically be affected by
468 the amount of flows set up and result in higher RTT and less PPS throughput.
470 The RTT results are similar throughout the different test dates and runs
471 between 0 ms and 42 ms with an average leatency of less than 15 ms. The PPS
472 results are not as consistent as the RTT results, for the mean packet
473 throughput of the four runs differ from 354.4 kpps to 381.8 kpps.
475 Network utilization is measured by sar, that is system activity reporter, which
476 can display the average statistics for the time since the system was started.
477 Network utilization statistics are collected during UDP flows sent between the
478 VMs using pktgen as packet generator tool. The largest total number of packets
479 transmitted per second look similar for three test runs, whose values change a
480 lot from 10 pps to 501 kpps. While results of the rest test run seem the same
481 and keep stable with the average number of packets transmitted per second of 10
482 pps. However, the total number of packets received per second of the four runs
483 look similar, which have a large wide range of 2 pps to 815 kpps.
485 In some test runs when running with less than approx. 251000 flows the PPS
486 throughput is normally flatter compared to when running with more flows, after
487 which the PPS throughput decreases. For the other test runs there is however no
488 significant change to the PPS throughput when the number of flows are
489 increased. In some test runs the PPS is also greater with 251000 flows
490 compared to other test runs where the PPS result is less with only 2 flows.
492 There are lost packets reported in most of the test runs. There is no observed
493 correlation between the amount of flows and the amount of lost packets.
494 The lost amount of packets normally differs a lot per test run.
496 Detailed test results
497 ---------------------
498 The scenario was run on Intel POD6_ with:
501 OpenVirtualSwitch 2.5.90
502 OpenDayLight Beryllium
504 Rationale for decisions
505 -----------------------
508 Conclusions and recommendations
509 -------------------------------
510 Tests were successfully executed and metrics collected.
511 No SLA was verified. To be decided on in next release of OPNFV.
Code Review / yardstick.git / blob