1 .. This work is licensed under a Creative Commons Attribution 4.0 International
3 .. http://creativecommons.org/licenses/by/4.0
6 ===============================
7 Test Results for os-onos-sfc-ha
8 ===============================
17 .. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
18 .. _POD2: https://wiki.opnfv.org/pharos?&#community_test_labs
20 Overview of test results
21 ------------------------
23 See Grafana_ for viewing test result metrics for each respective test case. It
24 is possible to chose which specific scenarios to look at, and then to zoom in
25 on the details of each run test scenario as well.
27 All of the test case results below are based on 4 scenario test runs, each run
28 on the Ericsson POD2_ or LF POD2_ between September 5 and 10 in 2016.
32 The round-trip-time (RTT) between 2 VMs on different blades is measured using
33 ping. Most test run measurements result on average between 0.5 and 0.6 ms.
34 A few runs start with a 1 - 1.5 ms RTT spike (This could be because of normal ARP
35 handling). One test run has a greater RTT spike of 1.9 ms, which is the same
36 one with the 0.7 ms average. The other runs have no similar spike at all.
37 To be able to draw conclusions more runs should be made.
38 SLA set to 10 ms. The SLA value is used as a reference, it has not
39 been defined by OPNFV.
43 The IO read bandwidth looks similar between different dates, with an
44 average between approx. 170 and 200 MB/s. Within each test run the results
45 vary, with a minimum 2 MB/s and maximum 838 MB/s on the totality. Most runs
46 have a minimum BW of 3 MB/s (two runs at 2 MB/s). The maximum BW varies more in
47 absolute numbers between the dates, between 617 and 838 MB/s.
48 SLA set to 400 MB/s. The SLA value is used as a reference, it has not been
53 The measurements for memory latency are similar between test dates and result
54 in approx. 1.2 ns. The variations within each test run are similar, between
55 1.215 and 1.219 ns. One exception is February 16, where the average is 1.222
56 and varies between 1.22 and 1.28 ns.
57 SLA set to 30 ns. The SLA value is used as a reference, it has not been defined
62 Packet delay variation between 2 VMs on different blades is measured using
63 Iperf3. On the first date the reported packet delay variation varies between
64 0.0025 and 0.011 ms, with an average delay variation of 0.0067 ms.
65 On the second date the delay variation varies between 0.002 and 0.006 ms, with
66 an average delay variation of 0.004 ms.
70 Between test dates, the average measurements for memory bandwidth vary between
71 17.4 and 17.9 GB/s. Within each test run the results vary more, with a minimal
72 BW of 16.4 GB/s and maximum of 18.2 GB/s on the totality.
73 SLA set to 15 GB/s. The SLA value is used as a reference, it has not been
78 The Unixbench processor test run results vary between scores 3080 and 3240,
79 one result each date. The average score on the total is 3150.
84 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
85 on different blades are measured when increasing the amount of UDP flows sent
86 between the VMs using pktgen as packet generator tool.
88 Round trip times and packet throughput between VMs can typically be affected by
89 the amount of flows set up and result in higher RTT and less PPS throughput.
91 The RTT results are similar throughout the different test dates and runs at
92 approx. 15 ms. Some test runs show an increase with many flows, in the range
93 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
94 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
96 In some test runs when running with less than approx. 10000 flows the PPS
97 throughput is normally flatter compared to when running with more flows, after
98 which the PPS throughput decreases. Around 20 percent decrease in the worst
99 case. For the other test runs there is however no significant change to the PPS
100 throughput when the number of flows are increased. In some test runs the PPS
101 is also greater with 1000000 flows compared to other test runs where the PPS
102 result is less with only 2 flows.
104 The average PPS throughput in the different runs varies between 414000 and
105 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
106 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
107 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
110 There are lost packets reported in most of the test runs. There is no observed
111 correlation between the amount of flows and the amount of lost packets.
112 The lost amount of packets normally range between 100 and 1000 per test run,
113 but there are spikes in the range of 10000 lost packets as well, and even
114 more in a rare cases.
116 CPU utilization statistics are collected during UDP flows sent between the VMs
117 using pktgen as packet generator tool. The average measurements for CPU
118 utilization ratio vary between 1% to 2%. The peak of CPU utilization ratio
123 Between test dates, the average measurements for memory bandwidth vary between
124 15.5 and 25.4 GB/s. Within each test run the results vary more, with a minimal
125 BW of 9.7 GB/s and maximum of 29.5 GB/s on the totality.
126 SLA set to 6 GB/s. The SLA value is used as a reference, it has not been
131 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
132 on different blades are measured when increasing the amount of UDP flows sent
133 between the VMs using pktgen as packet generator tool.
135 Round trip times and packet throughput between VMs can typically be affected by
136 the amount of flows set up and result in higher RTT and less PPS throughput.
138 The RTT results are similar throughout the different test dates and runs at
139 approx. 15 ms. Some test runs show an increase with many flows, in the range
140 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
141 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
143 In some test runs when running with less than approx. 10000 flows the PPS
144 throughput is normally flatter compared to when running with more flows, after
145 which the PPS throughput decreases. Around 20 percent decrease in the worst
146 case. For the other test runs there is however no significant change to the PPS
147 throughput when the number of flows are increased. In some test runs the PPS
148 is also greater with 1000000 flows compared to other test runs where the PPS
149 result is less with only 2 flows.
151 The average PPS throughput in the different runs varies between 414000 and
152 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
153 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
154 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
157 There are lost packets reported in most of the test runs. There is no observed
158 correlation between the amount of flows and the amount of lost packets.
159 The lost amount of packets normally range between 100 and 1000 per test run,
160 but there are spikes in the range of 10000 lost packets as well, and even
161 more in a rare cases.
163 Memory utilization statistics are collected during UDP flows sent between the
164 VMs using pktgen as packet generator tool. The average measurements for memory
165 utilization vary between 225MB to 246MB. The peak of memory utilization appears
170 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
171 on different blades are measured when increasing the amount of UDP flows sent
172 between the VMs using pktgen as packet generator tool.
174 Round trip times and packet throughput between VMs can typically be affected by
175 the amount of flows set up and result in higher RTT and less PPS throughput.
177 The RTT results are similar throughout the different test dates and runs at
178 approx. 15 ms. Some test runs show an increase with many flows, in the range
179 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
180 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
182 In some test runs when running with less than approx. 10000 flows the PPS
183 throughput is normally flatter compared to when running with more flows, after
184 which the PPS throughput decreases. Around 20 percent decrease in the worst
185 case. For the other test runs there is however no significant change to the PPS
186 throughput when the number of flows are increased. In some test runs the PPS
187 is also greater with 1000000 flows compared to other test runs where the PPS
188 result is less with only 2 flows.
190 The average PPS throughput in the different runs varies between 414000 and
191 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
192 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
193 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
196 There are lost packets reported in most of the test runs. There is no observed
197 correlation between the amount of flows and the amount of lost packets.
198 The lost amount of packets normally range between 100 and 1000 per test run,
199 but there are spikes in the range of 10000 lost packets as well, and even
200 more in a rare cases.
202 Cache utilization statistics are collected during UDP flows sent between the
203 VMs using pktgen as packet generator tool. The average measurements for cache
204 utilization vary between 205MB to 212MB.
208 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
209 on different blades are measured when increasing the amount of UDP flows sent
210 between the VMs using pktgen as packet generator tool.
212 Round trip times and packet throughput between VMs can typically be affected by
213 the amount of flows set up and result in higher RTT and less PPS throughput.
215 The RTT results are similar throughout the different test dates and runs at
216 approx. 15 ms. Some test runs show an increase with many flows, in the range
217 towards 16 to 17 ms. One exception standing out is Feb. 15 where the average
218 RTT is stable at approx. 13 ms. The PPS results are not as consistent as the
220 In some test runs when running with less than approx. 10000 flows the PPS
221 throughput is normally flatter compared to when running with more flows, after
222 which the PPS throughput decreases. Around 20 percent decrease in the worst
223 case. For the other test runs there is however no significant change to the PPS
224 throughput when the number of flows are increased. In some test runs the PPS
225 is also greater with 1000000 flows compared to other test runs where the PPS
226 result is less with only 2 flows.
228 The average PPS throughput in the different runs varies between 414000 and
229 452000 PPS. The total amount of packets in each test run is approx. 7500000 to
230 8200000 packets. One test run Feb. 15 sticks out with a PPS average of
231 558000 and approx. 1100000 packets in total (same as the on mentioned earlier
234 There are lost packets reported in most of the test runs. There is no observed
235 correlation between the amount of flows and the amount of lost packets.
236 The lost amount of packets normally range between 100 and 1000 per test run,
237 but there are spikes in the range of 10000 lost packets as well, and even
238 more in a rare cases.
240 Network utilization statistics are collected during UDP flows sent between the
241 VMs using pktgen as packet generator tool. Total number of packets received per
242 second was average on 200 kpps and total number of packets transmitted per
243 second was average on 600 kpps.
245 Detailed test results
246 ---------------------
247 The scenario was run on Ericsson POD2_ and LF POD2_ with:
251 OpenVirtualSwitch 2.5.90
252 OpenDayLight Beryllium
254 Rationale for decisions
255 -----------------------
258 Tests were successfully executed and metrics collected.
259 No SLA was verified. To be decided on in next release of OPNFV.
261 Conclusions and recommendations
262 -------------------------------
263 The pktgen test configuration has a relatively large base effect on RTT in
264 TC037 compared to TC002, where there is no background load at all. Approx.
265 15 ms compared to approx. 0.5 ms, which is more than a 3000 percentage
266 difference in RTT results.
267 Especially RTT and throughput come out with better results than for instance
268 the *fuel-os-nosdn-nofeature-ha* scenario does. The reason for this should
269 probably be further analyzed and understood. Also of interest could be
270 to make further analyzes to find patterns and reasons for lost traffic.
271 Also of interest could be to see if there are continuous variations where
272 some test cases stand out with better or worse results than the general test
279 .. _Grafana: http://testresults.opnfv.org/grafana/dashboard/db/yardstick-main
280 .. _POD6: https://wiki.opnfv.org/pharos?&#community_test_labs
282 Overview of test results
283 ------------------------
285 See Grafana_ for viewing test result metrics for each respective test case. It
286 is possible to chose which specific scenarios to look at, and then to zoom in
287 on the details of each run test scenario as well.
289 All of the test case results below are based on 4 scenario test runs, each run
290 on the Intel POD6_ between September 8 and 11 in 2016.
294 The round-trip-time (RTT) between 2 VMs on different blades is measured using
295 ping. Most test run measurements result on average between 1.35 ms and 1.57 ms.
296 Only one test run has reached greatest RTT spike of 2.58 ms. Meanwhile, the
297 smallest network latency is 1.11 ms, which is obtained on Sep. 11st. In
298 general, the average of network latency of the four test runs are between 1.35
299 ms and 1.57 ms. SLA set to be 10 ms. The SLA value is used as a reference, it
300 has not been defined by OPNFV.
304 The IO read bandwidth actually refers to the storage throughput, which is
305 measured by fio and the greatest IO read bandwidth of the four runs is 175.4
306 MB/s. The IO read bandwidth of the three runs looks similar, with an average
307 between 43.7 and 56.3 MB/s, except one on Sep. 8, for its maximum storage
308 throughput is only 107.9 MB/s. One of the runs has a minimum BW of 478 KM/s and
309 other has a maximum BW of 168.6 MB/s. The SLA of read bandwidth sets to be
310 400 MB/s, which is used as a reference, and it has not been defined by OPNFV.
312 The results of storage IOPS for the four runs look similar with each other. The
313 IO read times per second of the four test runs have an average value between
314 978 per second and 1.20 K/s, and meanwhile, the minimum result is only 36 times
319 The tool we use to measure memory read latency is lmbench, which is a series of
320 micro benchmarks intended to measure basic operating system and hardware system
321 metrics. The memory read latency of the four runs is between 1.164 ns and 1.244
322 ns on average. The variations within each test run are quite different, some
323 vary from a large range and others have a small change. For example, the
324 largest change is on September 10, the memory read latency of which is ranging
325 from 1.128 ns to 1.381 ns. However, the results on September 11 change very
326 little. The SLA sets to be 30 ns. The SLA value is used as a reference, it has
327 not been defined by OPNFV.
331 Iperf3 is a tool for evaluating the packet delay variation between 2 VMs on
332 different blades. The reported packet delay variations of the four test runs
333 differ from each other. In general, the packet delay of two runs look similar,
334 for they both stay stable within each run. And the mean packet delay of them
335 are 0.0772 ms and 0.0788 ms respectively. Of the four runs, the fourth has the
336 worst result, because the packet delay reaches 0.0838 ms. The rest one has a
337 large wide range from 0.0666 ms to 0.0798 ms. The SLA value sets to be 10 ms.
338 The SLA value is used as a reference, it has not been defined by OPNFV.
342 Lmbench is also used to measure the memory read and write bandwidth, in which
343 we use bw_mem to obtain the results. Among the four test runs, the trend of the
344 memory bandwidth almost look similar, which all have a large wide range, and
345 the minimum and maximum results are 9.02 GB/s and 18.14 GB/s. Here SLA set to
346 be 15 GB/s. The SLA value is used as a reference, it has not been defined by
351 The Unixbench is used to evaluate the IaaS processing speed with regards to
352 score of single cpu running and parallel running. It can be seen from the
353 dashboard that the processing test results vary from scores 3395 to 3475, and
354 there is only one result one date. No SLA set.
358 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
359 on different blades are measured when increasing the amount of UDP flows sent
360 between the VMs using pktgen as packet generator tool.
362 Round trip times and packet throughput between VMs can typically be affected by
363 the amount of flows set up and result in higher RTT and less PPS throughput.
365 The mean packet throughput of the four test runs is between 362.1 kpps and
366 363.5 kpps, of which the result of the third run is the highest. The RTT
367 results of all the test runs keep flat at approx. 17 ms. It is obvious that the
368 PPS results are not as consistent as the RTT results.
370 The No. flows of the four test runs are 240 k on average and the PPS results
371 look a little waved since the largest packet throughput is 418.1 kpps and the
372 minimum throughput is 326.5 kpps respectively.
374 There are no errors of packets received in the four runs, but there are still
375 lost packets in all the test runs. The RTT values obtained by ping of the four
376 runs have the similar average vaue, that is approx. 17 ms, of which the worst
377 RTT is 39 ms on Sep. 11st.
379 CPU load is measured by mpstat, and CPU load of the four test runs seem a
380 little similar, since the minimum value and the peak of CPU load is between 0
381 percent and nine percent respectively. And the best result is obtained on Sep.
382 10, with an CPU load of nine percent.
386 With the block size changing from 1 kb to 512 kb, the memory write bandwidth
387 tends to become larger first and then smaller within every run test, which
388 rangs from 25.9 GB/s to 26.6 GB/s and then to 18.1 GB/s on average. Since the
389 test id is one, it is that only the INT memory write bandwidth is tested. On
390 the whole, when the block size is from 2 kb to 16 kb, the memory write
391 bandwidth look similar with a minimal BW of 22.1 GB/s and peak value of 28.6
392 GB/s. And then with the block size becoming larger, the memory write bandwidth
393 tends to decrease. SLA sets to be 7 GB/s. The SLA value is used as a reference,
394 it has not been defined by OPNFV.
398 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
399 on different blades are measured when increasing the amount of UDP flows sent
400 between the VMs using pktgen as packet generator tool.
402 Round trip times and packet throughput between VMs can typically be affected by
403 the amount of flows set up and result in higher RTT and less PPS throughput.
405 The network latency is measured by ping, and the results of the four test runs
406 look similar with each other, and within these test runs, the maximum RTT can
407 reach 39 ms and the average RTT is usually approx. 17 ms. The network latency
408 tested on Sep. 11 shows that it has a peak latency of 39 ms. But on the whole,
409 the average RTTs of the five runs keep flat and the network latency is
412 Memory utilization is measured by free, which can display amount of free and
413 used memory in the system. The largest amount of used memory is 270 MiB on the
414 first two runs. In general, the mean used memory of two test runs have very
415 large memory utilization, which can reach 264 MiB on average. And the other two
416 runs have a large wide range of memory usage with the minimum value of 150 MiB
417 and the maximum value of 270 MiB. On the other hand, for the mean free memory,
418 the four test runs have the similar trend with that of the mean used memory.
419 In general, the mean free memory change from 220 MiB to 342 MiB.
421 Packet throughput and packet loss can be measured by pktgen, which is a tool
422 in the network for generating traffic loads for network experiments. The mean
423 packet throughput of the four test runs seem quite different, ranging from
424 326.5 kpps to 418.1 kpps. The average number of flows in these tests is
425 240000, and each run has a minimum number of flows of 2 and a maximum number
426 of flows of 1.001 Mil. At the same time, the corresponding packet throughput
427 differ between 326.5 kpps and 418.1 kpps with an average packet throughput between
428 361.7 kpps and 363.5 kpps. In summary, the PPS results seem consistent. Within each
429 test run of the four runs, when number of flows becomes larger, the packet
430 throughput seems not larger at the same time.
434 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
435 on different blades are measured when increasing the amount of UDP flows sent
436 between the VMs using pktgen as packet generator tool.
438 Round trip times and packet throughput between VMs can typically be affected by
439 the amount of flows set up and result in higher RTT and less PPS throughput.
441 The network latency is measured by ping, and the results of the four test runs
442 look similar with each other. Within each test run, the maximum RTT is only 47
443 ms and the average RTT is usually approx. 15 ms. On the whole, the average
444 RTTs of the four runs keep stable and the network latency is relatively small.
446 Cache utilization is measured by cachestat, which can display size of cache and
447 buffer in the system. Cache utilization statistics are collected during UDP
448 flows sent between the VMs using pktgen as packet generator tool. The largest
449 cache size is 214 MiB, which is same for the four runs, and the smallest cache
450 size is 94 MiB. On the whole, the average cache size of the four runs look the
451 same and is between 198 MiB and 207 MiB. Meanwhile, the tread of the buffer
452 size keep flat, since they have a minimum value of 7 MiB and a maximum value of
453 8 MiB, with an average value of about 7.9 MiB.
455 Packet throughput can be measured by pktgen, which is a tool in the network for
456 generating traffic loads for network experiments. The mean packet throughput of
457 the four test runs seem quite the same, which is approx. 363 kpps. The average
458 number of flows in these tests is 240k, and each run has a minimum number of
459 flows of 2 and a maximum number of flows of 1.001 Mil. At the same time, the
460 corresponding packet throughput differ between 327 kpps and 418 kpps with an
461 average packet throughput of about 363 kpps. Within each test run of the four
462 runs, when number of flows becomes larger, the packet throughput seems not
463 larger in the meantime.
467 The amount of packets per second (PPS) and round trip times (RTT) between 2 VMs
468 on different blades are measured when increasing the amount of UDP flows sent
469 between the VMs using pktgen as packet generator tool.
471 Round trip times and packet throughput between VMs can typically be affected by
472 the amount of flows set up and result in higher RTT and less PPS throughput.
474 The RTT results are similar throughout the different test dates and runs
475 between 0 ms and 47 ms with an average leatency of less than 16 ms. The PPS
476 results are not as consistent as the RTT results, for the mean packet
477 throughput of the four runs differ from 361.7 kpps to 365.0 kpps.
479 Network utilization is measured by sar, that is system activity reporter, which
480 can display the average statistics for the time since the system was started.
481 Network utilization statistics are collected during UDP flows sent between the
482 VMs using pktgen as packet generator tool. The largest total number of packets
483 transmitted per second look similar for two test runs, whose values change a
484 lot from 10 pps to 432 kpps. While results of the other test runs seem the same
485 and keep stable with the average number of packets transmitted per second of 10
486 pps. However, the total number of packets received per second of the four runs
487 look similar, which have a large wide range of 2 pps to 657 kpps.
489 In some test runs when running with less than approx. 250000 flows the PPS
490 throughput is normally flatter compared to when running with more flows, after
491 which the PPS throughput decreases. For the other test runs there is however no
492 significant change to the PPS throughput when the number of flows are
493 increased. In some test runs the PPS is also greater with 250000 flows
494 compared to other test runs where the PPS result is less with only 2 flows.
496 There are lost packets reported in most of the test runs. There is no observed
497 correlation between the amount of flows and the amount of lost packets.
498 The lost amount of packets normally differs a lot per test run.
500 Detailed test results
501 ---------------------
502 The scenario was run on Intel POD6_ with:
506 OpenVirtualSwitch 2.5.90
507 OpenDayLight Beryllium
509 Rationale for decisions
510 -----------------------
513 Conclusions and recommendations
514 -------------------------------
515 Tests were successfully executed and metrics collected.
516 No SLA was verified. To be decided on in next release of OPNFV.
Code Review / yardstick.git / blob