4 The NFV project requires fast live migration. The specific requirement is total
5 live migration time < 2Sec, while keeping the VM down time < 10ms when running
6 DPDK L2 forwarding workload.
8 We measured the baseline data of migrating an idle 8GiB guest running a DPDK L2
9 forwarding work load and observed that the total live migration time was 2271ms
10 while the VM downtime was 26ms. Both of these two indicators failed to satisfy
16 The following 4 features have been developed over the years to make the live
17 migration process faster.
20 Helps to reduce the network traffic by just sending the
23 Uses a specific NIC to increase the efficiency of data
25 + Multi thread compression:
26 Compresses the data before transmission.
28 Reduces the data rate of dirty pages.
30 Tests show none of the above features can satisfy the requirement of NFV.
31 XBZRLE and Multi thread compression do the compression entirely in software and
32 they are not fast enough in a 10Gbps network environment. RDMA is not flexible
33 because it has to transport all the guest memory to the destination without zero
34 page optimization. Auto convergence is not appropriate for NFV because it will
35 impact guest’s performance.
37 So we need to find other ways for optimization.
40 -------------------------
41 a. Delay non-emergency operations
42 By profiling, it was discovered that some of the cleanup operations during
43 the stop and copy stage are the main reason for the long VM down time. The
44 cleanup operation includes stopping the dirty page logging, which is a time
45 consuming operation. By deferring these operations until the data transmission
46 is completed the VM down time is reduced to about 5-7ms.
47 b. Optimize zero page checking
48 Currently QEMU uses the SSE2 instruction to optimize the zero pages
49 checking. The SSE2 instruction can process 16 bytes per instruction. By using
50 the AVX2 instruction, we can process 32 bytes per instruction. Testingt shows
51 that using AVX2 can speed up the zero pages checking process by about 25%.
52 c. Remove unnecessary context synchronization.
53 The CPU context was being synchronized twice during live migration. Removing
54 this unnecessary synchronization shortened the VM downtime by about 100us.
59 The source and destination host have the same hardware and OS:
62 CPU: Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
68 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
71 /root/qemu.git/x86_64-softmmu/qemu-system-x86_64-enable-kvm -cpu host -smp 4 –device virtio-net-pci,netdev=net1,mac=52:54:00:12:34:56 –netdev type=tap,id=net1,script=/etc/kvm/qemu-ifup,downscript=no,vhost=on–device virtio-net-pci,netdev=net2,mac=54:54:00:12:34:56 –netdevtype=tap,id=net2,script=/etc/kvm/qemu-ifup2,downscript=no,vhost=on -balloon virtio -m 8192-monitor stdio /mnt/liang/ia32e_rhel6u5.qcow
75 .. figure:: lmnetwork.jpg
77 :alt: live migration network connection
83 The down time is set to 10ms when doing the test. We use pktgen to send the
84 packages to guest, the package size is 64 bytes, and the line rate is 2013
87 a. Total live migration time
89 The total live migration time before and after optimization is shown in the
90 chart below. For an idle guest, we can reduce the total live migration time
91 from 2070ms to 401ms. For a guest running the DPDK L2 forwarding workload,
92 the total live migration time is reduced from 2271ms to 654ms.
94 .. figure:: lmtotaltime.jpg
96 :alt: total live migration time
100 The VM down time before and after optimization is shown in the chart below.
101 For an idle guest, we can reduce the VM down time from 29ms to 9ms. For a guest
102 running the DPDK L2 forwarding workload, the VM down time is reduced from 26ms to
105 .. figure:: lmdowntime.jpg