1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
|
% Migration
% Revision 2
\clearpage
# Basics
--------------- -------------
Status: **Supported**
Architecture: x86
Component: Toolstack
--------------- -------------
# Overview
Migration is a mechanism to move a virtual machine while the VM is
running. Live migration moves a running virtual machine between two
physical servers, but the same mechanism can be used for non-live
migration (pause and copy) and suspend/resume from disk.
# User details
No hardware requirements, although hypervisor logdirty support is
required for live migration.
From the command line, `xl migrate/save/restore` are the top level
interactions. e.g.
xl create my-vm.cfg
xl migrate my-vm localhost
or
xl create my-vm.cfg
xl save my-vm /path/to/save/file
xl restore /path/to/save/file
Xen 4.6 sees the introduction of Migration v2. There is no change for
people using `xl`, although the `libxl` API has had an extension.
# Technical details
Migration is formed of several layers. `libxc` is responsible for the
contents of the VM (ram, vcpus, etc) and the live migration loop, while
`libxl` is responsible for items such as emulator state.
The format of the migration v2 stream is specified in two documents, and
is architecture neutral. Compatibility with legacy streams is
maintained via the `convert-legacy-stream` script which transforms a
legacy stream into a migration v2 stream.
* Documents
* `docs/specs/libxc-migration-stream.pandoc`
* `docs/specs/libxl-migration-stream.pandoc`
* `libxc`
* `tools/libxc/xc_sr_*.[hc]`
* `libxl`
* `tools/libxl/libxl_stream_{read,write}.c`
* `tools/libxl/libxl_convert_callout.c`
* Scripts
* `tools/python/xen/migration/*.py`
* `tools/python/scripts/convert-legacy-stream`
* `tools/python/scripts/verify-stream-v2`
## libxl
With migration v2 support, LIBXL_HAVE_SRM_V2 and LIBXL_HAVE_SRM_V1
are introduced to indicate support. `domain_restore_params` gains a new
parameter, `stream_version`, which is used to distinguish between legacy and
v2 migration streams, and hence whether legacy conversion is required.
# Limitations
Hypervisor logdirty support is incompatible with hardware passthrough,
as IOMMU faults cannot be used to track writes.
While not a bug in migration specifically, VMs are very sensitive to
changes in cpuid information, and cpuid levelling support currently has
its issues. Extreme care should be taken when migrating VMs between
non-identical CPUs until the cpuid levelling improvements are complete.
# Testing
Changes in libxc should be tested with every guest type (32bit PV, 64bit
PV, HVM), while changes in libxl should test HVM guests with both
qemu-traditional and qemu-upstream.
In general, testing can be done on a single host using `xl
save/restore` or `xl migrate $VM localhost`.
Any changes to the conversion script should be tested in all upgrade
scenarios, which will involve starting with VMs from Xen 4.5
# Areas for improvement
* Arm support
* Live looping parameters
# Known issues
* x86 HVM guest physmap operations (not reflected in logdirty bitmap)
* x86 HVM with PoD pages (attempts to map cause PoD allocations)
* x86 HVM with nested-virt (no relevant information included in the
stream)
* x86 PV ballooning (P2M marked dirty, target frame not marked)
* x86 PV P2M structure changes (not noticed, stale mappings used) for
guests not using the linear p2m layout
# References
Xen Developer Summit 2015 Presentation
[video](https://www.youtube.com/watch?v=RwiDeG21lrc) and
[slides](http://events.linuxfoundation.org/sites/events/files/slides/migv2.pdf)
for Migration v2
# History
------------------------------------------------------------------------
Date Revision Version Notes
---------- -------- -------- -------------------------------------------
2015-10-24 1 Xen 4.6 Document written
2015-12-11 2 Xen 4.7 Support of linear p2m list
---------- -------- -------- -------------------------------------------
|
