/usr/share/doc/ganeti/html/design-hotplug.html is in ganeti-doc 2.16.0~rc2-1build1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Hotplug — Ganeti 2.16.0~rc2 documentation</title>
<link rel="stylesheet" href="_static/style.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '2.16.0~rc2',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Detection of user-initiated shutdown from inside an instance" href="design-internal-shutdown.html" />
<link rel="prev" title="HSqueeze tool" href="design-hsqueeze.html" />
</head>
<body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-internal-shutdown.html" title="Detection of user-initiated shutdown from inside an instance"
accesskey="N">next</a></li>
<li class="right" >
<a href="design-hsqueeze.html" title="HSqueeze tool"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="hotplug">
<h1><a class="toc-backref" href="#id1">Hotplug</a><a class="headerlink" href="#hotplug" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Created:</th><td class="field-body">2013-Jul-24</td>
</tr>
<tr class="field-even field"><th class="field-name">Status:</th><td class="field-body">Implemented</td>
</tr>
<tr class="field-odd field"><th class="field-name">Ganeti-Version:</th><td class="field-body">2.10.0</td>
</tr>
</tbody>
</table>
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#hotplug" id="id1">Hotplug</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings" id="id2">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes" id="id3">Proposed changes</a></li>
<li><a class="reference internal" href="#design-decisions" id="id4">Design decisions</a><ul>
<li><a class="reference internal" href="#configuration-changes" id="id5">Configuration changes</a></li>
<li><a class="reference internal" href="#backend-changes" id="id6">Backend changes</a></li>
<li><a class="reference internal" href="#hypervisor-changes" id="id7">Hypervisor changes</a></li>
<li><a class="reference internal" href="#user-interface" id="id8">User interface</a><ul>
<li><a class="reference internal" href="#enabling-hotplug" id="id9">Enabling hotplug</a></li>
<li><a class="reference internal" href="#nic-hotplug" id="id10">NIC Hotplug</a></li>
<li><a class="reference internal" href="#disk-hotplug" id="id11">Disk Hotplug</a></li>
</ul>
</li>
<li><a class="reference internal" href="#dealing-with-chroot-and-uid-pool-and-disks-in-general" id="id12">Dealing with chroot and uid pool (and disks in general)</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<p>This is a design document detailing the implementation of device
hotplugging in Ganeti. The logic used is hypervisor agnostic but still
the initial implementation will target the KVM hypervisor. The
implementation adds <code class="docutils literal"><span class="pre">python-fdsend</span></code> as a new dependency. In case
it is not installed hotplug will not be possible and the user will
be notified with a warning.</p>
<div class="section" id="current-state-and-shortcomings">
<h2><a class="toc-backref" href="#id2">Current state and shortcomings</a><a class="headerlink" href="#current-state-and-shortcomings" title="Permalink to this headline">¶</a></h2>
<p>Currently, Ganeti supports addition/removal/modification of devices
(NICs, Disks) but the actual modification takes place only after
rebooting the instance. To this end an instance cannot change network,
get a new disk etc. without a hard reboot.</p>
<p>Until now, in case of KVM hypervisor, code does not name devices nor
places them in specific PCI slots. Devices are appended in the KVM
command and Ganeti lets KVM decide where to place them. This means that
there is a possibility a device that resides in PCI slot 5, after a
reboot (due to another device removal) to be moved to another PCI slot
and probably get renamed too (due to udev rules, etc.).</p>
<p>In order for a migration to succeed, the process on the target node
should be started with exactly the same machine version, CPU
architecture and PCI configuration with the running process. During
instance creation/startup ganeti creates a KVM runtime file with all the
necessary information to generate the KVM command. This runtime file is
used during instance migration to start a new identical KVM process. The
current format includes the fixed part of the final KVM command, a list
of NICs’, and hvparams dict. It does not favor easy manipulations
concerning disks, because they are encapsulated in the fixed KVM
command.</p>
</div>
<div class="section" id="proposed-changes">
<h2><a class="toc-backref" href="#id3">Proposed changes</a><a class="headerlink" href="#proposed-changes" title="Permalink to this headline">¶</a></h2>
<p>For the case of the KVM hypervisor, QEMU exposes 32 PCI slots to the
instance. Disks and NICs occupy some of these slots. Recent versions of
QEMU have introduced monitor commands that allow addition/removal of PCI
devices. Devices are referenced based on their name or position on the
virtual PCI bus. To be able to use these commands, we need to be able to
assign each device a unique name.</p>
<p>To keep track where each device is plugged into, we add the
<code class="docutils literal"><span class="pre">pci</span></code> slot to Disk and NIC objects, but we save it only in runtime
files, since it is hypervisor specific info. This is added for easy
object manipulation and is ensured not to be written back to the config.</p>
<p>We propose to make use of QEMU 1.7 QMP commands so that
modifications to devices take effect instantly without the need for hard
reboot. The only change exposed to the end-user will be the addition of
a <code class="docutils literal"><span class="pre">--hotplug</span></code> option to the <code class="docutils literal"><span class="pre">gnt-instance</span> <span class="pre">modify</span></code> command.</p>
<p>Upon hotplugging the PCI configuration of an instance is changed.
Runtime files should be updated correspondingly. Currently this is
impossible in case of disk hotplug because disks are included in command
line entry of the runtime file, contrary to NICs that are correctly
treated separately. We change the format of runtime files, we remove
disks from the fixed KVM command and create new entry containing them
only. KVM options concerning disk are generated during
<code class="docutils literal"><span class="pre">_ExecuteKVMCommand()</span></code>, just like NICs.</p>
</div>
<div class="section" id="design-decisions">
<h2><a class="toc-backref" href="#id4">Design decisions</a><a class="headerlink" href="#design-decisions" title="Permalink to this headline">¶</a></h2>
<p>Which should be each device ID? Currently KVM does not support arbitrary
IDs for devices; supported are only names starting with a letter, max 32
chars length, and only including ‘.’ ‘_’ ‘-‘ special chars.
For debugging purposes and in order to be more informative, device will be
named after: <device type>-<part of uuid>-pci-<slot>.</p>
<p>Who decides where to hotplug each device? As long as this is a
hypervisor specific matter, there is no point for the master node to
decide such a thing. Master node just has to request noded to hotplug a
device. To this end, hypervisor specific code should parse the current
PCI configuration (i.e. <code class="docutils literal"><span class="pre">query-pci</span></code> QMP command), find the first
available slot and hotplug the device. Having noded to decide where to
hotplug a device we ensure that no error will occur due to duplicate
slot assignment (if masterd keeps track of PCI reservations and noded
fails to return the PCI slot that the device was plugged into then next
hotplug will fail).</p>
<p>Where should we keep track of devices’ PCI slots? As already mentioned,
we must keep track of devices PCI slots to successfully migrate
instances. First option is to save this info to config data, which would
allow us to place each device at the same PCI slot after reboot. This
would require to make the hypervisor return the PCI slot chosen for each
device, and storing this information to config data. Additionally the
whole instance configuration should be returned with PCI slots filled
after instance start and each instance should keep track of current PCI
reservations. We decide not to go towards this direction in order to
keep it simple and do not add hypervisor specific info to configuration
data (<code class="docutils literal"><span class="pre">pci_reservations</span></code> at instance level and <code class="docutils literal"><span class="pre">pci</span></code> at device
level). For the aforementioned reason, we decide to store this info only
in KVM runtime files.</p>
<p>Where to place the devices upon instance startup? QEMU has by default 4
pre-occupied PCI slots. So, hypervisor can use the remaining ones for
disks and NICs. Currently, PCI configuration is not preserved after
reboot. Each time an instance starts, KVM assigns PCI slots to devices
based on their ordering in Ganeti configuration, i.e. the second disk
will be placed after the first, the third NIC after the second, etc.
Since we decided that there is no need to keep track of devices PCI
slots, there is no need to change current functionality.</p>
<p>How to deal with existing instances? Hotplug depends on runtime file
manipulation. It stores there pci info and every device the kvm process is
currently using. Existing files have no pci info in devices and have block
devices encapsulated inside kvm_cmd entry. Thus hotplugging of existing devices
will not be possible. Still migration and hotplugging of new devices will
succeed. The workaround will happen upon loading kvm runtime: if we detect old
style format we will add an empty list for block devices and upon saving kvm
runtime we will include this empty list as well. Switching entirely to new
format will happen upon instance reboot.</p>
<div class="section" id="configuration-changes">
<h3><a class="toc-backref" href="#id5">Configuration changes</a><a class="headerlink" href="#configuration-changes" title="Permalink to this headline">¶</a></h3>
<p>The <code class="docutils literal"><span class="pre">NIC</span></code> and <code class="docutils literal"><span class="pre">Disk</span></code> objects get one extra slot: <code class="docutils literal"><span class="pre">pci</span></code>. It refers to
PCI slot that the device gets plugged into.</p>
<p>In order to be able to live migrate successfully, runtime files should
be updated every time a live modification (hotplug) takes place. To this
end we change the format of runtime files. The KVM options referring to
instance’s disks are no longer recorded as part of the KVM command line.
Disks are treated separately, just as we treat NICs right now. We insert
and remove entries to reflect the current PCI configuration.</p>
</div>
<div class="section" id="backend-changes">
<h3><a class="toc-backref" href="#id6">Backend changes</a><a class="headerlink" href="#backend-changes" title="Permalink to this headline">¶</a></h3>
<p>Introduce one new RPC call:</p>
<ul class="simple">
<li>hotplug_device(DEVICE_TYPE, ACTION, device, …)</li>
</ul>
<p>where DEVICE_TYPE can be either NIC or Disk, and ACTION either REMOVE or ADD.</p>
</div>
<div class="section" id="hypervisor-changes">
<h3><a class="toc-backref" href="#id7">Hypervisor changes</a><a class="headerlink" href="#hypervisor-changes" title="Permalink to this headline">¶</a></h3>
<p>We implement hotplug on top of the KVM hypervisor. We take advantage of
QEMU 1.7 QMP commands (<code class="docutils literal"><span class="pre">device_add</span></code>, <code class="docutils literal"><span class="pre">device_del</span></code>,
<code class="docutils literal"><span class="pre">blockdev-add</span></code>, <code class="docutils literal"><span class="pre">netdev_add</span></code>, <code class="docutils literal"><span class="pre">netdev_del</span></code>). Since <code class="docutils literal"><span class="pre">drive_del</span></code>
is not yet implemented in QMP we use the one of HMP. QEMU
refers to devices based on their id. We use <code class="docutils literal"><span class="pre">uuid</span></code> to name them
properly. If a device is about to be hotplugged we parse the output of
<code class="docutils literal"><span class="pre">query-pci</span></code> and find the occupied PCI slots. We choose the first
available and the whole device object is appended to the corresponding
entry in the runtime file.</p>
<p>Concerning NIC handling, we build on the top of the existing logic
(first create a tap with _OpenTap() and then pass its file descriptor to
the KVM process). To this end we need to pass access rights to the
corresponding file descriptor over the QMP socket (UNIX domain
socket). The open file is passed as a socket-level control message
(SCM), using the <code class="docutils literal"><span class="pre">fdsend</span></code> python library.</p>
</div>
<div class="section" id="user-interface">
<h3><a class="toc-backref" href="#id8">User interface</a><a class="headerlink" href="#user-interface" title="Permalink to this headline">¶</a></h3>
<p>The new <code class="docutils literal"><span class="pre">--hotplug</span></code> option to gnt-instance modify is introduced, which
forces live modifications.</p>
<div class="section" id="enabling-hotplug">
<h4><a class="toc-backref" href="#id9">Enabling hotplug</a><a class="headerlink" href="#enabling-hotplug" title="Permalink to this headline">¶</a></h4>
<p>Hotplug will be optional during gnt-instance modify. For existing
instance, after installing a version that supports hotplugging we
have the restriction that hotplug will not be supported for existing
devices. The reason is that old runtime files lack of:</p>
<ol class="arabic simple">
<li>Device pci configuration info.</li>
<li>Separate block device entry.</li>
</ol>
<p>Hotplug will be supported only for KVM in the first implementation. For
all other hypervisors, backend will raise an Exception case hotplug is
requested.</p>
</div>
<div class="section" id="nic-hotplug">
<h4><a class="toc-backref" href="#id10">NIC Hotplug</a><a class="headerlink" href="#nic-hotplug" title="Permalink to this headline">¶</a></h4>
<p>The user can add/modify/remove NICs either with hotplugging or not. If a
NIC is to be added a tap is created first and configured properly with
kvm-vif-bridge script. Then the instance gets a new network interface.
Since there is no QEMU monitor command to modify a NIC, we modify a NIC
by temporary removing the existing one and adding a new with the new
configuration. When removing a NIC the corresponding tap gets removed as
well.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">--</span><span class="n">net</span> <span class="n">add</span> <span class="o">--</span><span class="n">hotplug</span> <span class="n">test</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">--</span><span class="n">net</span> <span class="mi">1</span><span class="p">:</span><span class="n">mac</span><span class="o">=</span><span class="n">aa</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">55</span><span class="p">:</span><span class="mi">44</span><span class="p">:</span><span class="mi">33</span> <span class="o">--</span><span class="n">hotplug</span> <span class="n">test</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">--</span><span class="n">net</span> <span class="mi">1</span><span class="p">:</span><span class="n">remove</span> <span class="o">--</span><span class="n">hotplug</span> <span class="n">test</span>
</pre></div>
</div>
</div>
<div class="section" id="disk-hotplug">
<h4><a class="toc-backref" href="#id11">Disk Hotplug</a><a class="headerlink" href="#disk-hotplug" title="Permalink to this headline">¶</a></h4>
<p>The user can add and remove disks with hotplugging or not. QEMU monitor
supports resizing of disks, however the initial implementation will
support only disk addition/deletion.</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">--</span><span class="n">disk</span> <span class="n">add</span><span class="p">:</span><span class="n">size</span><span class="o">=</span><span class="mi">1</span><span class="n">G</span> <span class="o">--</span><span class="n">hotplug</span> <span class="n">test</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">--</span><span class="n">net</span> <span class="mi">1</span><span class="p">:</span><span class="n">remove</span> <span class="o">--</span><span class="n">hotplug</span> <span class="n">test</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="dealing-with-chroot-and-uid-pool-and-disks-in-general">
<h3><a class="toc-backref" href="#id12">Dealing with chroot and uid pool (and disks in general)</a><a class="headerlink" href="#dealing-with-chroot-and-uid-pool-and-disks-in-general" title="Permalink to this headline">¶</a></h3>
<p>The design so far covers all issues that arise without addressing the
case where the kvm process will not run with root privileges.
Specifically:</p>
<ul class="simple">
<li>in case of chroot, the kvm process cannot see the newly created device</li>
<li>in case of uid pool security model, the kvm process is not allowed
to access the device</li>
</ul>
<p>For NIC hotplug we address this problem by using the <code class="docutils literal"><span class="pre">getfd</span></code> QMP
command and passing the file descriptor to the kvm process over the
monitor socket using SCM_RIGHTS. For disk hotplug and in case of uid
pool we can let the hypervisor code temporarily <code class="docutils literal"><span class="pre">chown()</span></code> the device
before the actual hotplug. Still this is insufficient in case of chroot.
In this case, we need to <code class="docutils literal"><span class="pre">mknod()</span></code> the device inside the chroot. Both
workarounds can be avoided, if we make use of the <code class="docutils literal"><span class="pre">add-fd</span></code>
QMP command, that was introduced in version 1.7. This command is the
equivalent of NICs’ <cite>get-fd`</cite> for disks and will allow disk hotplug in
every case. So, if the QMP does not support the <code class="docutils literal"><span class="pre">add-fd</span></code>
command, we will not allow disk hotplug
and notify the user with the corresponding warning.</p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Hotplug</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes">Proposed changes</a></li>
<li><a class="reference internal" href="#design-decisions">Design decisions</a><ul>
<li><a class="reference internal" href="#configuration-changes">Configuration changes</a></li>
<li><a class="reference internal" href="#backend-changes">Backend changes</a></li>
<li><a class="reference internal" href="#hypervisor-changes">Hypervisor changes</a></li>
<li><a class="reference internal" href="#user-interface">User interface</a><ul>
<li><a class="reference internal" href="#enabling-hotplug">Enabling hotplug</a></li>
<li><a class="reference internal" href="#nic-hotplug">NIC Hotplug</a></li>
<li><a class="reference internal" href="#disk-hotplug">Disk Hotplug</a></li>
</ul>
</li>
<li><a class="reference internal" href="#dealing-with-chroot-and-uid-pool-and-disks-in-general">Dealing with chroot and uid pool (and disks in general)</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="design-hsqueeze.html"
title="previous chapter">HSqueeze tool</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="design-internal-shutdown.html"
title="next chapter">Detection of user-initiated shutdown from inside an instance</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/design-hotplug.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-internal-shutdown.html" title="Detection of user-initiated shutdown from inside an instance"
>next</a></li>
<li class="right" >
<a href="design-hsqueeze.html" title="HSqueeze tool"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2018, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Google Inc..
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
</div>
</body>
</html>
|