/usr/share/doc/ganeti/html/design-scsi-kvm.html is in ganeti-doc 2.16.0~rc2-1build1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>KVM + SCSI — Ganeti 2.16.0~rc2 documentation</title>
<link rel="stylesheet" href="_static/style.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '2.16.0~rc2',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Disks" href="design-disks.html" />
<link rel="prev" title="Ganeti Maintenance Daemon" href="design-repaird.html" />
</head>
<body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-disks.html" title="Disks"
accesskey="N">next</a></li>
<li class="right" >
<a href="design-repaird.html" title="Ganeti Maintenance Daemon"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="design-draft.html" accesskey="U">Design document drafts</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="kvm-scsi">
<h1><a class="toc-backref" href="#id1">KVM + SCSI</a><a class="headerlink" href="#kvm-scsi" title="Permalink to this headline">¶</a></h1>
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#kvm-scsi" id="id1">KVM + SCSI</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings" id="id2">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes" id="id3">Proposed changes</a></li>
<li><a class="reference internal" href="#design-decisions" id="id4">Design decisions</a><ul>
<li><a class="reference internal" href="#configuration-changes" id="id5">Configuration changes</a></li>
<li><a class="reference internal" href="#hypervisor-changes" id="id6">Hypervisor changes</a></li>
<li><a class="reference internal" href="#user-interface" id="id7">User interface</a></li>
</ul>
</li>
</ul>
</li>
</ul>
</div>
<p>This is a design document detailing the refactoring of device
handling in the KVM Hypervisor. More specifically, it will use
the latest QEMU device model and modify the hotplug implementation
so that both PCI and SCSI devices can be managed.</p>
<div class="section" id="current-state-and-shortcomings">
<h2><a class="toc-backref" href="#id2">Current state and shortcomings</a><a class="headerlink" href="#current-state-and-shortcomings" title="Permalink to this headline">¶</a></h2>
<p>Ganeti currently supports SCSI virtual devices in the KVM hypervisor by
setting the <cite>disk_type</cite> hvparam to <cite>scsi</cite>. Ganeti will eventually
instruct QEMU to use the deprecated device model (i.e. -drive if=scsi),
which will expose the backing store as an emulated SCSI device. This
means that currently SCSI pass-through is not supported.</p>
<p>On the other hand, the current hotplug implementation
<a class="reference internal" href="design-hotplug.html"><span class="doc">Hotplug</span></a> uses the latest QEMU
device model (via the -device option) and is tailored to paravirtual
devices, which leads to buggy behavior: if we hotplug a disk to an
instance that is configured with disk_type=scsi hvparam, the
disk which will get hot-plugged eventually will be a VirtIO device
(i.e., virtio-blk-pci) on the PCI bus.</p>
<p>The current implementation of creating the QEMU command line is
error-prone, since an instance might not be able to boot due to PCI slot
congestion.</p>
</div>
<div class="section" id="proposed-changes">
<h2><a class="toc-backref" href="#id3">Proposed changes</a><a class="headerlink" href="#proposed-changes" title="Permalink to this headline">¶</a></h2>
<p>We change the way that the KVM hypervisor handles block devices by
introducing latest QEMU device model for SCSI devices as well, so that
scsi-cd, scsi-hd, scsi-block, and scsi-generic device drivers are
supported too. Additionally we refactor the hotplug implementation in
order to support hotplugging of SCSI devices too. Finally, we change the
way we keep track of device info inside runtime files, and the way we
place each device upon instance startup.</p>
</div>
<div class="section" id="design-decisions">
<h2><a class="toc-backref" href="#id4">Design decisions</a><a class="headerlink" href="#design-decisions" title="Permalink to this headline">¶</a></h2>
<p>How to identify each device?</p>
<p>Currently KVM does not support arbitrary IDs for devices; supported are
only names starting with a letter, with max 32 chars length, and only
including the ‘.’, ‘_’, ‘-‘ special chars. Currently we generate an ID
with the following format: <device type>-<part of uuid>-pci-<slot>.
This assumes that the device will be plugged in a certain slot on the
PCI bus. Since we want to support devices on a SCSI bus too and adding
the PCI slot to the ID is redundant, we dump the last two parts of the
existing ID. Additionally we get rid of the ‘hot’ prefix of device type,
and we add the next two parts of the UUID so the chance of collitions
is reduced significantly. So, as an example, the device ID of a disk
with UUID ‘9e7c85f6-b6e5-4243-b27d-680b78c6d203’ would be now
‘disk-9e7c85f6-b6e5-4243’.</p>
<p>Which buses does the guest eventually see?</p>
<p>By default QEMU starts with a single PCI bus named “pci.0”. In case a
SCSI controller is added on this bus, a SCSI bus is created with
the corresponding name: “scsi.0”.
Any SCSI disks will be attached on this SCSI bus. Currently Ganeti does
not explicitly use a SCSI controller via a command line option, but lets
QEMU add one automatically if needed. Here, in case we have a SCSI disk,
a SCSI controller is explicitly added via the -device option. For the
SCSI controller, we do not specify the PCI slot to use, but let QEMU find
the first available (see below).</p>
<p>What type of SCSI controller to use?</p>
<p>QEMU uses the <cite>lsi</cite> controller by default. To make this configurable we
add a new hvparam, <cite>scsi_controller_type</cite>. The available types will be
<cite>lsi</cite>, <cite>megasas</cite>, and <cite>virtio-scsi-pci</cite>.</p>
<p>Where to place the devices upon instance startup?</p>
<p>The default QEMU machine type, <cite>pc</cite>, adds a <cite>i440FX-pcihost</cite>
controller on the root bus that creates a PCI bus with <cite>pci.0</cite> alias.
By default the first three slots of this bus are occupied: slot 0
for Host bridge, slot 1 for ISA bridge, and slot 2 for VGA controller.
Thereafter, the slots depend on the QEMU options passed in the command
line.</p>
<p>The main reason that we want to be fully aware of the configuration of a
running instance (machine type, PCI and SCSI bus state, devices, etc.)
is that in case of migration a QEMU process with the exact same
configuration should be created on the target node. The configuration is
kept in the runtime file created just before starting the instance.
Since hotplug has been introduced, the only thing that can change after
starting an instance is the configuration related to NICs and Disks.</p>
<p>Before implementing hotplug, Ganeti did not specify PCI slots
explicitly, but let QEMU decide how to place the devices on the
corresponding bus. This does not work if we want to have hotplug-able
devices and migrate-able VMs. Currently, upon runtime file creation, we
try to reserve PCI slots based on the hvparams, the disks, and the NICs
of the instance. This has three major shortcomings: first, we have to be
aware which options modify the PCI bus which is practically impossible
due to the huge amount of QEMU options, second, QEMU may change the
default PCI configuration from version to version, and third, we cannot
know if the extra options passed by the user via the <cite>kvm_extra</cite> hvparam
modify the PCI bus.</p>
<p>All the above makes the current implementation error prone: an instance
might not be able to boot if we explicitly add a NIC/Disk on a specific
PCI slot that QEMU has already used for another device while parsing
its command line options. Besides that, now, we want to use the SCSI bus
as well so the above mechanism is insufficient. Here, we decide to put
only disks and NICs on specific slots on the corresponding bus, and let
QEMU put everything else automatically. To this end, we decide to let
the first 12 PCI slots be managed by QEMU, and we start adding PCI
devices (VirtIO block and network devices) from the 13th slot onwards.
As far as the SCSI bus is concerned, we decide to put each SCSI
disk on a different scsi-id (which corresponds to a different target
number in SCSI terminology). The SCSI bus will not have any default
reservations.</p>
<p>How to support the theoretical maximum of devices, 16 disks and 8 NICs?</p>
<p>By default, one could add up to 20 devices on the PCI bus; that is the
32 slots of the PCI bus, minus the starting 12 slots that Ganeti
allows QEMU to manage on its own. In order to by able to add
more PCI devices, we add the new <cite>kvm_pci_reservations</cite> hvparam to
denote how many PCI slots QEMU will handle implicitly. The rest will be
available for disk and NICs inserted explicitly by Ganeti. By default
the default PCI reservations will be 12 as explained above.</p>
<p>How to keep track of the bus state of a running instance?</p>
<p>To be able to hotplug a device, we need to know which slot is
available on the desired bus. Until now, we were using the <code class="docutils literal"><span class="pre">query-pci</span></code>
QMP command that returns the state of the PCI buses (i.e., which devices
occupy which slots). Unfortunately, there is no equivalent for the SCSI
buses. We could use the <code class="docutils literal"><span class="pre">info</span> <span class="pre">qtree</span></code> HMP command that practically
dumps in plain text the whole device tree. This makes it really hard to
parse. So we decide to generate the bus state of a running instance
through our local runtime files.</p>
<p>What info should be kept in runtime files?</p>
<p>Runtime files are used for instance migration (to run a QEMU process on
the target node with the same configuration) and for hotplug actions (to
update the configuration of a running instance so that it can be
migrated). Until now we were using devices only on the PCI bus, so only
each device’s PCI slot should be kept in the runtime file. This is
obviously not enough. We decide to replace the <cite>pci</cite> slot of Disk and
NIC configuration objects, with an <cite>hvinfo</cite> dict. It will contain all
necessary info for constructing the appropriate -device QEMU option.
Specifically the <cite>driver</cite>, <cite>id</cite>, and <cite>bus</cite> parameters will be present to
all kind of devices. PCI devices will have the <cite>addr</cite> parameter, SCSI
devices will have <cite>channel</cite>, <cite>scsi-id</cite>, and <cite>lun</cite>. NICs and Disks will
have the extra <cite>netdev</cite> and <cite>drive</cite> parameters correspondingly.</p>
<p>How to deal with existing instances?</p>
<p>Only existing instances with paravirtual devices (configured via the
disk_type and nic_type hvparam) use the latest QEMU device model. Only
these have the <cite>pci</cite> slot filled. We will use the existing
_UpgradeSerializedRuntime() method to migrate the old runtime format
with <cite>pci</cite> slot in Disk and NIC configuration objects to the new one
with <cite>hvinfo</cite> instead. The new hvinfo will contain the old driver
(either virtio-blk-pci or virtio-net-pci), the old id
(hotdisk-123456-pci-4), the default PCI bus (pci.0), and the old PCI
slot (addr=4). This way those devices will still be hotplug-able, and
the instance will still be migrate-able. When those instances are
rebooted, the hvinfo will be re-generated.</p>
<p>How to support downgrades?</p>
<p>There are two possible ways, both not very pretty. The first one is to
use _UpgradeSerializedRuntime() to remove the hvinfo slot. This would
require the patching of all Ganeti versions down to 2.10 which is practically
imposible. Another way is to ssh to all nodes and remove this slot upon
a cluster downgrade. This ugly hack would go away on 2.17 since we support
downgrades only to the previous minor version.</p>
<div class="section" id="configuration-changes">
<h3><a class="toc-backref" href="#id5">Configuration changes</a><a class="headerlink" href="#configuration-changes" title="Permalink to this headline">¶</a></h3>
<p>The <code class="docutils literal"><span class="pre">NIC</span></code> and <code class="docutils literal"><span class="pre">Disk</span></code> objects get one extra slot: <code class="docutils literal"><span class="pre">hvinfo</span></code>. It is
hypervisor-specific and will never reach config.data. In case of the KVM
Hypervisor it will contain all necessary info for constructing the -device
QEMU option. Existing entries in runtime files that had a <cite>pci</cite> slot
will be upgraded to have the corresponding <cite>hvinfo</cite> (see above).</p>
<p>The new <cite>scsi_controller_type</cite> hvparam is added to denote what type of
SCSI controller should be added to PCI bus if we have a SCSI disk.
Allowed values will be <cite>lsi</cite>, <cite>virtio-scsi-pci</cite>, and <cite>megasas</cite>.
We decide to use <cite>lsi</cite> by default since this is the one that QEMU
adds automatically if not specified explicitly by an option.</p>
</div>
<div class="section" id="hypervisor-changes">
<h3><a class="toc-backref" href="#id6">Hypervisor changes</a><a class="headerlink" href="#hypervisor-changes" title="Permalink to this headline">¶</a></h3>
<p>The current implementation verifies if a hotplug action has succeeded
by scanning the PCI bus and searching for a specific device ID. This
will change, and we will use the <code class="docutils literal"><span class="pre">query-block</span></code> along with the
<code class="docutils literal"><span class="pre">query-pci</span></code> QMP command to find block devices that are attached to the
SCSI bus as well.</p>
<p>Up until now, if <cite>disk_type</cite> hvparam was set to <cite>scsi</cite>, QEMU would use the
deprecated device model and end up using SCSI emulation, e.g.:</p>
<blockquote>
<div><div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">-</span><span class="n">drive</span> <span class="n">file</span><span class="o">=/</span><span class="n">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">ganeti</span><span class="o">/</span><span class="n">instance</span><span class="o">-</span><span class="n">disks</span><span class="o">/</span><span class="n">test</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="k">if</span><span class="o">=</span><span class="n">scsi</span><span class="p">,</span><span class="nb">format</span><span class="o">=</span><span class="n">raw</span>
</pre></div>
</div>
</div></blockquote>
<p>Now the equivalent, which will also enable hotplugging, will be to set
disk_type to <cite>scsi-hd</cite>. The QEMU command line will include:</p>
<blockquote>
<div><div class="highlight-default"><div class="highlight"><pre><span></span><span class="o">-</span><span class="n">drive</span> <span class="n">file</span><span class="o">=/</span><span class="n">var</span><span class="o">/</span><span class="n">run</span><span class="o">/</span><span class="n">ganeti</span><span class="o">/</span><span class="n">instance</span><span class="o">-</span><span class="n">disks</span><span class="o">/</span><span class="n">test</span><span class="p">:</span><span class="mi">0</span><span class="p">,</span><span class="k">if</span><span class="o">=</span><span class="n">none</span><span class="p">,</span><span class="nb">format</span><span class="o">=</span><span class="n">raw</span><span class="p">,</span><span class="nb">id</span><span class="o">=</span><span class="n">disk</span><span class="o">-</span><span class="mf">9e7</span><span class="n">c85f6</span><span class="o">-</span><span class="n">b6e5</span><span class="o">-</span><span class="mi">4243</span>
<span class="o">-</span><span class="n">device</span> <span class="n">scsi</span><span class="o">-</span><span class="n">hd</span><span class="p">,</span><span class="nb">id</span><span class="o">=</span><span class="n">disk</span><span class="o">-</span><span class="mf">9e7</span><span class="n">c85f6</span><span class="o">-</span><span class="n">b6e5</span><span class="o">-</span><span class="mi">4243</span><span class="p">,</span><span class="n">drive</span><span class="o">=</span><span class="n">disk</span><span class="o">-</span><span class="mf">9e7</span><span class="n">c85f6</span><span class="o">-</span><span class="n">b6e5</span><span class="o">-</span><span class="mi">4243</span><span class="p">,</span><span class="n">bus</span><span class="o">=</span><span class="n">scsi</span><span class="o">.</span><span class="mi">0</span><span class="p">,</span><span class="n">channel</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">scsi</span><span class="o">-</span><span class="nb">id</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span><span class="n">lun</span><span class="o">=</span><span class="mi">0</span>
</pre></div>
</div>
</div></blockquote>
</div>
<div class="section" id="user-interface">
<h3><a class="toc-backref" href="#id7">User interface</a><a class="headerlink" href="#user-interface" title="Permalink to this headline">¶</a></h3>
<p>The <cite>disk_type</cite> hvparam will additionally support the <cite>scsi-hd</cite>,
<cite>scsi-block</cite>, and <cite>scsi-generic</cite> values. The first one is equivalent to
the existing <cite>scsi</cite> value and will make QEMU emulate a SCSI device,
while the last two will add support for SCSI pass-through and will
require a real SCSI device on the host.</p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">KVM + SCSI</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes">Proposed changes</a></li>
<li><a class="reference internal" href="#design-decisions">Design decisions</a><ul>
<li><a class="reference internal" href="#configuration-changes">Configuration changes</a></li>
<li><a class="reference internal" href="#hypervisor-changes">Hypervisor changes</a></li>
<li><a class="reference internal" href="#user-interface">User interface</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="design-repaird.html"
title="previous chapter">Ganeti Maintenance Daemon</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="design-disks.html"
title="next chapter">Disks</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/design-scsi-kvm.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-disks.html" title="Disks"
>next</a></li>
<li class="right" >
<a href="design-repaird.html" title="Ganeti Maintenance Daemon"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
<li class="nav-item nav-item-1"><a href="design-draft.html" >Design document drafts</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2018, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Google Inc..
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
</div>
</body>
</html>
|