/usr/share/doc/ganeti/html/design-virtual-clusters.html is in ganeti-doc 2.16.0~rc2-1build1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Design for virtual clusters support — Ganeti 2.16.0~rc2 documentation</title>
<link rel="stylesheet" href="_static/style.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '2.16.0~rc2',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Developer notes" href="devnotes.html" />
<link rel="prev" title="Automatized Upgrade Procedure for Ganeti" href="design-upgrade.html" />
</head>
<body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="devnotes.html" title="Developer notes"
accesskey="N">next</a></li>
<li class="right" >
<a href="design-upgrade.html" title="Automatized Upgrade Procedure for Ganeti"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="design-for-virtual-clusters-support">
<h1>Design for virtual clusters support<a class="headerlink" href="#design-for-virtual-clusters-support" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Created:</th><td class="field-body">2011-Oct-14</td>
</tr>
<tr class="field-even field"><th class="field-name">Status:</th><td class="field-body">Partial Implementation</td>
</tr>
<tr class="field-odd field"><th class="field-name">Ganeti-Version:</th><td class="field-body">2.7.0</td>
</tr>
</tbody>
</table>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h2>
<p>Currently there are two ways to test the Ganeti (including HTools) code
base:</p>
<ul class="simple">
<li>unittests, which run using mocks as normal user and test small bits of
the code</li>
<li>QA/burnin/live-test, which require actual hardware (either physical or
virtual) and will build an actual cluster, with one machine to one
node correspondence</li>
</ul>
<p>The difference in time between these two is significant:</p>
<ul class="simple">
<li>the unittests run in about 1-2 minutes</li>
<li>a so-called ‘quick’ QA (without burnin) runs in about an hour, and a
full QA could be double that time</li>
</ul>
<p>On one hand, the unittests have a clear advantage: quick to run, not
requiring many machines, but on the other hand QA is actually able to
run end-to-end tests (including HTools, for example).</p>
<p>Ideally, we would have an intermediate step between these two extremes:
be able to test most, if not all, of Ganeti’s functionality but without
requiring actual hardware, full machine ownership or root access.</p>
</div>
<div class="section" id="current-situation">
<h2>Current situation<a class="headerlink" href="#current-situation" title="Permalink to this headline">¶</a></h2>
<div class="section" id="ganeti">
<h3>Ganeti<a class="headerlink" href="#ganeti" title="Permalink to this headline">¶</a></h3>
<p>It is possible, given a manually built <code class="docutils literal"><span class="pre">config.data</span></code> and
<code class="docutils literal"><span class="pre">_autoconf.py</span></code>, to run the masterd under the current user as a
single-node cluster master. However, the node daemon and related
functionality (cluster initialisation, master failover, etc.) are not
directly runnable in this model.</p>
<p>Also, masterd only works as a master of a single node cluster, due to
our current “hostname” method of identifying nodes, which results in a
limit of maximum one node daemon per machine, unless we use multiple
name and IP aliases.</p>
</div>
<div class="section" id="htools">
<h3>HTools<a class="headerlink" href="#htools" title="Permalink to this headline">¶</a></h3>
<p>In HTools the situation is better, since it doesn’t have to deal with
actual machine management: all tools can use a custom LUXI path, and can
even load RAPI data from the filesystem (so the RAPI backend can be
tested), and both the ‘text’ backend for hbal/hspace and the input files
for hail are text-based, loaded from the file-system.</p>
</div>
</div>
<div class="section" id="proposed-changes">
<h2>Proposed changes<a class="headerlink" href="#proposed-changes" title="Permalink to this headline">¶</a></h2>
<p>The end-goal is to have full support for “virtual clusters”, i.e. be
able to run a “big” (hundreds of virtual nodes and towards thousands of
virtual instances) on a reasonably powerful, but single machine, under a
single user account and without any special privileges.</p>
<p>This would have significant advantages:</p>
<ul class="simple">
<li>being able to test end-to-end certain changes, without requiring a
complicated setup</li>
<li>better able to estimate Ganeti’s behaviour and performance as the
cluster size grows; this is something that we haven’t been able to
test reliably yet, and as such we still have not yet diagnosed
scaling problems</li>
<li>easier integration with external tools (and even with HTools)</li>
</ul>
<div class="section" id="masterd">
<h3><code class="docutils literal"><span class="pre">masterd</span></code><a class="headerlink" href="#masterd" title="Permalink to this headline">¶</a></h3>
<p>As described above, <code class="docutils literal"><span class="pre">masterd</span></code> already works reasonably well in a
virtual setup, as it won’t execute external programs and it shouldn’t
directly read files from the local filesystem (or at least not
virtualisation-related, as the master node can be a non-vm_capable
node).</p>
</div>
<div class="section" id="noded">
<h3><code class="docutils literal"><span class="pre">noded</span></code><a class="headerlink" href="#noded" title="Permalink to this headline">¶</a></h3>
<p>The node daemon executes many privileged operations, but they can be
split in a few general categories:</p>
<table border="1" class="docutils">
<colgroup>
<col width="20%" />
<col width="31%" />
<col width="49%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Category</th>
<th class="head">Description</th>
<th class="head">Solution</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>disk operations</td>
<td>Disk creation and
removal</td>
<td>Use only diskless or file-based
instances</td>
</tr>
<tr class="row-odd"><td>disk query</td>
<td>Node disk total/free,
used in node listing
and htools</td>
<td>Not supported currently, could use
file-based</td>
</tr>
<tr class="row-even"><td>hypervisor
operations</td>
<td>Instance start, stop
and query</td>
<td>Use the <em>fake</em> hypervisor</td>
</tr>
<tr class="row-odd"><td>instance
networking</td>
<td>Bridge existence query</td>
<td>Unprivileged operation, can be used
with an existing bridge at system
level or use NIC-less instances</td>
</tr>
<tr class="row-even"><td>instance OS
operations</td>
<td>OS add, OS rename,
export and import</td>
<td>Only used with non-diskless
instances; could work with custom OS
scripts that just <code class="docutils literal"><span class="pre">dd</span></code> without
mounting filesystems</td>
</tr>
<tr class="row-odd"><td>node networking</td>
<td>IP address management
(master ip), IP query,
etc.</td>
<td>Not supported; Ganeti will need to
work without a master IP; for the IP
query operations the test machine
would need externally-configured IPs</td>
</tr>
<tr class="row-even"><td>node add</td>
<td><ul class="first last simple">
<li></li>
</ul>
</td>
<td>SSH command must be adjusted</td>
</tr>
<tr class="row-odd"><td>node setup</td>
<td>ssh, /etc/hosts, so on</td>
<td>Can already be disabled from the
cluster config</td>
</tr>
<tr class="row-even"><td>master failover</td>
<td>start/stop the master
daemon</td>
<td>Doable (as long as we use a single
user), might get tricky w.r.t. paths
to executables</td>
</tr>
<tr class="row-odd"><td>file upload</td>
<td>Uploading of system
files, job queue files
and ganeti config</td>
<td>The only issue could be with system
files, which are not owned by the
current user; internal ganeti files
should be working fine</td>
</tr>
<tr class="row-even"><td>node oob</td>
<td>Out-of-band commands</td>
<td>Since these are user-defined, we can
mock them easily</td>
</tr>
<tr class="row-odd"><td>node OS
discovery</td>
<td>List the existing OSes
and their properties</td>
<td>No special privileges needed, so
works fine as-is</td>
</tr>
<tr class="row-even"><td>hooks</td>
<td>Running hooks for given
operations</td>
<td>No special privileges needed</td>
</tr>
<tr class="row-odd"><td>iallocator</td>
<td>Calling an iallocator
script</td>
<td>No special privileges needed</td>
</tr>
<tr class="row-even"><td>export/import</td>
<td>Exporting and importing
instances</td>
<td>When exporting/importing file-based
instances, this should work, as the
listening ports are dynamically
chosen</td>
</tr>
<tr class="row-odd"><td>hypervisor
validation</td>
<td>The validation of
hypervisor parameters</td>
<td>As long as the hypervisors don’t
call to privileged commands, it
should work</td>
</tr>
<tr class="row-even"><td>node powercycle</td>
<td>The ability to power
cycle a node remotely</td>
<td>Privileged, so not supported, but
anyway not very interesting for
testing</td>
</tr>
</tbody>
</table>
<p>It seems that much of the functionality works as is, or could work with
small adjustments, even in a non-privileged setup. The bigger problem is
the actual use of multiple node daemons per machine.</p>
<div class="section" id="multiple-noded-per-machine">
<h4>Multiple <code class="docutils literal"><span class="pre">noded</span></code> per machine<a class="headerlink" href="#multiple-noded-per-machine" title="Permalink to this headline">¶</a></h4>
<p>Currently Ganeti identifies node simply by their hostname. Since
changing this method would imply significant changes to tracking the
nodes, the proposal is to simply have as many IPs per the (single)
machine that is used for tests as nodes, and have each IP correspond to
a different name, and thus no changes are needed to the core RPC
library. Unfortunately this has the downside of requiring root rights
for setting up the extra IPs and hostnames.</p>
<p>An alternative option is to implement per-node IP/port support in Ganeti
(especially in the RPC layer), which would eliminate the root rights. We
expect that this will get implemented as a second step of this design,
but as the port is currently static will require changes in many places.</p>
<p>The only remaining problem is with sharing the <code class="docutils literal"><span class="pre">localstatedir</span></code>
structure (lib, run, log) amongst the daemons, for which we propose to
introduce an environment variable (<code class="docutils literal"><span class="pre">GANETI_ROOTDIR</span></code>) acting as a
prefix for essentially all paths. An environment variable is easier to
transport through several levels of programs (shell scripts, Python,
etc.) than a command line parameter. In Python code this prefix will be
applied to all paths in <code class="docutils literal"><span class="pre">constants.py</span></code>. Every virtual node will get
its own root directory. The rationale for this is two-fold:</p>
<ul class="simple">
<li>having two or more node daemons writing to the same directory might
introduce artificial scenarios not existent in real life; currently
noded either owns the entire <code class="docutils literal"><span class="pre">/var/lib/ganeti</span></code> directory or shares
it with masterd, but never with another noded</li>
<li>having separate directories allows cluster verify to check correctly
consistency of file upload operations; otherwise, as long as one node
daemon wrote a file successfully, the results from all others are
“lost”</li>
</ul>
<p>In case the use of an environment variable turns out to be too difficult
a compile-time prefix path could be used. This would then require one
Ganeti installation per virtual node, but it might be good enough.</p>
</div>
</div>
<div class="section" id="rapi">
<h3><code class="docutils literal"><span class="pre">rapi</span></code><a class="headerlink" href="#rapi" title="Permalink to this headline">¶</a></h3>
<p>The RAPI daemon is not privileged and furthermore we only need one per
cluster, so it presents no issues.</p>
</div>
<div class="section" id="confd">
<h3><code class="docutils literal"><span class="pre">confd</span></code><a class="headerlink" href="#confd" title="Permalink to this headline">¶</a></h3>
<p><code class="docutils literal"><span class="pre">confd</span></code> has somewhat the same issues as the node daemon regarding
multiple daemons per machine, but the per-address binding still works.</p>
</div>
<div class="section" id="ganeti-watcher">
<h3><code class="docutils literal"><span class="pre">ganeti-watcher</span></code><a class="headerlink" href="#ganeti-watcher" title="Permalink to this headline">¶</a></h3>
<p>Since the startup of daemons will be customised with per-IP binds, the
watcher either has to be modified to not activate the daemons, or the
start-stop tool has to take this into account. Due to watcher’s use of
the hostname, it’s recommended that the master node is set to the
machine hostname (also a requirement for the master daemon).</p>
</div>
<div class="section" id="cli-scripts">
<h3>CLI scripts<a class="headerlink" href="#cli-scripts" title="Permalink to this headline">¶</a></h3>
<p>As long as the master node is set to the machine hostname, these should
work fine.</p>
</div>
<div class="section" id="cluster-initialisation">
<h3>Cluster initialisation<a class="headerlink" href="#cluster-initialisation" title="Permalink to this headline">¶</a></h3>
<p>It could be possible that the cluster initialisation procedure is a bit
more involved (this was not tried yet). A script will be used to set up
all necessary IP addresses and hostnames, as well as creating the
initial directory structure. Building <code class="docutils literal"><span class="pre">config.data</span></code> manually should
not be necessary.</p>
</div>
</div>
<div class="section" id="needed-tools">
<h2>Needed tools<a class="headerlink" href="#needed-tools" title="Permalink to this headline">¶</a></h2>
<p>With the above investigation results in mind, the only thing we need
are:</p>
<ul class="simple">
<li>a tool to setup per-virtual node tree structure of <code class="docutils literal"><span class="pre">localstatedir</span></code>
(with the help of <code class="docutils literal"><span class="pre">ensure-dirs</span></code>) and setup correctly the extra
IP/hostnames</li>
<li>changes to the startup daemon tools to launch correctly the daemons
per virtual node</li>
<li>changes to <code class="docutils literal"><span class="pre">constants.py</span></code> to override the <code class="docutils literal"><span class="pre">localstatedir</span></code> path</li>
<li>documentation for running such a virtual cluster</li>
<li>and eventual small fixes to the node daemon backend functionality, to
better separate privileged and non-privileged code</li>
</ul>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Design for virtual clusters support</a><ul>
<li><a class="reference internal" href="#introduction">Introduction</a></li>
<li><a class="reference internal" href="#current-situation">Current situation</a><ul>
<li><a class="reference internal" href="#ganeti">Ganeti</a></li>
<li><a class="reference internal" href="#htools">HTools</a></li>
</ul>
</li>
<li><a class="reference internal" href="#proposed-changes">Proposed changes</a><ul>
<li><a class="reference internal" href="#masterd"><code class="docutils literal"><span class="pre">masterd</span></code></a></li>
<li><a class="reference internal" href="#noded"><code class="docutils literal"><span class="pre">noded</span></code></a><ul>
<li><a class="reference internal" href="#multiple-noded-per-machine">Multiple <code class="docutils literal"><span class="pre">noded</span></code> per machine</a></li>
</ul>
</li>
<li><a class="reference internal" href="#rapi"><code class="docutils literal"><span class="pre">rapi</span></code></a></li>
<li><a class="reference internal" href="#confd"><code class="docutils literal"><span class="pre">confd</span></code></a></li>
<li><a class="reference internal" href="#ganeti-watcher"><code class="docutils literal"><span class="pre">ganeti-watcher</span></code></a></li>
<li><a class="reference internal" href="#cli-scripts">CLI scripts</a></li>
<li><a class="reference internal" href="#cluster-initialisation">Cluster initialisation</a></li>
</ul>
</li>
<li><a class="reference internal" href="#needed-tools">Needed tools</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="design-upgrade.html"
title="previous chapter">Automatized Upgrade Procedure for Ganeti</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="devnotes.html"
title="next chapter">Developer notes</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/design-virtual-clusters.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="devnotes.html" title="Developer notes"
>next</a></li>
<li class="right" >
<a href="design-upgrade.html" title="Automatized Upgrade Procedure for Ganeti"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2018, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Google Inc..
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
</div>
</body>
</html>
|