This file is indexed.

/usr/share/doc/ganeti-doc/html/design-virtual-clusters.html is in ganeti-doc 2.15.2-3.

This file is owned by root:root, with mode 0o644.

The actual contents of the file can be viewed below.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">


<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>Design for virtual clusters support &mdash; Ganeti 2.15.2 documentation</title>
    
    <link rel="stylesheet" href="_static/style.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    './',
        VERSION:     '2.15.2',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="top" title="Ganeti 2.15.2 documentation" href="index.html" />
    <link rel="next" title="Developer notes" href="devnotes.html" />
    <link rel="prev" title="Automatized Upgrade Procedure for Ganeti" href="design-upgrade.html" /> 
  </head>
  <body role="document">
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="devnotes.html" title="Developer notes"
             accesskey="N">next</a></li>
        <li class="right" >
          <a href="design-upgrade.html" title="Automatized Upgrade Procedure for Ganeti"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.15.2 documentation</a> &raquo;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="design-for-virtual-clusters-support">
<h1>Design for virtual clusters support<a class="headerlink" href="#design-for-virtual-clusters-support" title="Permalink to this headline"></a></h1>
<div class="section" id="introduction">
<h2>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline"></a></h2>
<p>Currently there are two ways to test the Ganeti (including HTools) code
base:</p>
<ul class="simple">
<li>unittests, which run using mocks as normal user and test small bits of
the code</li>
<li>QA/burnin/live-test, which require actual hardware (either physical or
virtual) and will build an actual cluster, with one machine to one
node correspondence</li>
</ul>
<p>The difference in time between these two is significant:</p>
<ul class="simple">
<li>the unittests run in about 1-2 minutes</li>
<li>a so-called ‘quick’ QA (without burnin) runs in about an hour, and a
full QA could be double that time</li>
</ul>
<p>On one hand, the unittests have a clear advantage: quick to run, not
requiring many machines, but on the other hand QA is actually able to
run end-to-end tests (including HTools, for example).</p>
<p>Ideally, we would have an intermediate step between these two extremes:
be able to test most, if not all, of Ganeti&#8217;s functionality but without
requiring actual hardware, full machine ownership or root access.</p>
</div>
<div class="section" id="current-situation">
<h2>Current situation<a class="headerlink" href="#current-situation" title="Permalink to this headline"></a></h2>
<div class="section" id="ganeti">
<h3>Ganeti<a class="headerlink" href="#ganeti" title="Permalink to this headline"></a></h3>
<p>It is possible, given a manually built <code class="docutils literal"><span class="pre">config.data</span></code> and
<code class="docutils literal"><span class="pre">_autoconf.py</span></code>, to run the masterd under the current user as a
single-node cluster master. However, the node daemon and related
functionality (cluster initialisation, master failover, etc.) are not
directly runnable in this model.</p>
<p>Also, masterd only works as a master of a single node cluster, due to
our current “hostname” method of identifying nodes, which results in a
limit of maximum one node daemon per machine, unless we use multiple
name and IP aliases.</p>
</div>
<div class="section" id="htools">
<h3>HTools<a class="headerlink" href="#htools" title="Permalink to this headline"></a></h3>
<p>In HTools the situation is better, since it doesn&#8217;t have to deal with
actual machine management: all tools can use a custom LUXI path, and can
even load RAPI data from the filesystem (so the RAPI backend can be
tested), and both the ‘text’ backend for hbal/hspace and the input files
for hail are text-based, loaded from the file-system.</p>
</div>
</div>
<div class="section" id="proposed-changes">
<h2>Proposed changes<a class="headerlink" href="#proposed-changes" title="Permalink to this headline"></a></h2>
<p>The end-goal is to have full support for “virtual clusters”, i.e. be
able to run a “big” (hundreds of virtual nodes and towards thousands of
virtual instances) on a reasonably powerful, but single machine, under a
single user account and without any special privileges.</p>
<p>This would have significant advantages:</p>
<ul class="simple">
<li>being able to test end-to-end certain changes, without requiring a
complicated setup</li>
<li>better able to estimate Ganeti&#8217;s behaviour and performance as the
cluster size grows; this is something that we haven&#8217;t been able to
test reliably yet, and as such we still have not yet diagnosed
scaling problems</li>
<li>easier integration with external tools (and even with HTools)</li>
</ul>
<div class="section" id="masterd">
<h3><code class="docutils literal"><span class="pre">masterd</span></code><a class="headerlink" href="#masterd" title="Permalink to this headline"></a></h3>
<p>As described above, <code class="docutils literal"><span class="pre">masterd</span></code> already works reasonably well in a
virtual setup, as it won&#8217;t execute external programs and it shouldn&#8217;t
directly read files from the local filesystem (or at least not
virtualisation-related, as the master node can be a non-vm_capable
node).</p>
</div>
<div class="section" id="noded">
<h3><code class="docutils literal"><span class="pre">noded</span></code><a class="headerlink" href="#noded" title="Permalink to this headline"></a></h3>
<p>The node daemon executes many privileged operations, but they can be
split in a few general categories:</p>
<table border="1" class="docutils">
<colgroup>
<col width="20%" />
<col width="31%" />
<col width="49%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Category</th>
<th class="head">Description</th>
<th class="head">Solution</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>disk operations</td>
<td>Disk creation and
removal</td>
<td>Use only diskless or file-based
instances</td>
</tr>
<tr class="row-odd"><td>disk query</td>
<td>Node disk total/free,
used in node listing
and htools</td>
<td>Not supported currently, could use
file-based</td>
</tr>
<tr class="row-even"><td>hypervisor
operations</td>
<td>Instance start, stop
and query</td>
<td>Use the <em>fake</em> hypervisor</td>
</tr>
<tr class="row-odd"><td>instance
networking</td>
<td>Bridge existence query</td>
<td>Unprivileged operation, can be used
with an existing bridge at system
level or use NIC-less instances</td>
</tr>
<tr class="row-even"><td>instance OS
operations</td>
<td>OS add, OS rename,
export and import</td>
<td>Only used with non-diskless
instances; could work with custom OS
scripts that just <code class="docutils literal"><span class="pre">dd</span></code> without
mounting filesystems</td>
</tr>
<tr class="row-odd"><td>node networking</td>
<td>IP address management
(master ip), IP query,
etc.</td>
<td>Not supported; Ganeti will need to
work without a master IP; for the IP
query operations the test machine
would need externally-configured IPs</td>
</tr>
<tr class="row-even"><td>node add</td>
<td><ul class="first last simple">
<li></li>
</ul>
</td>
<td>SSH command must be adjusted</td>
</tr>
<tr class="row-odd"><td>node setup</td>
<td>ssh, /etc/hosts, so on</td>
<td>Can already be disabled from the
cluster config</td>
</tr>
<tr class="row-even"><td>master failover</td>
<td>start/stop the master
daemon</td>
<td>Doable (as long as we use a single
user), might get tricky w.r.t. paths
to executables</td>
</tr>
<tr class="row-odd"><td>file upload</td>
<td>Uploading of system
files, job queue files
and ganeti config</td>
<td>The only issue could be with system
files, which are not owned by the
current user; internal ganeti files
should be working fine</td>
</tr>
<tr class="row-even"><td>node oob</td>
<td>Out-of-band commands</td>
<td>Since these are user-defined, we can
mock them easily</td>
</tr>
<tr class="row-odd"><td>node OS
discovery</td>
<td>List the existing OSes
and their properties</td>
<td>No special privileges needed, so
works fine as-is</td>
</tr>
<tr class="row-even"><td>hooks</td>
<td>Running hooks for given
operations</td>
<td>No special privileges needed</td>
</tr>
<tr class="row-odd"><td>iallocator</td>
<td>Calling an iallocator
script</td>
<td>No special privileges needed</td>
</tr>
<tr class="row-even"><td>export/import</td>
<td>Exporting and importing
instances</td>
<td>When exporting/importing file-based
instances, this should work, as the
listening ports are dynamically
chosen</td>
</tr>
<tr class="row-odd"><td>hypervisor
validation</td>
<td>The validation of
hypervisor parameters</td>
<td>As long as the hypervisors don&#8217;t
call to privileged commands, it
should work</td>
</tr>
<tr class="row-even"><td>node powercycle</td>
<td>The ability to power
cycle a node remotely</td>
<td>Privileged, so not supported, but
anyway not very interesting for
testing</td>
</tr>
</tbody>
</table>
<p>It seems that much of the functionality works as is, or could work with
small adjustments, even in a non-privileged setup. The bigger problem is
the actual use of multiple node daemons per machine.</p>
<div class="section" id="multiple-noded-per-machine">
<h4>Multiple <code class="docutils literal"><span class="pre">noded</span></code> per machine<a class="headerlink" href="#multiple-noded-per-machine" title="Permalink to this headline"></a></h4>
<p>Currently Ganeti identifies node simply by their hostname. Since
changing this method would imply significant changes to tracking the
nodes, the proposal is to simply have as many IPs per the (single)
machine that is used for tests as nodes, and have each IP correspond to
a different name, and thus no changes are needed to the core RPC
library. Unfortunately this has the downside of requiring root rights
for setting up the extra IPs and hostnames.</p>
<p>An alternative option is to implement per-node IP/port support in Ganeti
(especially in the RPC layer), which would eliminate the root rights. We
expect that this will get implemented as a second step of this design,
but as the port is currently static will require changes in many places.</p>
<p>The only remaining problem is with sharing the <code class="docutils literal"><span class="pre">localstatedir</span></code>
structure (lib, run, log) amongst the daemons, for which we propose to
introduce an environment variable (<code class="docutils literal"><span class="pre">GANETI_ROOTDIR</span></code>) acting as a
prefix for essentially all paths. An environment variable is easier to
transport through several levels of programs (shell scripts, Python,
etc.) than a command line parameter. In Python code this prefix will be
applied to all paths in <code class="docutils literal"><span class="pre">constants.py</span></code>. Every virtual node will get
its own root directory. The rationale for this is two-fold:</p>
<ul class="simple">
<li>having two or more node daemons writing to the same directory might
introduce artificial scenarios not existent in real life; currently
noded either owns the entire <code class="docutils literal"><span class="pre">/var/lib/ganeti</span></code> directory or shares
it with masterd, but never with another noded</li>
<li>having separate directories allows cluster verify to check correctly
consistency of file upload operations; otherwise, as long as one node
daemon wrote a file successfully, the results from all others are
“lost”</li>
</ul>
<p>In case the use of an environment variable turns out to be too difficult
a compile-time prefix path could be used. This would then require one
Ganeti installation per virtual node, but it might be good enough.</p>
</div>
</div>
<div class="section" id="rapi">
<h3><code class="docutils literal"><span class="pre">rapi</span></code><a class="headerlink" href="#rapi" title="Permalink to this headline"></a></h3>
<p>The RAPI daemon is not privileged and furthermore we only need one per
cluster, so it presents no issues.</p>
</div>
<div class="section" id="confd">
<h3><code class="docutils literal"><span class="pre">confd</span></code><a class="headerlink" href="#confd" title="Permalink to this headline"></a></h3>
<p><code class="docutils literal"><span class="pre">confd</span></code> has somewhat the same issues as the node daemon regarding
multiple daemons per machine, but the per-address binding still works.</p>
</div>
<div class="section" id="ganeti-watcher">
<h3><code class="docutils literal"><span class="pre">ganeti-watcher</span></code><a class="headerlink" href="#ganeti-watcher" title="Permalink to this headline"></a></h3>
<p>Since the startup of daemons will be customised with per-IP binds, the
watcher either has to be modified to not activate the daemons, or the
start-stop tool has to take this into account. Due to watcher&#8217;s use of
the hostname, it&#8217;s recommended that the master node is set to the
machine hostname (also a requirement for the master daemon).</p>
</div>
<div class="section" id="cli-scripts">
<h3>CLI scripts<a class="headerlink" href="#cli-scripts" title="Permalink to this headline"></a></h3>
<p>As long as the master node is set to the machine hostname, these should
work fine.</p>
</div>
<div class="section" id="cluster-initialisation">
<h3>Cluster initialisation<a class="headerlink" href="#cluster-initialisation" title="Permalink to this headline"></a></h3>
<p>It could be possible that the cluster initialisation procedure is a bit
more involved (this was not tried yet). A script will be used to set up
all necessary IP addresses and hostnames, as well as creating the
initial directory structure. Building <code class="docutils literal"><span class="pre">config.data</span></code> manually should
not be necessary.</p>
</div>
</div>
<div class="section" id="needed-tools">
<h2>Needed tools<a class="headerlink" href="#needed-tools" title="Permalink to this headline"></a></h2>
<p>With the above investigation results in mind, the only thing we need
are:</p>
<ul class="simple">
<li>a tool to setup per-virtual node tree structure of <code class="docutils literal"><span class="pre">localstatedir</span></code>
(with the help of <code class="docutils literal"><span class="pre">ensure-dirs</span></code>) and setup correctly the extra
IP/hostnames</li>
<li>changes to the startup daemon tools to launch correctly the daemons
per virtual node</li>
<li>changes to <code class="docutils literal"><span class="pre">constants.py</span></code> to override the <code class="docutils literal"><span class="pre">localstatedir</span></code> path</li>
<li>documentation for running such a virtual cluster</li>
<li>and eventual small fixes to the node daemon backend functionality, to
better separate privileged and non-privileged code</li>
</ul>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h3><a href="index.html">Table Of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Design for virtual clusters support</a><ul>
<li><a class="reference internal" href="#introduction">Introduction</a></li>
<li><a class="reference internal" href="#current-situation">Current situation</a><ul>
<li><a class="reference internal" href="#ganeti">Ganeti</a></li>
<li><a class="reference internal" href="#htools">HTools</a></li>
</ul>
</li>
<li><a class="reference internal" href="#proposed-changes">Proposed changes</a><ul>
<li><a class="reference internal" href="#masterd"><code class="docutils literal"><span class="pre">masterd</span></code></a></li>
<li><a class="reference internal" href="#noded"><code class="docutils literal"><span class="pre">noded</span></code></a><ul>
<li><a class="reference internal" href="#multiple-noded-per-machine">Multiple <code class="docutils literal"><span class="pre">noded</span></code> per machine</a></li>
</ul>
</li>
<li><a class="reference internal" href="#rapi"><code class="docutils literal"><span class="pre">rapi</span></code></a></li>
<li><a class="reference internal" href="#confd"><code class="docutils literal"><span class="pre">confd</span></code></a></li>
<li><a class="reference internal" href="#ganeti-watcher"><code class="docutils literal"><span class="pre">ganeti-watcher</span></code></a></li>
<li><a class="reference internal" href="#cli-scripts">CLI scripts</a></li>
<li><a class="reference internal" href="#cluster-initialisation">Cluster initialisation</a></li>
</ul>
</li>
<li><a class="reference internal" href="#needed-tools">Needed tools</a></li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="design-upgrade.html"
                        title="previous chapter">Automatized Upgrade Procedure for Ganeti</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="devnotes.html"
                        title="next chapter">Developer notes</a></p>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="_sources/design-virtual-clusters.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <input type="text" name="q" />
      <input type="submit" value="Go" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    <p class="searchtip" style="font-size: 90%">
    Enter search terms or a module, class or function name.
    </p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="devnotes.html" title="Developer notes"
             >next</a></li>
        <li class="right" >
          <a href="design-upgrade.html" title="Automatized Upgrade Procedure for Ganeti"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.15.2 documentation</a> &raquo;</li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &copy; Copyright 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013 Google Inc..
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.6.
    </div>
  </body>
</html>