python-h5py-doc 2.6.0-1 / usr / share / doc / python-h5py-doc / html / mpi.html

This file is indexed.

/usr/share/doc/python-h5py-doc/html/mpi.html is in python-h5py-doc 2.6.0-1.

This file is owned by root:root, with mode 0o644.

The actual contents of the file can be viewed below. 

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">


<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>Parallel HDF5 &mdash; h5py 2.6.0 documentation</title>
    
    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    './',
        VERSION:     '2.6.0',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="top" title="h5py 2.6.0 documentation" href="index.html" />
    <link rel="next" title="Single Writer Multiple Reader (SWMR)" href="swmr.html" />
    <link rel="prev" title="Object and Region References" href="refs.html" /> 
  </head>
  <body role="document">
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="swmr.html" title="Single Writer Multiple Reader (SWMR)"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="refs.html" title="Object and Region References"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">h5py 2.6.0 documentation</a> &raquo;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="parallel-hdf5">
<span id="parallel"></span><h1>Parallel HDF5<a class="headerlink" href="#parallel-hdf5" title="Permalink to this headline"></a></h1>
<p>Starting with version 2.2.0, h5py includes support for Parallel HDF5.  This
is the &#8220;native&#8221; way to use HDF5 in a parallel computing environment.</p>
<div class="section" id="how-does-parallel-hdf5-work">
<h2>How does Parallel HDF5 work?<a class="headerlink" href="#how-does-parallel-hdf5-work" title="Permalink to this headline"></a></h2>
<p>Parallel HDF5 is a configuration of the HDF5 library which lets you share
open files across multiple parallel processes.  It uses the MPI (Message
Passing Interface) standard for interprocess communication.  Consequently,
when using Parallel HDF5 from Python, your application will also have to use
the MPI library.</p>
<p>This is accomplished through the <a class="reference external" href="http://mpi4py.scipy.org/">mpi4py</a> Python package, which provides
excellent, complete Python bindings for MPI.  Here&#8217;s an example
&#8220;Hello World&#8221; using <code class="docutils literal"><span class="pre">mpi4py</span></code>:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="kn">from</span> <span class="nn">mpi4py</span> <span class="kn">import</span> <span class="n">MPI</span>
<span class="gp">&gt;&gt;&gt; </span><span class="k">print</span> <span class="s2">&quot;Hello World (from process </span><span class="si">%d</span><span class="s2">)&quot;</span> <span class="o">%</span> <span class="n">MPI</span><span class="o">.</span><span class="n">COMM_WORLD</span><span class="o">.</span><span class="n">Get_rank</span><span class="p">()</span>
</pre></div>
</div>
<p>To run an MPI-based parallel program, use the <code class="docutils literal"><span class="pre">mpiexec</span></code> program to launch
several parallel instances of Python:</p>
<div class="highlight-python"><div class="highlight"><pre>$ mpiexec -n 4 python demo.py
Hello World (from process 1)
Hello World (from process 2)
Hello World (from process 3)
Hello World (from process 0)
</pre></div>
</div>
<p>The <code class="docutils literal"><span class="pre">mpi4py</span></code> package includes all kinds of mechanisms to share data between
processes, synchronize, etc.  It&#8217;s a different flavor of parallelism than,
say, threads or <code class="docutils literal"><span class="pre">multiprocessing</span></code>, but easy to get used to.</p>
<p>Check out the <a class="reference external" href="http://mpi4py.scipy.org/">mpi4py web site</a> for more information
and a great tutorial.</p>
</div>
<div class="section" id="building-against-parallel-hdf5">
<h2>Building against Parallel HDF5<a class="headerlink" href="#building-against-parallel-hdf5" title="Permalink to this headline"></a></h2>
<p>HDF5 must be built with at least the following options:</p>
<div class="highlight-python"><div class="highlight"><pre>$./configure --enable-parallel --enable-shared
</pre></div>
</div>
<p>Note that <code class="docutils literal"><span class="pre">--enable-shared</span></code> is required.</p>
<p>Often, a &#8220;parallel&#8221; version of HDF5 will be available through your package
manager.  You can check to see what build options were used by using the
program <code class="docutils literal"><span class="pre">h5cc</span></code>:</p>
<div class="highlight-python"><div class="highlight"><pre>$ h5cc -showconfig
</pre></div>
</div>
<p>Once you&#8217;ve got a Parallel-enabled build of HDF5, h5py has to be compiled in
&#8220;MPI mode&#8221;.  This is simple; set your default compiler to the <code class="docutils literal"><span class="pre">mpicc</span></code> wrapper
and build h5py with the <code class="docutils literal"><span class="pre">--mpi</span></code> option:</p>
<div class="highlight-python"><div class="highlight"><pre>$ export CC=mpicc
$ python setup.py configure --mpi [--hdf5=/path/to/parallel/hdf5]
$ python setup.py build
</pre></div>
</div>
</div>
<div class="section" id="using-parallel-hdf5-from-h5py">
<h2>Using Parallel HDF5 from h5py<a class="headerlink" href="#using-parallel-hdf5-from-h5py" title="Permalink to this headline"></a></h2>
<p>The parallel features of HDF5 are mostly transparent.  To open a file shared
across multiple processes, use the <code class="docutils literal"><span class="pre">mpio</span></code> file driver.  Here&#8217;s an example
program which opens a file, creates a single dataset and fills it with the
process ID:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">mpi4py</span> <span class="kn">import</span> <span class="n">MPI</span>
<span class="kn">import</span> <span class="nn">h5py</span>

<span class="n">rank</span> <span class="o">=</span> <span class="n">MPI</span><span class="o">.</span><span class="n">COMM_WORLD</span><span class="o">.</span><span class="n">rank</span>  <span class="c1"># The process ID (integer 0-3 for 4-process run)</span>

<span class="n">f</span> <span class="o">=</span> <span class="n">h5py</span><span class="o">.</span><span class="n">File</span><span class="p">(</span><span class="s1">&#39;parallel_test.hdf5&#39;</span><span class="p">,</span> <span class="s1">&#39;w&#39;</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="s1">&#39;mpio&#39;</span><span class="p">,</span> <span class="n">comm</span><span class="o">=</span><span class="n">MPI</span><span class="o">.</span><span class="n">COMM_WORLD</span><span class="p">)</span>

<span class="n">dset</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">create_dataset</span><span class="p">(</span><span class="s1">&#39;test&#39;</span><span class="p">,</span> <span class="p">(</span><span class="mi">4</span><span class="p">,),</span> <span class="n">dtype</span><span class="o">=</span><span class="s1">&#39;i&#39;</span><span class="p">)</span>
<span class="n">dset</span><span class="p">[</span><span class="n">rank</span><span class="p">]</span> <span class="o">=</span> <span class="n">rank</span>

<span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
</pre></div>
</div>
<p>Run the program:</p>
<div class="highlight-python"><div class="highlight"><pre>$ mpiexec -n 4 python demo2.py
</pre></div>
</div>
<p>Looking at the file with <code class="docutils literal"><span class="pre">h5dump</span></code>:</p>
<div class="highlight-python"><div class="highlight"><pre>$ h5dump parallel_test.hdf5
HDF5 &quot;parallel_test.hdf5&quot; {
GROUP &quot;/&quot; {
   DATASET &quot;test&quot; {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 4 ) / ( 4 ) }
      DATA {
      (0): 0, 1, 2, 3
      }
   }
}
}
</pre></div>
</div>
</div>
<div class="section" id="collective-versus-independent-operations">
<h2>Collective versus independent operations<a class="headerlink" href="#collective-versus-independent-operations" title="Permalink to this headline"></a></h2>
<p>MPI-based programs work by launching many instances of the Python interpreter,
each of which runs your script.  There are certain requirements imposed on
what each process can do.  Certain operations in HDF5, for example, anything
which modifies the file metadata, must be performed by all processes.  Other
operations, for example, writing data to a dataset, can be performed by some
processes and not others.</p>
<p>These two classes are called <em>collective</em> and <em>independent</em> operations.  Anything
which modifies the <em>structure</em> or metadata of a file must be done collectively.
For example, when creating a group, each process must participate:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">grp</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">create_group</span><span class="p">(</span><span class="s1">&#39;x&#39;</span><span class="p">)</span>  <span class="c1"># right</span>

<span class="gp">&gt;&gt;&gt; </span><span class="k">if</span> <span class="n">rank</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
<span class="gp">... </span>    <span class="n">grp</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">create_group</span><span class="p">(</span><span class="s1">&#39;x&#39;</span><span class="p">)</span>   <span class="c1"># wrong; all processes must do this</span>
</pre></div>
</div>
<p>On the other hand, writing data to a dataset can be done independently:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="k">if</span> <span class="n">rank</span> <span class="o">&gt;</span> <span class="mi">2</span><span class="p">:</span>
<span class="gp">... </span>    <span class="n">dset</span><span class="p">[</span><span class="n">rank</span><span class="p">]</span> <span class="o">=</span> <span class="mi">42</span>   <span class="c1"># this is fine</span>
</pre></div>
</div>
</div>
<div class="section" id="mpi-atomic-mode">
<h2>MPI atomic mode<a class="headerlink" href="#mpi-atomic-mode" title="Permalink to this headline"></a></h2>
<p>HDF5 versions 1.8.9+ support the MPI &#8220;atomic&#8221; file access mode, which trades
speed for more stringent consistency requirements.  Once you&#8217;ve opened a
file with the <code class="docutils literal"><span class="pre">mpio</span></code> driver, you can place it in atomic mode using the
settable <code class="docutils literal"><span class="pre">atomic</span></code> property:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">f</span> <span class="o">=</span> <span class="n">h5py</span><span class="o">.</span><span class="n">File</span><span class="p">(</span><span class="s1">&#39;parallel_test.hdf5&#39;</span><span class="p">,</span> <span class="s1">&#39;w&#39;</span><span class="p">,</span> <span class="n">driver</span><span class="o">=</span><span class="s1">&#39;mpio&#39;</span><span class="p">,</span> <span class="n">comm</span><span class="o">=</span><span class="n">MPI</span><span class="o">.</span><span class="n">COMM_WORLD</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">f</span><span class="o">.</span><span class="n">atomic</span> <span class="o">=</span> <span class="bp">True</span>
</pre></div>
</div>
</div>
<div class="section" id="more-information">
<h2>More information<a class="headerlink" href="#more-information" title="Permalink to this headline"></a></h2>
<p>Parallel HDF5 is a new feature in h5py.  If you have any questions, feel free to
ask on the mailing list (h5py at google groups).  We welcome bug reports,
enhancements and general inquiries.</p>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h3><a href="index.html">Table Of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Parallel HDF5</a><ul>
<li><a class="reference internal" href="#how-does-parallel-hdf5-work">How does Parallel HDF5 work?</a></li>
<li><a class="reference internal" href="#building-against-parallel-hdf5">Building against Parallel HDF5</a></li>
<li><a class="reference internal" href="#using-parallel-hdf5-from-h5py">Using Parallel HDF5 from h5py</a></li>
<li><a class="reference internal" href="#collective-versus-independent-operations">Collective versus independent operations</a></li>
<li><a class="reference internal" href="#mpi-atomic-mode">MPI atomic mode</a></li>
<li><a class="reference internal" href="#more-information">More information</a></li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="refs.html"
                        title="previous chapter">Object and Region References</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="swmr.html"
                        title="next chapter">Single Writer Multiple Reader (SWMR)</a></p>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="_sources/mpi.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <input type="text" name="q" />
      <input type="submit" value="Go" />
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
    <p class="searchtip" style="font-size: 90%">
    Enter search terms or a module, class or function name.
    </p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="swmr.html" title="Single Writer Multiple Reader (SWMR)"
             >next</a> |</li>
        <li class="right" >
          <a href="refs.html" title="Object and Region References"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">h5py 2.6.0 documentation</a> &raquo;</li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &copy; Copyright 2014, Andrew Collette and contributors.
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.3.5.
    </div>
  </body>
</html>

APT Browse - Built by Thomas Orozco.