/usr/share/doc/stilts/sun256/pipes.html is in stilts-doc 3.1.2-2.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 | <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<link rel="stylesheet" type="text/css" href="sun-style.css">
<title>Table Pipelines</title>
</head>
<body>
<hr>
<a href="filterSteps.html">Next</a> <a href="outFormats.html">Previous</a> <a href="index.html">Up</a> <a href="index.html">Contents</a> <br> <b>Next: </b><a href="filterSteps.html">Processing Filters</a><br>
<b>Up: </b><a href="index.html">Top</a><br>
<b>Previous: </b><a href="outFormats.html">Output Formats</a><br>
<hr>
<hr>
<h2><a name="pipes">6 Table Pipelines</a></h2>
<p>Several of the tasks available in STILTS take one or more input tables,
do something or other with them, and produce one or more output tables.
This is a pretty obvious way to go about things, and in the most
straightforward case that's exactly what happens: you name one
or more input tables,
specify the processing parameters, and name an output table;
the task then reads the input tables from disk, does the processing
and writes the output table to disk.
</p>
<p>However, many of the tasks in STILTS allow you to do pre-processing
of the input tables before the main job, post-processing of the
output table after the main job, and to decide what happens to
the final tabular result, without any intermediate storage of the data.
Examples of the kind of pre-processing
you might want to do are to rearrange the columns so that they
have the right units for the main task, or replace 'magic' values
such as -999 with genuine blank values; the kind of post-processing
you might want to do is to sort the rows in the output table or
delete some of the columns you're not interested in.
As for the destination of the final table, you might want to
write it to disk, but equally you might not want to store it anywhere,
but only be interested in counting the number of rows, or seeing
the minima/maxima of a few of the columns, or you might want to
send it straight to TOPCAT or some other table viewing application
for interactive analysis.
</p>
<p>Clearly, you could achieve the same effect by running multiple
applications: preprocess your
original input tables to write intermediate files on disk,
run the main processing application which reads those files
from disk and writes a new output file,
run another application to postprocess the output file and
write a new final output file,
and finally do something with this such as counting the rows in it
or viewing it in TOPCAT.
However, by doing it all within a single task instead,
no intermediate results have to be stored,
and the whole sequence can be very much more efficient.
You can think of this (if it helps) like a Unix pipeline,
except what is being streamed from the start to the end of the
pipe is not bytes, but table metadata and data.
In most cases, the table data is streamed through the pipeline a
row at a time, meaning that the amount of memory required is small
(though in some cases, for instance row sorting and crossmatching,
this is not possible).
</p>
<p>Tasks which allow this pre/post-processing, or "filtering",
have parameters with names like "<code>cmd</code>" which you
use to specify processing steps.
Tasks with multiple input tables
(<a href="tmatch2.html"><code>tmatch2</code></a>,
<a href="tskymatch2.html"><code>tskymatch2</code></a>,
<a href="tcatn.html"><code>tcatn</code></a>,
<a href="tjoin.html"><code>tjoin</code></a>)
may have parameters named <code>icmd1</code>, <code>icmd2</code>, ...
for preprocessing the different input tables and
<code>ocmd</code> for postprocessing the output table.
<a href="tpipe.html"><code>tpipe</code></a> does nothing except
filtering, so there is no distinction between pre- and post-processing,
and its filter parameter is just named <code>cmd</code>.
<code>tpipe</code> additionally has a <code>script</code>
parameter which allows you to use a text file to write the
commands in, to prevent the command line getting too long.
In both cases there is a parameter named <code>omode</code>
which defines the "output mode", that is, what happens to the
post-processed output table that comes out of the end of the pipeline.
</p>
<p><a href="filterSteps.html">Section 6.1</a> lists the processing steps available,
and explains how to use them,
<a href="col-id.html">Section 6.2</a> and <a href="colid-list.html">Section 6.3</a> describe the syntax
used in some of these filter commands for specifying columns,
and <a href="outModes.html">Section 6.4</a> describes the available output modes.
See the examples in the
<a href="cmdUsage.html">command reference</a>,
and particularly the
<a href="tpipeExamples.html"><code>tpipe</code> examples</a>,
for some examples putting all this together.
</p>
<ul>
<li><a href="filterSteps.html">6.1 Processing Filters</a></li>
<ul>
<li><a href="addcol.html">6.1.1 <code>addcol</code></a></li>
<li><a href="addpixsample.html">6.1.2 <code>addpixsample</code></a></li>
<li><a href="addresolve.html">6.1.3 <code>addresolve</code></a></li>
<li><a href="addskycoords.html">6.1.4 <code>addskycoords</code></a></li>
<li><a href="assert.html">6.1.5 <code>assert</code></a></li>
<li><a href="badval.html">6.1.6 <code>badval</code></a></li>
<li><a href="cache.html">6.1.7 <code>cache</code></a></li>
<li><a href="check.html">6.1.8 <code>check</code></a></li>
<li><a href="clearparams.html">6.1.9 <code>clearparams</code></a></li>
<li><a href="colmeta.html">6.1.10 <code>colmeta</code></a></li>
<li><a href="delcols.html">6.1.11 <code>delcols</code></a></li>
<li><a href="every.html">6.1.12 <code>every</code></a></li>
<li><a href="explodeall.html">6.1.13 <code>explodeall</code></a></li>
<li><a href="explodecols.html">6.1.14 <code>explodecols</code></a></li>
<li><a href="fixcolnames.html">6.1.15 <code>fixcolnames</code></a></li>
<li><a href="head.html">6.1.16 <code>head</code></a></li>
<li><a href="keepcols.html">6.1.17 <code>keepcols</code></a></li>
<li><a href="meta.html">6.1.18 <code>meta</code></a></li>
<li><a href="progress.html">6.1.19 <code>progress</code></a></li>
<li><a href="random.html">6.1.20 <code>random</code></a></li>
<li><a href="randomview.html">6.1.21 <code>randomview</code></a></li>
<li><a href="repeat.html">6.1.22 <code>repeat</code></a></li>
<li><a href="replacecol.html">6.1.23 <code>replacecol</code></a></li>
<li><a href="replaceval.html">6.1.24 <code>replaceval</code></a></li>
<li><a href="rowrange.html">6.1.25 <code>rowrange</code></a></li>
<li><a href="select.html">6.1.26 <code>select</code></a></li>
<li><a href="seqview.html">6.1.27 <code>seqview</code></a></li>
<li><a href="setparam.html">6.1.28 <code>setparam</code></a></li>
<li><a href="sort.html">6.1.29 <code>sort</code></a></li>
<li><a href="sorthead.html">6.1.30 <code>sorthead</code></a></li>
<li><a href="stats.html">6.1.31 <code>stats</code></a></li>
<li><a href="tablename.html">6.1.32 <code>tablename</code></a></li>
<li><a href="tail.html">6.1.33 <code>tail</code></a></li>
<li><a href="transpose.html">6.1.34 <code>transpose</code></a></li>
<li><a href="uniq.html">6.1.35 <code>uniq</code></a></li>
</ul>
<li><a href="col-id.html">6.2 Specifying a Single Column</a></li>
<li><a href="colid-list.html">6.3 Specifying a List of Columns</a></li>
<li><a href="outModes.html">6.4 Output Modes</a></li>
<ul>
<li><a href="mode-cgi.html">6.4.1 <code>cgi</code></a></li>
<li><a href="mode-count.html">6.4.2 <code>count</code></a></li>
<li><a href="mode-discard.html">6.4.3 <code>discard</code></a></li>
<li><a href="mode-gui.html">6.4.4 <code>gui</code></a></li>
<li><a href="mode-meta.html">6.4.5 <code>meta</code></a></li>
<li><a href="mode-out.html">6.4.6 <code>out</code></a></li>
<li><a href="mode-samp.html">6.4.7 <code>samp</code></a></li>
<li><a href="mode-stats.html">6.4.8 <code>stats</code></a></li>
<li><a href="mode-topcat.html">6.4.9 <code>topcat</code></a></li>
<li><a href="mode-tosql.html">6.4.10 <code>tosql</code></a></li>
</ul>
</ul>
<hr><a href="filterSteps.html">Next</a> <a href="outFormats.html">Previous</a> <a href="index.html">Up</a> <a href="index.html">Contents</a> <br> <b>Next: </b><a href="filterSteps.html">Processing Filters</a><br>
<b>Up: </b><a href="index.html">Top</a><br>
<b>Previous: </b><a href="outFormats.html">Output Formats</a><br>
<hr><i>STILTS - Starlink Tables Infrastructure Library Tool Set<br>Starlink User Note256<br>STILTS web page:
<a href="http://www.starlink.ac.uk/stilts/">http://www.starlink.ac.uk/stilts/</a><br>Author email:
<a href="mailto:m.b.taylor@bristol.ac.uk">m.b.taylor@bristol.ac.uk</a><br>Mailing list:
<a href="mailto:topcat-user@jiscmail.ac.uk">topcat-user@jiscmail.ac.uk</a><br></i></body>
</html>
|