botch-doc 0.16-2ubuntu2 / usr / share / doc / botch / wiki / Tools.html

<h1>botch-native</h1>

<p>The tools will be explained by going through what <code>native.sh</code> does, step-by-step. This way it might be possible to get a good overview of how the tools can work together in a meaningful manner.</p>

<p><code>botch-native</code> takes three or no positional arguments. If no argument is given, then <code>botch-download-pkgsrc</code> is run to download a default set of Packages and Sources files from http://snapshot.debian.org.</p>

<h2>botch-download-pkgsrc</h2>

<p><code>botch-download-pkgsrc</code> takes two or no positional arguments. If no argument is given, then a default set of Packages and Sources for amd64 is downloaded from http://snapshot.debian.org. Otherwise the two positional arguments are an architecture and a suite name which can be any of etch, lenny, squeeze, wheezy, jessie or sid. In addition the suite can be a year from 2007 to 2013 and will then download Debian sid from January 01 that year.</p>

<h2>botch-single-version</h2>

<p>If the <code>--latest</code> option is given to <code>botch-native</code>, then <code>botch-single-version</code> will be run on the input Packages file.</p>

<p><code>botch-single-version</code> takes two positional arguments which are the input and output Packages files. If either of them is a <code>-</code> then data is read from standard inptut or written to standard output, respectively. It goes through all packages in the given file and keeps for each architecture, the package with the highest version.</p>

<h2>botch-clean-repository</h2>

<p>If the <code>--clean</code> options is given to <code>botch-native</code>, then <code>botch-clean-repository</code> will be run on the input Packages and Sources files.</p>

<p><code>botch-clean-repository</code> removes all binary packages which:</p>

<ul>
<li>can't be installed</li>
<li>don't have a corresponding source package (<code>Architecture:all</code> binary packages are excluded from this check by default. This can be changed with the <code>--reqsrcarchall</code> option)</li>
</ul>

<p><code>botch-clean-repository</code> also removes all source packages which:</p>

<ul>
<li>can't be compiled</li>
<li>do not produce any binary packages (Source packages only producing <code>Architecture:all</code> binary packages are removed by default. This can be changed with the <code>--addsrcarchall</code> option)</li>
</ul>

<p><code>botch-clean-repository</code> does this removal iteratively until nothing gets removed anymore. The result is a self-contained repository (see [[Terminology]]).</p>

<p><code>botch-build-closure</code> also outputs a self-contained repository but while <code>botch-build-closure</code> results in roughly the smallest possible self-contained repository, <code>botch-clean-repository</code> will calculate the biggest possible self-contained repository.</p>

<h2>grep-dctrl</h2>

<p>After this initial cleanup, <code>grep-dctrl</code> is used to select the binary packages which have to be part of the minimal native build system. The default is:</p>

<pre><code>grep-dctrl -X \( \
    -FPackage build-essential --or \
    -FPackage apt --or \
    -FPackage debhelper --or \
    -FEssential yes \
\)
</code></pre>

<p>These packages alone are not enough to be installed together. To find a selection of packages which make it possible to install this selection (a co-installation set), <code>dose-deb-coinstall</code> will be run.</p>

<h2>dose-deb-coinstall</h2>

<p><code>dose-deb-coinstall</code> takes a set of foreground packages and a set of background packages. It outputs a set of packages which make it possible to install the foreground packages into the background packages.</p>

<h2>botch-bin2src</h2>

<p>Lots of operations like those carried out by <code>botch-single-version</code>, <code>botch-clean-repository</code> or <code>dose-deb-coinstall</code> only work on binary packages. But bootstrapping needs a list of source packages as well.</p>

<p><code>botch-bin2src</code> takes a list of binary packages and outputs the source packages that are needed to build them.</p>

<p>The analogous tool to <code>botch-bin2src</code> is <code>botch-src2bin</code>.</p>

<h2>botch-build-closure</h2>

<p>If the <code>--self-contained</code> option is given to <code>botch-native</code> then <code>botch-build-closure</code> will be run on the input Packages file.</p>

<p><code>botch-build-closure</code> is given a minimal selection of co-installable packages and then:</p>

<ul>
<li>adds all source packages that build the included binary packages</li>
<li>adds all binary packages needed to compile the included source packages</li>
</ul>

<p><code>botch-build-closure</code> does this addition iteratively until nothing gets added anymore.
The result is a self-contained repository (see [[Terminology]]).</p>

<p><code>botch-clean-repository</code> also outputs a self-contained repository but while <code>botch-clean-repository</code> results in the biggest possible self-contained repository, <code>botch-build-closure</code> will calculate the roughly smallest possible self-contained repository.</p>

<h2>dose-builddebcheck</h2>

<p>If the <code>--self-contained</code> option is given to <code>botch-native</code> then <code>dose-builddebcheck</code> will be run on the output of <code>botch-build-closure</code>.</p>

<p><code>dose-builddebcheck</code> takes a Packages and Sources file and checks if all source packages in Sources can be compiled with the binary packages in Packages. If not it outputs the reason by giving the arguments <code>--explain</code> and <code>--failures</code>.</p>

<h2>botch-build-fixpoint</h2>

<p>If the <code>--no-fixpoint</code> option is not given to <code>botch-native</code> then <code>botch-build-fixpoint</code> will be run the input Packages file.</p>

<p>Given a Packages and Sources file, as well as a set of available packages, <code>botch-build-fixpoint</code> finds which source packages are already compilable and adds the resulting binary packages to the set of available packages. This step is repeated until no more source packages can be built. It calcuates the build order of source packages without having to break any build dependencies. This step can help to increase the amount of initially available binary packages which in turn helps to simplify the dependency graph.</p>

<p>The <code>--output-order</code> option specifies a file into which to write the deduced build order.</p>

<p><code>botch-build-fixpoint</code> writes the list of compilable source packages to standard output.</p>

<h2>botch-src2bin</h2>

<p>Since <code>botch-build-fixpoint</code> outputs source packages, <code>botch-src2bin</code> has to be used to calculate a list of corresponding binary packages. Given a list of source packages, <code>botch-src2bin</code> outputs the list of binary packages that these source packages build.</p>

<h2>botch-packages-union</h2>

<p>The list of available binary packages is extended using the tool <code>botch-packages-union</code>. It takes two or more positional arguments. All arguments but the last are treated as input while the last is treated as the output. If the argument is a <code>-</code> then it will be read from standard input or written to standard output, respectively.</p>

<p><code>botch-packages-union</code> creates a union of all input files, writing the result to the output file. It merges multiple packages files into one, leaving no duplicates.</p>

<h2>botch-create-graph</h2>

<p>After all these preprocessing steps, <code>botch-create-graph</code> is run to calculate the dependency graph.</p>

<p><code>botch-create-graph</code> takes a Packages and Sources file as well as a list of Packages which are available (see [[Terminology]]). Available packages are those which were initially cross compiled (here, the calculated co-installation set) as well as <code>Architecture:all</code> binary packages. All binary packages which are marked as available are not connected to a source package in the dependency graph.</p>

<p>The output of <code>botch-create-graph</code> is given on standard output and redirected to <code>./buildgraph.xml</code> by <code>botch-native</code>.</p>

<p>The default dependency graph is created by arbitrarily choosing installation sets for binary package installation and source package compilation. The choice of installation sets can be influenced by passing the <code>--custom-is</code> option with a path to a file which lists dependencies that should not part of the calculated installation set.</p>

<p>By default, <code>Build-Depends-Indep</code> dependencies are not considered. This can be changed by passing the <code>--keep-indep</code> option.</p>

<p>Instead of generating an installation set and connecting vertices accordingly with edges, the <code>--strongtype</code> option can be used to calculate strong dependencies while the <code>--closuretype</code> option can be used to calculate dependency closures instead.</p>

<h2>botch-buildgraph2srcgraph</h2>

<p>After <code>buildgraph.xml</code> has been created <code>botch-buildgraph2srcgraph</code> is used to create <code>srcgraph.xml</code>. Build graphs and source graphs are explained in [[Terminology]].</p>

<h2>botch-annotate-strong</h2>

<p>If the <code>--strong</code> option is given to <code>botch-native</code> then <code>botch-annotate-strong</code> will be run on the source graph to mark edges as strong or not.</p>

<h2>botch-print-stats</h2>

<p>After all this processing, the build graph and source graph are analyzed and heuristics for how to make them acyclic are computed by <code>botch-print-stats</code>. Some of these heuristics for the source graph require it to be annotated with strong dependency information first.</p>

<p><code>botch-print-stats</code> runs all algorithms by default except for calculating strong bridges and strong articulation points which can be calculated by giving the <code>--sapsb</code> option.</p>

<p>The <code>--max-length</code> option indicates the maximum length for the cycles that shall be listed.</p>

<p>The <code>--max-length-fas</code> option indicates the maximum length for cycles that shall initially be used in the feedback arc set algorithm.
The list of available binary packages as passed with the <code>--available</code> option is needed to identify self-cycles in the source graph.</p>

<p>Listing dependencies which could possibly be marked as droppable by using build profiles or which could be moved to <code>Build-Depends-Indep</code> requires to specify the list of weak dependencies with the <code>--weak-deps</code> option.</p>

<p>The output of <code>botch-print-stats</code> is given in json format on standard output. It is redirected to <code>./stats.json</code> by <code>botch-native</code>.</p>

<h2>botch-stat-html</h2>

<p>Since the json file is not human readable it is converted into html format by <code>botch-stat-html</code>. The output is written to standard output and redirected to <code>stats.html</code> by <code>botch-native</code>.</p>

<h2>botch-fasofstats</h2>

<p>The existing droppable build dependencies might not have been enough to make the whole graph acyclic. The tool <code>botch-fasofstats</code> extracts a set of build dependencies which would make the graph acyclic from <code>./stats.json</code>. The output is written to standard output and redirected to <code>./remove.list</code> by <code>botch-native</code>.</p>

<h2>botch-extract-scc</h2>

<p>The tool <code>botch-extract-scc</code> extracts all strongly connected components from the build graph and saves them into separate files with a given prefix. The default prefix given by <code>botch-native</code> is <code>cyclic</code>. The output filename is structured as <code>${prefix}_${pkgname}_${vertnum}.xml</code> where <code>${prefix}</code> is the selected prefix, <code>${pkgname}</code> is the smallest package in the strongly connected component and <code>${vertnum}</code> is the number of vertices in the strongly connected component.</p>

<h2>botch-find-fvs</h2>

<p>Given a dependency graph and a list of known droppable build dependencies, <code>botch-find-fvs</code> will find a small selection of source packages (a feedback vertex set) which, if their droppable build dependencies were removed from the graph, made the whole graph acyclic.</p>

<p><code>botch-native</code> runs <code>botch-find-fvs</code> on all extracted strongly connected components and concatenates the resulting selection into the file <code>./feedback_vertex_set.list</code>.</p>

<h2>botch-profile-build-fvs</h2>

<p>After a feedback vertex set which would make the graph acyclic has been found, <code>botch-profile-build-fvs</code> is used to modify the build graph accordingly.</p>

<p><code>botch-profile-build-fvs</code> writes the resulting graph to standard output which is redirected into <code>./buildgraph_acyclic.xml</code> by <code>botch-native</code>.</p>

<h2>botch-partial-order</h2>

<p>Given an acyclic source graph, <code>botch-partial-order</code> will output a build order of source packages on standard output. The output is redirected to <code>./order2.lst</code> by <code>botch-native</code>.</p>

<p>All source packages which can be built in parallel are given in the same line.</p>

<h2>botch-graph-info</h2>

<p>The tool <code>botch-graph-info</code> takes a buildgraph or source graph as an argument and outputs one line with tab delimetered values. The values are:</p>

<ul>
<li>the filename</li>
<li>the graph type (build graph or source graph)</li>
<li>the number of vertices</li>
<li>the number of edges</li>
<li>the number of strongly connected components (if this number is zero, then the graph is acyclic)</li>
</ul>

<h1>Other tools</h1>

<h2>botch-collapse-srcgraph</h2>

<p>A build order can only be deduced from an acyclic graph. Sometimes, only a small remote cycle remains in the source graph which doesnt affect the major part of the graph. In that case, <code>botch-collapse-srcgraph</code> allows to collapse all remaining strongly connected components of the input source graph into one single vertex, thus making the input graph acyclic.</p>

<h2>botch-extract-neighborhood</h2>

<p>Rendering a big graph takes time and the graph might be too big to make a rendering of it useful. For this purpose, <code>botch-extract-neighborhood</code> allows to extract the neighborhood of a vertex. It takes three positional arguments: the nodename, the input graph and the output graph. The <code>--depth</code> argument allows to specify until what depth the neighborhood shall be extracted.</p>

<h2>botch-graph-difference</h2>

<p>For regression testing, <code>botch-graph-difference</code> allows to check whether two input graphs are equal or not. It does not calculate which parts are different but just outputs the first difference in case there is one.</p>

<h2>botch-graphml2dot</h2>

<p>The graphviz tools contain powerful visualization and processing tools but graphs have to be in the dot format. <code>botch-graphml2dot</code> converts an input build graph or source graph in graphml format into dot format for consumption by graphviz tools.</p>

<h2>botch-packages-difference</h2>

<p><code>botch-packages-difference</code> works analogous to <code>botch-packages-union</code> and <code>botch-packages-intersection</code>. Given three positional arguments it removes the packages in the second file from the first file and writes the result to the third.</p>

<h2>botch-packages-intersection</h2>

<p><code>botch-packages-intersection</code> works analogous to <code>botch-packages-difference</code> and <code>botch-packages-union</code>. Given three positional arguments it outputs all packages to the third file which are present in both, the first and the second.</p>