/usr/share/doc/ubuntu-packaging-guide-html/auto-pkg-test.html is in ubuntu-packaging-guide-html 0.3.9.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>3. autopkgtest: Automatic testing for packages — Ubuntu Packaging Guide</title>
<link rel="shortcut icon" href="./_static/images/favicon.ico" type="image/x-icon" />
<link rel="stylesheet" href="./_static/reset.css" type="text/css" />
<link rel="stylesheet" href="./_static/960.css" type="text/css" />
<link rel="stylesheet" href="./_static/base.css" type="text/css" />
<link rel="stylesheet" href="./_static/home.css" type="text/css" />
<link rel="stylesheet" href="./_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="./_static/guide.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: '../',
VERSION: '0.3.9',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="./_static/jquery.js"></script>
<script type="text/javascript" src="./_static/underscore.js"></script>
<script type="text/javascript" src="./_static/doctools.js"></script>
<script type="text/javascript" src="./_static/main.js"></script>
<link rel="search" title="Search" href="./search.html" />
<link rel="top" title="Ubuntu Packaging Guide" href="./index.html" />
<link rel="next" title="4. Getting the Source" href="udd-getting-the-source.html" />
<link rel="prev" title="2. Basic Overview of the debian/ Directory" href="debian-dir-overview.html" />
</head>
<body class="home">
<a name="top"></a>
<div class="header-navigation">
<div>
<nav role="navigation">
<ul>
<li class="page_item current_page_item"><a title="Contents" href="index.html">Contents</a>
<li>
<form id="form-search" method="get" action="search.html">
<fieldset>
<input id="input-search" type="text" name="q" value="Search" />
</fieldset>
</form>
</li>
</ul>
</nav>
<a class="logo-ubuntu" href="http://packaging.ubuntu.com/">
<img src="./_static/images/logo-ubuntu.png" width="119" height="27" alt="Ubuntu logo" />
</a>
<a href="http://packaging.ubuntu.com/"><h2>Packaging Guide</h2></a>
</div>
</div>
<div class="header-content">
<div class="clearfix">
<div class="header-navigation-secondary">
<div>
<nav role="navigation">
<ul class="clearfix">
<li class="page_item"><a style="margin-right: 10px"
href="udd-getting-the-source.html" title="4. Getting the Source"
accesskey="N">next</a></li>
<li class="page_item"><a
href="debian-dir-overview.html" title="2. Basic Overview of the debian/ Directory"
accesskey="P">previous</a></li>
<li class="page_item"><a class="sub-nav-item" href="index.html">Ubuntu Packaging Guide »</a></li>
</ul>
</nav>
</div>
</div>
</div>
</div>
<div id="content" class="body container_12">
<div class="grid_12">
<!--<section id="main-section">-->
<div class="grid_9 alpha">
<div class="section" id="autopkgtest-automatic-testing-for-packages">
<h1>3. autopkgtest: Automatic testing for packages<a class="headerlink" href="#autopkgtest-automatic-testing-for-packages" title="Permalink to this headline">¶</a></h1>
<p>The <a class="reference external" href="http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/tree/doc/README.package-tests.rst">DEP 8 specification</a> defines how automatic testing can very easily be
integrated into packages. To integrate a test into a package, all you need to
do is:</p>
<ul class="simple">
<li>add a file called <code class="docutils literal"><span class="pre">debian/tests/control</span></code> which specifies the requirements
for the testbed,</li>
<li>add the tests in <code class="docutils literal"><span class="pre">debian/tests/</span></code>.</li>
</ul>
<div class="section" id="testbed-requirements">
<h2>3.1. Testbed requirements<a class="headerlink" href="#testbed-requirements" title="Permalink to this headline">¶</a></h2>
<p>In <code class="docutils literal"><span class="pre">debian/tests/control</span></code> you specify what to expect from the testbed. So
for example you list all the required packages for the tests, if the testbed
gets broken during the build or if <code class="docutils literal"><span class="pre">root</span></code> permissions are required. The
<a class="reference external" href="http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/tree/doc/README.package-tests.rst">DEP 8 specification</a> lists all available options.</p>
<p>Below we are having a look at the <code class="docutils literal"><span class="pre">glib2.0</span></code> source package. In a very
simple case the file would look like this:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">Tests</span><span class="p">:</span> <span class="n">build</span>
<span class="n">Depends</span><span class="p">:</span> <span class="n">libglib2</span><span class="o">.</span><span class="mi">0</span><span class="o">-</span><span class="n">dev</span><span class="p">,</span> <span class="n">build</span><span class="o">-</span><span class="n">essential</span>
</pre></div>
</div>
<p>For the test in <code class="docutils literal"><span class="pre">debian/tests/build</span></code> this would ensure that the packages
<code class="docutils literal"><span class="pre">libglib2.0-dev</span></code> and <code class="docutils literal"><span class="pre">build-essential</span></code> are installed.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">You can use <code class="docutils literal"><span class="pre">@</span></code> in the <code class="docutils literal"><span class="pre">Depends</span></code> line to indicate that you want
all the packages installed which are built by the source package in
question.</p>
</div>
</div>
<div class="section" id="the-actual-tests">
<h2>3.2. The actual tests<a class="headerlink" href="#the-actual-tests" title="Permalink to this headline">¶</a></h2>
<p>The accompanying test for the example above might be:</p>
<div class="highlight-sh"><div class="highlight"><pre><span></span><span class="ch">#!/bin/sh</span>
<span class="c1"># autopkgtest check: Build and run a program against glib, to verify that the</span>
<span class="c1"># headers and pkg-config file are installed correctly</span>
<span class="c1"># (C) 2012 Canonical Ltd.</span>
<span class="c1"># Author: Martin Pitt <martin.pitt@ubuntu.com></span>
<span class="nb">set</span> -e
<span class="nv">WORKDIR</span><span class="o">=</span><span class="k">$(</span>mktemp -d<span class="k">)</span>
<span class="nb">trap</span> <span class="s2">"rm -rf </span><span class="nv">$WORKDIR</span><span class="s2">"</span> <span class="m">0</span> INT QUIT ABRT PIPE TERM
<span class="nb">cd</span> <span class="nv">$WORKDIR</span>
cat <span class="s"><<EOF > glibtest.c</span>
<span class="s">#include <glib.h></span>
<span class="s">int main()</span>
<span class="s">{</span>
<span class="s"> g_assert_cmpint (g_strcmp0 (NULL, "hello"), ==, -1);</span>
<span class="s"> g_assert_cmpstr (g_find_program_in_path ("bash"), ==, "/bin/bash");</span>
<span class="s"> return 0;</span>
<span class="s">}</span>
<span class="s">EOF</span>
gcc -o glibtest glibtest.c <span class="sb">`</span>pkg-config --cflags --libs glib-2.0<span class="sb">`</span>
<span class="nb">echo</span> <span class="s2">"build: OK"</span>
<span class="o">[</span> -x glibtest <span class="o">]</span>
./glibtest
<span class="nb">echo</span> <span class="s2">"run: OK"</span>
</pre></div>
</div>
<p>Here a very simple piece of C code is written to a temporary directory. Then
this is compiled with system libraries (using flags and library paths as
provided by <cite>pkg-config</cite>). Then the compiled binary, which just exercises some
parts of core glib functionality, is run.</p>
<p>While this test is very small and simple, it covers quite a lot: that your -dev
package has all necessary dependencies, that your package installs working
pkg-config files, headers and libraries are put into the right place, or that
the compiler and linker work. This helps to uncover critical issues early on.</p>
</div>
<div class="section" id="executing-the-test">
<h2>3.3. Executing the test<a class="headerlink" href="#executing-the-test" title="Permalink to this headline">¶</a></h2>
<p>While the test script can be easily executed on its own, it is strongly
recommended to actually use <code class="docutils literal"><span class="pre">autopkgtest</span></code> from the <code class="docutils literal"><span class="pre">autopkgtest</span></code> package for
verifying that your test works; otherwise, if it fails in the Ubuntu Continuous
Integration (CI) system, it will not land in Ubuntu. This also avoids cluttering
your workstation with test packages or test configuration if the test does
something more intrusive than the simple example above.</p>
<p>The <a class="reference external" href="file:///usr/share/doc/autopkgtest/RREADME.running-tests.html">README.running-tests</a>
(<a class="reference external" href="http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/tree/doc/README.package-tests.rst">online version</a>) documentation explains all
available testbeds (schroot, LXD, QEMU, etc.) and the most common scenarios how
to run your tests with <code class="docutils literal"><span class="pre">autopkgtest</span></code>, e. g. with locally built binaries, locally
modified tests, etc.</p>
<p>The Ubuntu CI system uses the QEMU runner and runs the tests from the packages
in the archive, with <code class="docutils literal"><span class="pre">-proposed</span></code> enabled. To reproduce the exact same
environment, first install the necessary packages:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">sudo</span> <span class="n">apt</span><span class="o">-</span><span class="n">get</span> <span class="n">install</span> <span class="n">autopkgtest</span> <span class="n">qemu</span><span class="o">-</span><span class="n">system</span> <span class="n">qemu</span><span class="o">-</span><span class="n">utils</span>
</pre></div>
</div>
<p>Now build a testbed with:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">autopkgtest</span><span class="o">-</span><span class="n">buildvm</span><span class="o">-</span><span class="n">ubuntu</span><span class="o">-</span><span class="n">cloud</span> <span class="o">-</span><span class="n">v</span>
</pre></div>
</div>
<p>(Please see its manpage and <code class="docutils literal"><span class="pre">--help</span></code> output for selecting different releases,
architectures, output directory, or using proxies). This will build e. g.
<code class="docutils literal"><span class="pre">adt-trusty-amd64-cloud.img</span></code>.</p>
<p>Then run the tests of a source package like <code class="docutils literal"><span class="pre">libpng</span></code> in that QEMU image:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">autopkgtest</span> <span class="n">libpng</span> <span class="o">---</span> <span class="n">qemu</span> <span class="n">adt</span><span class="o">-</span><span class="n">trusty</span><span class="o">-</span><span class="n">amd64</span><span class="o">-</span><span class="n">cloud</span><span class="o">.</span><span class="n">img</span>
</pre></div>
</div>
<p>The Ubuntu CI system runs packages with only selected packages from
<code class="docutils literal"><span class="pre">-proposed</span></code> available (the package which caused the test to be run); to
enable that, run:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">autopkgtest</span> <span class="n">libpng</span> <span class="o">-</span><span class="n">U</span> <span class="o">--</span><span class="n">apt</span><span class="o">-</span><span class="n">pocket</span><span class="o">=</span><span class="n">proposed</span><span class="o">=</span><span class="n">src</span><span class="p">:</span><span class="n">foo</span> <span class="o">---</span> <span class="n">qemu</span> <span class="n">adt</span><span class="o">-</span><span class="n">release</span><span class="o">-</span><span class="n">amd64</span><span class="o">-</span><span class="n">cloud</span><span class="o">.</span><span class="n">img</span>
</pre></div>
</div>
<p>or to run with all packages from <code class="docutils literal"><span class="pre">-proposed</span></code>:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">autopkgtest</span> <span class="n">libpng</span> <span class="o">-</span><span class="n">U</span> <span class="o">--</span><span class="n">apt</span><span class="o">-</span><span class="n">pocket</span><span class="o">=</span><span class="n">proposed</span> <span class="o">---</span> <span class="n">qemu</span> <span class="n">adt</span><span class="o">-</span><span class="n">release</span><span class="o">-</span><span class="n">amd64</span><span class="o">-</span><span class="n">cloud</span><span class="o">.</span><span class="n">img</span>
</pre></div>
</div>
<p>The <code class="docutils literal"><span class="pre">autopkgtest</span></code> manpage has a lot more valuable information on other
testing options.</p>
</div>
<div class="section" id="further-examples">
<h2>3.4. Further examples<a class="headerlink" href="#further-examples" title="Permalink to this headline">¶</a></h2>
<p>This list is not comprehensive, but might help you get a better idea of how
automated tests are implemented and used in Ubuntu.</p>
<ul class="simple">
<li>The <a class="reference external" href="https://bazaar.launchpad.net/+branch/ubuntu/libxml2/files/head:/debian/tests/">libxml2 tests</a> are very similar. They also run a test-build of a
simple piece of C code and execute it.</li>
<li>The <a class="reference external" href="https://bazaar.launchpad.net/+branch/ubuntu/gtk+3.0/files/head:/debian/tests/">gtk+3.0 tests</a> also do a compile/link/run check in the “build” test.
There is an additional “python3-gi” test which verifies that the GTK
library can also be used through introspection.</li>
<li>In the <a class="reference external" href="https://bazaar.launchpad.net/+branch/ubiquity/files/head:/debian/tests/">ubiquity tests</a> the upstream test-suite is executed.</li>
<li>The <a class="reference external" href="https://bazaar.launchpad.net/+branch/ubuntu/gvfs/files/head:/debian/tests/">gvfs tests</a> have comprehensive testing of their functionality and
are very interesting because they emulate usage of CDs, Samba, DAV and
other bits.</li>
</ul>
</div>
<div class="section" id="ubuntu-infrastructure">
<h2>3.5. Ubuntu infrastructure<a class="headerlink" href="#ubuntu-infrastructure" title="Permalink to this headline">¶</a></h2>
<p>Packages which have <code class="docutils literal"><span class="pre">autopkgtest</span></code> enabled will have their tests run whenever
they get uploaded or any of their dependencies change. The output of
<a class="reference external" href="http://autopkgtest.ubuntu.com/">automatically run autopkgtest tests</a> can be viewed on the web and is
regularly updated.</p>
<p>Debian also uses <code class="docutils literal"><span class="pre">autopkgtest</span></code> to run package tests, although currently only
in schroots, so results may vary a bit. Results and logs can be seen on
<a class="reference external" href="http://ci.debian.net">http://ci.debian.net</a>. So please submit any test fixes or new tests to Debian as
well.</p>
</div>
<div class="section" id="getting-the-test-into-ubuntu">
<h2>3.6. Getting the test into Ubuntu<a class="headerlink" href="#getting-the-test-into-ubuntu" title="Permalink to this headline">¶</a></h2>
<p>The process of submitting an autopkgtest for a package is largely similar to
<a class="reference internal" href="fixing-a-bug.html"><span class="doc">fixing a bug in Ubuntu</span></a>. Essentially you simply:</p>
<ul class="simple">
<li>run <code class="docutils literal"><span class="pre">bzr</span> <span class="pre">branch</span> <span class="pre">ubuntu:<packagename></span></code>,</li>
<li>edit <code class="docutils literal"><span class="pre">debian/control</span></code> to enable the tests,</li>
<li>add the <code class="docutils literal"><span class="pre">debian/tests</span></code> directory,</li>
<li>write the <code class="docutils literal"><span class="pre">debian/tests/control</span></code> based on the <a class="reference external" href="http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/tree/doc/README.package-tests.rst">DEP 8 Specification</a>,</li>
<li>add your test case(s) to <code class="docutils literal"><span class="pre">debian/tests</span></code>,</li>
<li>commit your changes, push them to Launchpad, propose a merge and get it
reviewed just like any other improvement in a source package.</li>
</ul>
</div>
<div class="section" id="what-you-can-do">
<h2>3.7. What you can do<a class="headerlink" href="#what-you-can-do" title="Permalink to this headline">¶</a></h2>
<p>The Ubuntu Engineering team put together a <a class="reference external" href="https://wiki.ubuntu.com/QATeam/RequiredTests">list of required test-cases</a>,
where packages which need tests are put into different categories. Here you
can find examples of these tests and easily assign them to yourself.</p>
<p>If you should run into any problems, you can join the <a class="reference external" href="http://webchat.freenode.net/?channels=ubuntu-quality">#ubuntu-quality IRC
channel</a> to get in touch with developers who can help you.</p>
</div>
</div>
<div class="divide"></div>
</div>
<div id="sidebar" class="grid_3 omega">
<div class="container-tweet">
<h3>Table Of Contents</h3>
<div class="toc">
<ul>
<li><a class="reference internal" href="#">3. autopkgtest: Automatic testing for packages</a><ul>
<li><a class="reference internal" href="#testbed-requirements">3.1. Testbed requirements</a></li>
<li><a class="reference internal" href="#the-actual-tests">3.2. The actual tests</a></li>
<li><a class="reference internal" href="#executing-the-test">3.3. Executing the test</a></li>
<li><a class="reference internal" href="#further-examples">3.4. Further examples</a></li>
<li><a class="reference internal" href="#ubuntu-infrastructure">3.5. Ubuntu infrastructure</a></li>
<li><a class="reference internal" href="#getting-the-test-into-ubuntu">3.6. Getting the test into Ubuntu</a></li>
<li><a class="reference internal" href="#what-you-can-do">3.7. What you can do</a></li>
</ul>
</li>
</ul>
</div>
<div class="browse-guide">
<h3>Browse The Guide:</h3>
<ul>
<li class="prev">
<a href="debian-dir-overview.html"
title="Previous topic: 2. Basic Overview of the <code class="docutils literal"><span class="pre">debian/</span></code> Directory">Go Previous</a>
</li>
<li class="center">
<a title="Back to Index" href="index.html">Index Guide</a>
</li>
<li class="next">
<a href="udd-getting-the-source.html"
title="Next topic: 4. Getting the Source">Go Next</a>
</li>
</ul>
<span>This Page:</span> <a href="./_sources/auto-pkg-test.rst.txt"
rel="nofollow">Show Source</a>
</div>
</div>
<div id="back_top"><a href="#top">Back to Top</a></div>
</div>
<!--</section>-->
</div>
</div>
<div class="shadow"></div>
<footer>
<div>
Version: 0.3.9.
<a href="https://bugs.launchpad.net/ubuntu-packaging-guide">Report bugs</a> or
<a href="https://code.launchpad.net/~ubuntu-packaging-guide-team/ubuntu-packaging-guide/trunk">grab the source code</a> from Launchpad.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.5.6.
<br />
© Copyright 2010-2017, Ubuntu Developers, Creative Commons Attribution-ShareAlike 3.0.
<a rel="license" href="http://creativecommons.org/licenses/by-sa/3.0/">
Creative Commons Attribution-ShareAlike 3.0 Unported License</a>.
<a rel="license" href="http://creativecommons.org/licenses/by-sa/3.0/">
<img alt="Creative Commons License" style="border-width:0"
src="./_static/images/cc-by-sa.png" /></a>
<br />
<a href="http://people.ubuntu.com/~mitya57/ubuntu-packaging-guide-readme.html#translating">Help translate</a> or
<a href="./_static/translators.html">view the list of translators</a>.
</div>
</footer>
</body>
</html>
|