/usr/share/doc/python-patsy-doc/html/R-comparison.html is in python-patsy-doc 0.3.0-3.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Differences between R and Patsy formulas — patsy 0.3.0 documentation</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<link rel="stylesheet" href="_static/facebox.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '0.3.0',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<script type="text/javascript" src="_static/show-code.js"></script>
<script type="text/javascript" src="_static/facebox.js"></script>
<link rel="top" title="patsy 0.3.0 documentation" href="index.html" />
<link rel="next" title="Python 2 versus Python 3" href="py2-versus-py3.html" />
<link rel="prev" title="Using Patsy in your library" href="library-developers.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="py2-versus-py3.html" title="Python 2 versus Python 3"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="library-developers.html" title="Using Patsy in your library"
accesskey="P">previous</a> |</li>
<li><a href="index.html">patsy 0.3.0 documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="differences-between-r-and-patsy-formulas">
<span id="r-comparison"></span><h1>Differences between R and Patsy formulas<a class="headerlink" href="#differences-between-r-and-patsy-formulas" title="Permalink to this headline">ΒΆ</a></h1>
<p>Patsy has a very high degree of compatibility with R. Almost any
formula you would use in R will also work in Patsy – with a few
caveats.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">All R quirks described herein were last verified with R
2.15.0.</p>
</div>
<p>Differences from R:</p>
<ul>
<li><p class="first">Most obviously, we both support using arbitrary code to perform
variable transformations, but in Patsy this code is written in
Python, not R.</p>
</li>
<li><p class="first">Patsy has no <tt class="docutils literal"><span class="pre">%in%</span></tt>. In R, <tt class="docutils literal"><span class="pre">a</span> <span class="pre">%in%</span> <span class="pre">b</span></tt> is identical to
<tt class="docutils literal"><span class="pre">b:a</span></tt>. Patsy only supports the <tt class="docutils literal"><span class="pre">b:a</span></tt> version of this syntax.</p>
</li>
<li><p class="first">In Patsy, only <tt class="docutils literal"><span class="pre">**</span></tt> can be used for exponentiation. In R, both
<tt class="docutils literal"><span class="pre">^</span></tt> and <tt class="docutils literal"><span class="pre">**</span></tt> can be used for exponentiation, i.e., you can write
either <tt class="docutils literal"><span class="pre">(a</span> <span class="pre">+</span> <span class="pre">b)^2</span></tt> or <tt class="docutils literal"><span class="pre">(a</span> <span class="pre">+</span> <span class="pre">b)**2</span></tt>. In Patsy (as in Python
generally), only <tt class="docutils literal"><span class="pre">**</span></tt> indicates exponentiation; <tt class="docutils literal"><span class="pre">^</span></tt> is ignored
by the parser (and if present, will be interpreted as a call to the
Python binary XOR operator).</p>
</li>
<li><p class="first">In Patsy, the left-hand side of a formula uses the same
evaluation rules as the right-hand side. In R, the left hand side is
treated as R code, so a formula like <tt class="docutils literal"><span class="pre">y1</span> <span class="pre">+</span> <span class="pre">y2</span> <span class="pre">~</span> <span class="pre">x1</span> <span class="pre">+</span> <span class="pre">x2</span></tt> actually
regresses the <em>sum</em> of <tt class="docutils literal"><span class="pre">y1</span></tt> and <tt class="docutils literal"><span class="pre">y2</span></tt> onto the <em>set of
predictors</em> <tt class="docutils literal"><span class="pre">x1</span></tt> and <tt class="docutils literal"><span class="pre">x2</span></tt>. In Patsy, the only difference
between the left-hand side and the right-hand side is that there is
no automatic intercept added to the left-hand side. (In this regard
Patsy is similar to the R enhanced formula package <a class="reference external" href="http://cran.r-project.org/web/packages/Formula/index.html">Formula</a>.)</p>
</li>
<li><p class="first">Patsy produces a different column ordering for formulas involving
numeric predictors. In R, there are two rules for term ordering:
first, lower-order interactions are sorted before higher-order
interactions, and second, interactions of the same order are listed
in whatever order they appeared in the formula. In Patsy, we add
another rule: terms are first grouped together based on which
numeric factors they include. Then within each group, we use the
same ordering as R.</p>
</li>
<li><p class="first">Patsy has more rigorous handling of the presence or absence of
the intercept term. In R, the rules for when deciding whether to
include an intercept are somewhat idiosyncratic and can ignore
things like parentheses. To understand the difference, first
consider the formula <tt class="docutils literal"><span class="pre">a</span> <span class="pre">+</span> <span class="pre">(b</span> <span class="pre">-</span> <span class="pre">a)</span></tt>. In both Patsy and R, we
first evaluate the <tt class="docutils literal"><span class="pre">(b</span> <span class="pre">-</span> <span class="pre">a)</span></tt> part; since there is no <tt class="docutils literal"><span class="pre">a</span></tt> term to
remove, this simplifies to just <tt class="docutils literal"><span class="pre">b</span></tt>. We then evaluate <tt class="docutils literal"><span class="pre">a</span> <span class="pre">+</span> <span class="pre">b</span></tt>:
the end result is a model which contains an <tt class="docutils literal"><span class="pre">a</span></tt> term in it.</p>
<p>Now consider the formula <tt class="docutils literal"><span class="pre">1</span> <span class="pre">+</span> <span class="pre">(b</span> <span class="pre">-</span> <span class="pre">1)</span></tt>. In Patsy, this is
analogous to the case above: first <tt class="docutils literal"><span class="pre">(b</span> <span class="pre">-</span> <span class="pre">1)</span></tt> is reduced to just <tt class="docutils literal"><span class="pre">b</span></tt>,
and then <tt class="docutils literal"><span class="pre">1</span> <span class="pre">+</span> <span class="pre">b</span></tt> produces a model with intercept included. In R, the
parentheses are ignored, and <tt class="docutils literal"><span class="pre">1</span> <span class="pre">+</span> <span class="pre">(b</span> <span class="pre">-</span> <span class="pre">1)</span></tt> gives a model that does
<em>not</em> include the intercept.</p>
<p>This can be slightly more confusing when it comes to the implicit
intercept term. In Patsy, this is handled exactly as if the
right-hand side of each formula has an invisible <tt class="docutils literal"><span class="pre">"1</span> <span class="pre">+"</span></tt> inserted at
the beginning. Therefore in Patsy, these formulas are different:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># Python:</span>
<span class="n">dmatrices</span><span class="p">(</span><span class="s">"y ~ b - 1"</span><span class="p">)</span> <span class="c"># equivalent to 1 + b - 1: no intercept</span>
<span class="n">dmatrices</span><span class="p">(</span><span class="s">"y ~ (b - 1)"</span><span class="p">)</span> <span class="c"># equivalent to 1 + (b - 1): has intercept</span>
</pre></div>
</div>
<p>In R, these two formulas are equivalent.</p>
</li>
<li><p class="first">Patsy has a more accurate algorithm for deciding whether to use a
full- or reduced-rank coding scheme for categorical factors. There
are two situations in which R’s coding algorithm for categorical
variables can become confused and produce over- or under-specified
model matrices. Patsy, so far as we are aware, produces correctly
specified matrices in all cases. It’s unlikely that you’ll run into
these in actual usage, but they’re worth mentioning. To illustrate,
let’s define <tt class="docutils literal"><span class="pre">a</span></tt> and <tt class="docutils literal"><span class="pre">b</span></tt> as categorical predictors, each with 2
levels:</p>
<div class="highlight-rconsole"><div class="highlight"><pre><span class="go"># R:</span>
<span class="gp">> </span>a <span class="o"><-</span> factor<span class="p">(</span>c<span class="p">(</span><span class="s">"a1"</span><span class="p">,</span> <span class="s">"a1"</span><span class="p">,</span> <span class="s">"a2"</span><span class="p">,</span> <span class="s">"a2"</span><span class="p">))</span>
<span class="gp">> </span>b <span class="o"><-</span> factor<span class="p">(</span>c<span class="p">(</span><span class="s">"b1"</span><span class="p">,</span> <span class="s">"b2"</span><span class="p">,</span> <span class="s">"b1"</span><span class="p">,</span> <span class="s">"b2"</span><span class="p">))</span>
</pre></div>
</div>
<p>The first problem occurs for formulas like <tt class="docutils literal"><span class="pre">1</span> <span class="pre">+</span> <span class="pre">a:b</span></tt>. This produces
a model matrix with rank 4, just like many other formulas that
include <tt class="docutils literal"><span class="pre">a:b</span></tt>, such as <tt class="docutils literal"><span class="pre">0</span> <span class="pre">+</span> <span class="pre">a:b</span></tt>, <tt class="docutils literal"><span class="pre">1</span> <span class="pre">+</span> <span class="pre">a</span> <span class="pre">+</span> <span class="pre">a:b</span></tt>, and <tt class="docutils literal"><span class="pre">a*b</span></tt>:</p>
<div class="highlight-rconsole"><div class="highlight"><pre><span class="go"># R:</span>
<span class="gp">> </span>qr<span class="p">(</span>model.matrix<span class="p">(</span><span class="o">~</span> <span class="m">1</span> <span class="o">+</span> a<span class="o">:</span>b<span class="p">))</span><span class="o">$</span>rank
<span class="go">[1] 4</span>
</pre></div>
</div>
<p>However, the matrix produced for this formula has 5 columns, meaning
that it contains redundant overspecification:</p>
<div class="highlight-rconsole"><div class="highlight"><pre><span class="go"># R:</span>
<span class="gp">> </span>mat <span class="o"><-</span> model.matrix<span class="p">(</span><span class="o">~</span> <span class="m">1</span> <span class="o">+</span> a<span class="o">:</span>b<span class="p">)</span>
<span class="gp">> </span>ncol<span class="p">(</span>mat<span class="p">)</span>
<span class="go">[1] 5</span>
</pre></div>
</div>
<p>The underlying problem is that R’s algorithm does not pay attention
to ‘non-local’ redundancies – it will adjust its coding to avoid a
redundancy between two terms of degree-n, or a term of degree-n and
one of degree-(n+1), but it is blind to a redundancy between a term
of degree-n and one of degree-(n+2), as we have here.</p>
<p>Patsy’s algorithm has no such limitation:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="go"># Python:</span>
<span class="gp">In [1]: </span><span class="n">a</span> <span class="o">=</span> <span class="p">[</span><span class="s">"a1"</span><span class="p">,</span> <span class="s">"a1"</span><span class="p">,</span> <span class="s">"a2"</span><span class="p">,</span> <span class="s">"a2"</span><span class="p">]</span>
<span class="gp">In [2]: </span><span class="n">b</span> <span class="o">=</span> <span class="p">[</span><span class="s">"b1"</span><span class="p">,</span> <span class="s">"b2"</span><span class="p">,</span> <span class="s">"b1"</span><span class="p">,</span> <span class="s">"b2"</span><span class="p">]</span>
<span class="gp">In [3]: </span><span class="n">mat</span> <span class="o">=</span> <span class="n">dmatrix</span><span class="p">(</span><span class="s">"1 + a:b"</span><span class="p">)</span>
<span class="gp">In [4]: </span><span class="n">mat</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="gh">Out[4]: </span><span class="go">4</span>
</pre></div>
</div>
<p>To produce this result, it codes <tt class="docutils literal"><span class="pre">a:b</span></tt> uses the same columns that
would be used to code <tt class="docutils literal"><span class="pre">b</span> <span class="pre">+</span> <span class="pre">a:b</span></tt> in the formula <tt class="docutils literal"><span class="pre">"1</span> <span class="pre">+</span> <span class="pre">b</span> <span class="pre">+</span> <span class="pre">a:b"</span></tt>.</p>
<p>The second problem occurs for formulas involving numeric
predictors. Effectively, when determining coding schemes, R assumes
that all factors are categorical. So for the formula <tt class="docutils literal"><span class="pre">0</span> <span class="pre">+</span> <span class="pre">a:c</span> <span class="pre">+</span>
<span class="pre">a:b</span></tt>, R will notice that if it used a full-rank coding for the <tt class="docutils literal"><span class="pre">c</span></tt>
and <tt class="docutils literal"><span class="pre">b</span></tt> factors, then both terms would be collinear with <tt class="docutils literal"><span class="pre">a</span></tt>, and
thus each other. Therefore, it encodes <tt class="docutils literal"><span class="pre">c</span></tt> with a full-rank
encoding, and uses a reduced-rank encoding for <tt class="docutils literal"><span class="pre">b</span></tt>. (And the <tt class="docutils literal"><span class="pre">0</span> <span class="pre">+</span></tt>
lets it avoid the previous bug.) So far, so good.</p>
<p>But now consider the formula <tt class="docutils literal"><span class="pre">0</span> <span class="pre">+</span> <span class="pre">a:x</span> <span class="pre">+</span> <span class="pre">a:b</span></tt>, where <tt class="docutils literal"><span class="pre">x</span></tt> is
numeric. Here, <tt class="docutils literal"><span class="pre">a:x</span></tt> and <tt class="docutils literal"><span class="pre">a:b</span></tt> will not be collinear, even if we do
use a full-rank encoding for <tt class="docutils literal"><span class="pre">b</span></tt>. Therefore, we <em>should</em> use a
full-rank encoding for <tt class="docutils literal"><span class="pre">b</span></tt>, and produce a model matrix with 6
columns. But in fact, R gives us only 4:</p>
<div class="highlight-rconsole"><div class="highlight"><pre><span class="go"># R:</span>
<span class="gp">> </span>x <span class="o"><-</span> c<span class="p">(</span><span class="m">1</span><span class="p">,</span> <span class="m">2</span><span class="p">,</span> <span class="m">3</span><span class="p">,</span> <span class="m">4</span><span class="p">)</span>
<span class="gp">> </span>mat <span class="o"><-</span> model.matrix<span class="p">(</span><span class="o">~</span> <span class="m">0</span> <span class="o">+</span> a<span class="o">:</span>x <span class="o">+</span> a<span class="o">:</span>b<span class="p">)</span>
<span class="gp">> </span>ncol<span class="p">(</span>mat<span class="p">)</span>
<span class="go">[1] 4</span>
</pre></div>
</div>
<p>The problem is that it cannot tell the difference between <tt class="docutils literal"><span class="pre">0</span> <span class="pre">+</span> <span class="pre">a:x</span> <span class="pre">+</span>
<span class="pre">a:b</span></tt> and <tt class="docutils literal"><span class="pre">0</span> <span class="pre">+</span> <span class="pre">a:c</span> <span class="pre">+</span> <span class="pre">a:b</span></tt>: it uses the same coding for both, whether
it’s appropriate or not.</p>
<p>(The alert reader might wonder whether this bug could be triggered
by a simpler formula, like <tt class="docutils literal"><span class="pre">0</span> <span class="pre">+</span> <span class="pre">x</span> <span class="pre">+</span> <span class="pre">b</span></tt>. It turns out that R’s code
<tt class="docutils literal"><span class="pre">do_modelmatrix</span></tt> function has a special-case where for first-order
interactions only, it <em>will</em> peek at the type of the data before
deciding on a coding scheme.)</p>
<p>Patsy always checks whether each factor is categorical or numeric
before it makes coding decisions, and thus handles this case
correctly:</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="go"># Python:</span>
<span class="gp">In [5]: </span><span class="n">x</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="gp">In [6]: </span><span class="n">mat</span> <span class="o">=</span> <span class="n">dmatrix</span><span class="p">(</span><span class="s">"0 + a:x + a:b"</span><span class="p">)</span>
<span class="gp">In [7]: </span><span class="n">mat</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="gh">Out[7]: </span><span class="go">6</span>
</pre></div>
</div>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h4>Previous topic</h4>
<p class="topless"><a href="library-developers.html"
title="previous chapter">Using Patsy in your library</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="py2-versus-py3.html"
title="next chapter">Python 2 versus Python 3</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/R-comparison.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="py2-versus-py3.html" title="Python 2 versus Python 3"
>next</a> |</li>
<li class="right" >
<a href="library-developers.html" title="Using Patsy in your library"
>previous</a> |</li>
<li><a href="index.html">patsy 0.3.0 documentation</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2011-2013, Nathaniel J. Smith.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.2.2.
</div>
</body>
</html>
|