/usr/share/doc/python-whoosh-doc/html/schema.html is in python-whoosh-doc 2.5.7-1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Designing a schema — Whoosh 2.5.7 documentation</title>
<link rel="stylesheet" href="_static/default.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '2.5.7',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="top" title="Whoosh 2.5.7 documentation" href="index.html" />
<link rel="next" title="How to index documents" href="indexing.html" />
<link rel="prev" title="Glossary" href="glossary.html" />
</head>
<body>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
accesskey="I">index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="indexing.html" title="How to index documents"
accesskey="N">next</a> |</li>
<li class="right" >
<a href="glossary.html" title="Glossary"
accesskey="P">previous</a> |</li>
<li><a href="index.html">Whoosh 2.5.7 documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body">
<div class="section" id="designing-a-schema">
<h1>Designing a schema<a class="headerlink" href="#designing-a-schema" title="Permalink to this headline">¶</a></h1>
<div class="section" id="about-schemas-and-fields">
<h2>About schemas and fields<a class="headerlink" href="#about-schemas-and-fields" title="Permalink to this headline">¶</a></h2>
<p>The schema specifies the fields of documents in an index.</p>
<p>Each document can have multiple fields, such as title, content, url, date, etc.</p>
<p>Some fields can be indexed, and some fields can be stored with the document so
the field value is available in search results.
Some fields will be both indexed and stored.</p>
<p>The schema is the set of all possible fields in a document. Each individual
document might only use a subset of the available fields in the schema.</p>
<p>For example, a simple schema for indexing emails might have fields like
<tt class="docutils literal"><span class="pre">from_addr</span></tt>, <tt class="docutils literal"><span class="pre">to_addr</span></tt>, <tt class="docutils literal"><span class="pre">subject</span></tt>, <tt class="docutils literal"><span class="pre">body</span></tt>, and <tt class="docutils literal"><span class="pre">attachments</span></tt>, where
the <tt class="docutils literal"><span class="pre">attachments</span></tt> field lists the names of attachments to the email. For
emails without attachments, you would omit the attachments field.</p>
</div>
<div class="section" id="built-in-field-types">
<h2>Built-in field types<a class="headerlink" href="#built-in-field-types" title="Permalink to this headline">¶</a></h2>
<p>Whoosh provides some useful predefined field types:</p>
<dl class="docutils">
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.TEXT" title="whoosh.fields.TEXT"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.TEXT</span></tt></a></dt>
<dd><p class="first">This type is for body text. It indexes (and optionally stores) the text and
stores term positions to allow phrase searching.</p>
<p><tt class="docutils literal"><span class="pre">TEXT</span></tt> fields use <a class="reference internal" href="api/analysis.html#whoosh.analysis.StandardAnalyzer" title="whoosh.analysis.StandardAnalyzer"><tt class="xref py py-class docutils literal"><span class="pre">StandardAnalyzer</span></tt></a> by default. To specify a different
analyzer, use the <tt class="docutils literal"><span class="pre">analyzer</span></tt> keyword argument to the constructor, e.g.
<tt class="docutils literal"><span class="pre">TEXT(analyzer=analysis.StemmingAnalyzer())</span></tt>. See <a class="reference internal" href="analysis.html"><em>About analyzers</em></a>.</p>
<p>By default, <tt class="docutils literal"><span class="pre">TEXT</span></tt> fields store position information for each indexed term, to
allow you to search for phrases. If you don’t need to be able to search for
phrases in a text field, you can turn off storing term positions to save
space. Use <tt class="docutils literal"><span class="pre">TEXT(phrase=False)</span></tt>.</p>
<p class="last">By default, <tt class="docutils literal"><span class="pre">TEXT</span></tt> fields are not stored. Usually you will not want to store
the body text in the search index. Usually you have the indexed documents
themselves available to read or link to based on the search results, so you
don’t need to store their text in the search index. However, in some
circumstances it can be useful (see <a class="reference internal" href="highlight.html"><em>How to create highlighted search result excerpts</em></a>). Use
<tt class="docutils literal"><span class="pre">TEXT(stored=True)</span></tt> to specify that the text should be stored in the index.</p>
</dd>
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.KEYWORD" title="whoosh.fields.KEYWORD"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.KEYWORD</span></tt></a></dt>
<dd><p class="first">This field type is designed for space- or comma-separated keywords. This
type is indexed and searchable (and optionally stored). To save space, it
does not support phrase searching.</p>
<p>To store the value of the field in the index, use <tt class="docutils literal"><span class="pre">stored=True</span></tt> in the
constructor. To automatically lowercase the keywords before indexing them,
use <tt class="docutils literal"><span class="pre">lowercase=True</span></tt>.</p>
<p>By default, the keywords are space separated. To separate the keywords by
commas instead (to allow keywords containing spaces), use <tt class="docutils literal"><span class="pre">commas=True</span></tt>.</p>
<p class="last">If your users will use the keyword field for searching, use <tt class="docutils literal"><span class="pre">scorable=True</span></tt>.</p>
</dd>
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.ID" title="whoosh.fields.ID"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.ID</span></tt></a></dt>
<dd><p class="first">The <tt class="docutils literal"><span class="pre">ID</span></tt> field type simply indexes (and optionally stores) the entire value of
the field as a single unit (that is, it doesn’t break it up into individual
terms). This type of field does not store frequency information, so it’s
quite compact, but not very useful for scoring.</p>
<p>Use <tt class="docutils literal"><span class="pre">ID</span></tt> for fields like url or path (the URL or file path of a document),
date, category – fields where the value must be treated as a whole, and
each document only has one value for the field.</p>
<p class="last">By default, <tt class="docutils literal"><span class="pre">ID</span></tt> fields are not stored. Use <tt class="docutils literal"><span class="pre">ID(stored=True)</span></tt> to specify that
the value of the field should be stored with the document for use in the
search results. For example, you would want to store the value of a url
field so you could provide links to the original in your search results.</p>
</dd>
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.STORED" title="whoosh.fields.STORED"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.STORED</span></tt></a></dt>
<dd>This field is stored with the document, but not indexed and not searchable.
This is useful for document information you want to display to the user in
the search results, but don’t need to be able to search for.</dd>
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.NUMERIC" title="whoosh.fields.NUMERIC"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.NUMERIC</span></tt></a></dt>
<dd>This field stores int, long, or floating point numbers in a compact,
sortable format.</dd>
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.DATETIME" title="whoosh.fields.DATETIME"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.DATETIME</span></tt></a></dt>
<dd>This field stores datetime objects in a compact, sortable format.</dd>
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.BOOLEAN" title="whoosh.fields.BOOLEAN"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.BOOLEAN</span></tt></a></dt>
<dd>This simple filed indexes boolean values and allows users to search for
<tt class="docutils literal"><span class="pre">yes</span></tt>, <tt class="docutils literal"><span class="pre">no</span></tt>, <tt class="docutils literal"><span class="pre">true</span></tt>, <tt class="docutils literal"><span class="pre">false</span></tt>, <tt class="docutils literal"><span class="pre">1</span></tt>, <tt class="docutils literal"><span class="pre">0</span></tt>, <tt class="docutils literal"><span class="pre">t</span></tt> or <tt class="docutils literal"><span class="pre">f</span></tt>.</dd>
<dt><a class="reference internal" href="api/fields.html#whoosh.fields.NGRAM" title="whoosh.fields.NGRAM"><tt class="xref py py-class docutils literal"><span class="pre">whoosh.fields.NGRAM</span></tt></a></dt>
<dd>TBD.</dd>
</dl>
<p>Expert users can create their own field types.</p>
</div>
<div class="section" id="creating-a-schema">
<h2>Creating a Schema<a class="headerlink" href="#creating-a-schema" title="Permalink to this headline">¶</a></h2>
<p>To create a schema:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">whoosh.fields</span> <span class="kn">import</span> <span class="n">Schema</span><span class="p">,</span> <span class="n">TEXT</span><span class="p">,</span> <span class="n">KEYWORD</span><span class="p">,</span> <span class="n">ID</span><span class="p">,</span> <span class="n">STORED</span>
<span class="kn">from</span> <span class="nn">whoosh.analysis</span> <span class="kn">import</span> <span class="n">StemmingAnalyzer</span>
<span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="p">(</span><span class="n">from_addr</span><span class="o">=</span><span class="n">ID</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span>
<span class="n">to_addr</span><span class="o">=</span><span class="n">ID</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span>
<span class="n">subject</span><span class="o">=</span><span class="n">TEXT</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span>
<span class="n">body</span><span class="o">=</span><span class="n">TEXT</span><span class="p">(</span><span class="n">analyzer</span><span class="o">=</span><span class="n">StemmingAnalyzer</span><span class="p">()),</span>
<span class="n">tags</span><span class="o">=</span><span class="n">KEYWORD</span><span class="p">)</span>
</pre></div>
</div>
<p>If you aren’t specifying any constructor keyword arguments to one of the
predefined fields, you can leave off the brackets (e.g. <tt class="docutils literal"><span class="pre">fieldname=TEXT</span></tt> instead
of <tt class="docutils literal"><span class="pre">fieldname=TEXT()</span></tt>). Whoosh will instantiate the class for you.</p>
<p>Alternatively you can create a schema declaratively using the <tt class="docutils literal"><span class="pre">SchemaClass</span></tt>
base class:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">whoosh.fields</span> <span class="kn">import</span> <span class="n">SchemaClass</span><span class="p">,</span> <span class="n">TEXT</span><span class="p">,</span> <span class="n">KEYWORD</span><span class="p">,</span> <span class="n">ID</span><span class="p">,</span> <span class="n">STORED</span>
<span class="k">class</span> <span class="nc">MySchema</span><span class="p">(</span><span class="n">SchemaClass</span><span class="p">):</span>
<span class="n">path</span> <span class="o">=</span> <span class="n">ID</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">TEXT</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">content</span> <span class="o">=</span> <span class="n">TEXT</span>
<span class="n">tags</span> <span class="o">=</span> <span class="n">KEYWORD</span>
</pre></div>
</div>
<p>You can pass a declarative class to <a class="reference internal" href="api/index.html#whoosh.index.create_in" title="whoosh.index.create_in"><tt class="xref py py-func docutils literal"><span class="pre">create_in()</span></tt></a> or
<tt class="xref py py-meth docutils literal"><span class="pre">create_index()</span></tt> instead of a
<a class="reference internal" href="api/fields.html#whoosh.fields.Schema" title="whoosh.fields.Schema"><tt class="xref py py-class docutils literal"><span class="pre">Schema</span></tt></a> instance.</p>
</div>
<div class="section" id="modifying-the-schema-after-indexing">
<h2>Modifying the schema after indexing<a class="headerlink" href="#modifying-the-schema-after-indexing" title="Permalink to this headline">¶</a></h2>
<p>After you have created an index, you can add or remove fields to the schema
using the <tt class="docutils literal"><span class="pre">add_field()</span></tt> and <tt class="docutils literal"><span class="pre">remove_field()</span></tt> methods. These methods are
on the <tt class="docutils literal"><span class="pre">Writer</span></tt> object:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">writer</span> <span class="o">=</span> <span class="n">ix</span><span class="o">.</span><span class="n">writer</span><span class="p">()</span>
<span class="n">writer</span><span class="o">.</span><span class="n">add_field</span><span class="p">(</span><span class="s">"fieldname"</span><span class="p">,</span> <span class="n">fields</span><span class="o">.</span><span class="n">TEXT</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">))</span>
<span class="n">writer</span><span class="o">.</span><span class="n">remove_field</span><span class="p">(</span><span class="s">"content"</span><span class="p">)</span>
<span class="n">writer</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
</pre></div>
</div>
<p>(If you’re going to modify the schema <em>and</em> add documents using the same
writer, you must call <tt class="docutils literal"><span class="pre">add_field()</span></tt> and/or <tt class="docutils literal"><span class="pre">remove_field</span></tt> <em>before</em> you
add any documents.)</p>
<p>These methods are also on the <tt class="docutils literal"><span class="pre">Index</span></tt> object as a convenience, but when you
call them on an <tt class="docutils literal"><span class="pre">Index</span></tt>, the Index object simply creates the writer, calls
the corresponding method on it, and commits, so if you want to add or remove
more than one field, it’s much more efficient to create the writer yourself:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">ix</span><span class="o">.</span><span class="n">add_field</span><span class="p">(</span><span class="s">"fieldname"</span><span class="p">,</span> <span class="n">fields</span><span class="o">.</span><span class="n">KEYWORD</span><span class="p">)</span>
</pre></div>
</div>
<p>In the <tt class="docutils literal"><span class="pre">filedb</span></tt> backend, removing a field simply removes that field from the
<em>schema</em> – the index will not get smaller, data about that field will remain
in the index until you optimize. Optimizing will compact the index, removing
references to the deleted field as it goes:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">writer</span> <span class="o">=</span> <span class="n">ix</span><span class="o">.</span><span class="n">writer</span><span class="p">()</span>
<span class="n">writer</span><span class="o">.</span><span class="n">add_field</span><span class="p">(</span><span class="s">"uuid"</span><span class="p">,</span> <span class="n">fields</span><span class="o">.</span><span class="n">ID</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">))</span>
<span class="n">writer</span><span class="o">.</span><span class="n">remove_field</span><span class="p">(</span><span class="s">"path"</span><span class="p">)</span>
<span class="n">writer</span><span class="o">.</span><span class="n">commit</span><span class="p">(</span><span class="n">optimize</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
</div>
<p>Because data is stored on disk with the field name, <em>do not</em> add a new field with
the same name as a deleted field without optimizing the index in between:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">writer</span> <span class="o">=</span> <span class="n">ix</span><span class="o">.</span><span class="n">writer</span><span class="p">()</span>
<span class="n">writer</span><span class="o">.</span><span class="n">delete_field</span><span class="p">(</span><span class="s">"path"</span><span class="p">)</span>
<span class="c"># Don't do this!!!</span>
<span class="n">writer</span><span class="o">.</span><span class="n">add_field</span><span class="p">(</span><span class="s">"path"</span><span class="p">,</span> <span class="n">fields</span><span class="o">.</span><span class="n">KEYWORD</span><span class="p">)</span>
</pre></div>
</div>
<p>(A future version of Whoosh may automatically prevent this error.)</p>
</div>
<div class="section" id="dynamic-fields">
<h2>Dynamic fields<a class="headerlink" href="#dynamic-fields" title="Permalink to this headline">¶</a></h2>
<p>Dynamic fields let you associate a field type with any field name that matches
a given “glob” (a name pattern containing <tt class="docutils literal"><span class="pre">*</span></tt>, <tt class="docutils literal"><span class="pre">?</span></tt>, and/or <tt class="docutils literal"><span class="pre">[abc]</span></tt>
wildcards).</p>
<p>You can add dynamic fields to a new schema using the <tt class="docutils literal"><span class="pre">add()</span></tt> method with the
<tt class="docutils literal"><span class="pre">glob</span></tt> keyword set to True:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">schema</span> <span class="o">=</span> <span class="n">fields</span><span class="o">.</span><span class="n">Schema</span><span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="c"># Any name ending in "_d" will be treated as a stored</span>
<span class="c"># DATETIME field</span>
<span class="n">schema</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s">"*_d"</span><span class="p">,</span> <span class="n">fields</span><span class="o">.</span><span class="n">DATETIME</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span> <span class="n">glob</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
</pre></div>
</div>
<p>To set up a dynamic field on an existing index, use the same
<tt class="docutils literal"><span class="pre">IndexWriter.add_field</span></tt> method as if you were adding a regular field, but
with the <tt class="docutils literal"><span class="pre">glob</span></tt> keyword argument set to <tt class="docutils literal"><span class="pre">True</span></tt>:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">writer</span> <span class="o">=</span> <span class="n">ix</span><span class="o">.</span><span class="n">writer</span><span class="p">()</span>
<span class="n">writer</span><span class="o">.</span><span class="n">add_field</span><span class="p">(</span><span class="s">"*_d"</span><span class="p">,</span> <span class="n">fields</span><span class="o">.</span><span class="n">DATETIME</span><span class="p">(</span><span class="n">stored</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span> <span class="n">glob</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">writer</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
</pre></div>
</div>
<p>To remove a dynamic field, use the <tt class="docutils literal"><span class="pre">IndexWriter.remove_field()</span></tt> method with
the glob as the name:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">writer</span> <span class="o">=</span> <span class="n">ix</span><span class="o">.</span><span class="n">writer</span><span class="p">()</span>
<span class="n">writer</span><span class="o">.</span><span class="n">remove_field</span><span class="p">(</span><span class="s">"*_d"</span><span class="p">)</span>
<span class="n">writer</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
</pre></div>
</div>
<p>For example, to allow documents to contain any field name that ends in <tt class="docutils literal"><span class="pre">_id</span></tt>
and associate it with the <tt class="docutils literal"><span class="pre">ID</span></tt> field type:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">schema</span> <span class="o">=</span> <span class="n">fields</span><span class="o">.</span><span class="n">Schema</span><span class="p">(</span><span class="n">path</span><span class="o">=</span><span class="n">fields</span><span class="o">.</span><span class="n">ID</span><span class="p">)</span>
<span class="n">schema</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="s">"*_id"</span><span class="p">,</span> <span class="n">fields</span><span class="o">.</span><span class="n">ID</span><span class="p">,</span> <span class="n">glob</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">ix</span> <span class="o">=</span> <span class="n">index</span><span class="o">.</span><span class="n">create_in</span><span class="p">(</span><span class="s">"myindex"</span><span class="p">,</span> <span class="n">schema</span><span class="p">)</span>
<span class="n">w</span> <span class="o">=</span> <span class="n">ix</span><span class="o">.</span><span class="n">writer</span><span class="p">()</span>
<span class="n">w</span><span class="o">.</span><span class="n">add_document</span><span class="p">(</span><span class="n">path</span><span class="o">=</span><span class="s">u"/a"</span><span class="p">,</span> <span class="n">test_id</span><span class="o">=</span><span class="s">u"alfa"</span><span class="p">)</span>
<span class="n">w</span><span class="o">.</span><span class="n">add_document</span><span class="p">(</span><span class="n">path</span><span class="o">=</span><span class="s">u"/b"</span><span class="p">,</span> <span class="n">class_id</span><span class="o">=</span><span class="s">u"MyClass"</span><span class="p">)</span>
<span class="c"># ...</span>
<span class="n">w</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">qp</span> <span class="o">=</span> <span class="n">qparser</span><span class="o">.</span><span class="n">QueryParser</span><span class="p">(</span><span class="s">"path"</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">)</span>
<span class="n">q</span> <span class="o">=</span> <span class="n">qp</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="s">u"test_id:alfa"</span><span class="p">)</span>
<span class="k">with</span> <span class="n">ix</span><span class="o">.</span><span class="n">searcher</span><span class="p">()</span> <span class="k">as</span> <span class="n">s</span><span class="p">:</span>
<span class="n">results</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="n">q</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="advanced-schema-setup">
<h2>Advanced schema setup<a class="headerlink" href="#advanced-schema-setup" title="Permalink to this headline">¶</a></h2>
<div class="section" id="field-boosts">
<h3>Field boosts<a class="headerlink" href="#field-boosts" title="Permalink to this headline">¶</a></h3>
<p>You can specify a field boost for a field. This is a multiplier applied to the
score of any term found in the field. For example, to make terms found in the
title field score twice as high as terms in the body field:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="n">schema</span> <span class="o">=</span> <span class="n">Schema</span><span class="p">(</span><span class="n">title</span><span class="o">=</span><span class="n">TEXT</span><span class="p">(</span><span class="n">field_boost</span><span class="o">=</span><span class="mf">2.0</span><span class="p">),</span> <span class="n">body</span><span class="o">=</span><span class="n">TEXT</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="section" id="field-types">
<h3>Field types<a class="headerlink" href="#field-types" title="Permalink to this headline">¶</a></h3>
<p>The predefined field types listed above are subclasses of <tt class="docutils literal"><span class="pre">fields.FieldType</span></tt>.
<tt class="docutils literal"><span class="pre">FieldType</span></tt> is a pretty simple class. Its attributes contain information that
define the behavior of a field.</p>
<table border="1" class="docutils">
<colgroup>
<col width="14%" />
<col width="18%" />
<col width="67%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Attribute</th>
<th class="head">Type</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>format</td>
<td>fields.Format</td>
<td>Defines what kind of information a field records
about each term, and how the information is stored
on disk.</td>
</tr>
<tr class="row-odd"><td>vector</td>
<td>fields.Format</td>
<td>Optional: if defined, the format in which to store
per-document forward-index information for this field.</td>
</tr>
<tr class="row-even"><td>scorable</td>
<td>bool</td>
<td>If True, the length of (number of terms in) the field in
each document is stored in the index. Slightly misnamed,
since field lengths are not required for all scoring.
However, field lengths are required to get proper
results from BM25F.</td>
</tr>
<tr class="row-odd"><td>stored</td>
<td>bool</td>
<td>If True, the value of this field is stored
in the index.</td>
</tr>
<tr class="row-even"><td>unique</td>
<td>bool</td>
<td>If True, the value of this field may be used to
replace documents with the same value when the user
calls
<tt class="xref py py-meth docutils literal"><span class="pre">document_update()</span></tt>
on an <tt class="docutils literal"><span class="pre">IndexWriter</span></tt>.</td>
</tr>
</tbody>
</table>
<p>The constructors for most of the predefined field types have parameters that let
you customize these parts. For example:</p>
<ul class="simple">
<li>Most of the predefined field types take a stored keyword argument that sets
FieldType.stored.</li>
<li>The <tt class="docutils literal"><span class="pre">TEXT()</span></tt> constructor takes an <tt class="docutils literal"><span class="pre">analyzer</span></tt> keyword argument that is
passed on to the format object.</li>
</ul>
</div>
<div class="section" id="formats">
<h3>Formats<a class="headerlink" href="#formats" title="Permalink to this headline">¶</a></h3>
<p>A <tt class="docutils literal"><span class="pre">Format</span></tt> object defines what kind of information a field records about each
term, and how the information is stored on disk.</p>
<p>For example, the <tt class="docutils literal"><span class="pre">Existence</span></tt> format would store postings like this:</p>
<table border="1" class="docutils">
<colgroup>
<col width="50%" />
<col width="50%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Doc</th>
<th class="head"> </th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>10</td>
<td> </td>
</tr>
<tr class="row-odd"><td>20</td>
<td> </td>
</tr>
<tr class="row-even"><td>30</td>
<td> </td>
</tr>
</tbody>
</table>
<p>Whereas the <tt class="docutils literal"><span class="pre">Positions</span></tt> format would store postings like this:</p>
<table border="1" class="docutils">
<colgroup>
<col width="28%" />
<col width="72%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Doc</th>
<th class="head">Positions</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>10</td>
<td><tt class="docutils literal"><span class="pre">[1,5,23]</span></tt></td>
</tr>
<tr class="row-odd"><td>20</td>
<td><tt class="docutils literal"><span class="pre">[45]</span></tt></td>
</tr>
<tr class="row-even"><td>30</td>
<td><tt class="docutils literal"><span class="pre">[7,12]</span></tt></td>
</tr>
</tbody>
</table>
<p>The indexing code passes the unicode string for a field to the field’s <tt class="docutils literal"><span class="pre">Format</span></tt>
object. The <tt class="docutils literal"><span class="pre">Format</span></tt> object calls its analyzer (see text analysis) to break the
string into tokens, then encodes information about each token.</p>
<p>Whoosh ships with the following pre-defined formats.</p>
<table border="1" class="docutils">
<colgroup>
<col width="19%" />
<col width="81%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Class name</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>Stored</td>
<td>A “null” format for fields that are stored but not indexed.</td>
</tr>
<tr class="row-odd"><td>Existence</td>
<td>Records only whether a term is in a document or not, i.e. it
does not store term frequency. Useful for identifier fields
(e.g. path or id) and “tag”-type fields, where the frequency
is expected to always be 0 or 1.</td>
</tr>
<tr class="row-even"><td>Frequency</td>
<td>Stores the number of times each term appears in each document.</td>
</tr>
<tr class="row-odd"><td>Positions</td>
<td>Stores the number of times each term appears in each document,
and at what positions.</td>
</tr>
</tbody>
</table>
<p>The <tt class="docutils literal"><span class="pre">STORED</span></tt> field type uses the <tt class="docutils literal"><span class="pre">Stored</span></tt> format (which does nothing, so <tt class="docutils literal"><span class="pre">STORED</span></tt>
fields are not indexed). The <tt class="docutils literal"><span class="pre">ID</span></tt> type uses the <tt class="docutils literal"><span class="pre">Existence</span></tt> format. The <tt class="docutils literal"><span class="pre">KEYWORD</span></tt> type
uses the <tt class="docutils literal"><span class="pre">Frequency</span></tt> format. The <tt class="docutils literal"><span class="pre">TEXT</span></tt> type uses the <tt class="docutils literal"><span class="pre">Positions</span></tt> format if it is
instantiated with <tt class="docutils literal"><span class="pre">phrase=True</span></tt> (the default), or <tt class="docutils literal"><span class="pre">Frequency</span></tt> if <tt class="docutils literal"><span class="pre">phrase=False</span></tt>.</p>
<p>In addition, the following formats are implemented for the possible convenience
of expert users, but are not currently used in Whoosh:</p>
<table border="1" class="docutils">
<colgroup>
<col width="21%" />
<col width="79%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Class name</th>
<th class="head">Description</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>DocBoosts</td>
<td>Like Existence, but also stores per-document boosts</td>
</tr>
<tr class="row-odd"><td>Characters</td>
<td>Like Positions, but also stores the start and end character
indices of each term</td>
</tr>
<tr class="row-even"><td>PositionBoosts</td>
<td>Like Positions, but also stores per-position boosts</td>
</tr>
<tr class="row-odd"><td>CharacterBoosts</td>
<td>Like Positions, but also stores the start and end character
indices of each term and per-position boosts</td>
</tr>
</tbody>
</table>
</div>
<div class="section" id="vectors">
<h3>Vectors<a class="headerlink" href="#vectors" title="Permalink to this headline">¶</a></h3>
<p>The main index is an inverted index. It maps terms to the documents they appear
in. It is also sometimes useful to store a forward index, also known as a term
vector, that maps documents to the terms that appear in them.</p>
<p>For example, imagine an inverted index like this for a field:</p>
<table border="1" class="docutils">
<colgroup>
<col width="15%" />
<col width="85%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Term</th>
<th class="head">Postings</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>apple</td>
<td><tt class="docutils literal"><span class="pre">[(doc=1,</span> <span class="pre">freq=2),</span> <span class="pre">(doc=2,</span> <span class="pre">freq=5),</span> <span class="pre">(doc=3,</span> <span class="pre">freq=1)]</span></tt></td>
</tr>
<tr class="row-odd"><td>bear</td>
<td><tt class="docutils literal"><span class="pre">[(doc=2,</span> <span class="pre">freq=7)]</span></tt></td>
</tr>
</tbody>
</table>
<p>The corresponding forward index, or term vector, would be:</p>
<table border="1" class="docutils">
<colgroup>
<col width="16%" />
<col width="84%" />
</colgroup>
<thead valign="bottom">
<tr class="row-odd"><th class="head">Doc</th>
<th class="head">Postings</th>
</tr>
</thead>
<tbody valign="top">
<tr class="row-even"><td>1</td>
<td><tt class="docutils literal"><span class="pre">[(text=apple,</span> <span class="pre">freq=2)]</span></tt></td>
</tr>
<tr class="row-odd"><td>2</td>
<td><tt class="docutils literal"><span class="pre">[(text=apple,</span> <span class="pre">freq=5),</span> <span class="pre">(text='bear',</span> <span class="pre">freq=7)]</span></tt></td>
</tr>
<tr class="row-even"><td>3</td>
<td><tt class="docutils literal"><span class="pre">[(text=apple,</span> <span class="pre">freq=1)]</span></tt></td>
</tr>
</tbody>
</table>
<p>If you set <tt class="docutils literal"><span class="pre">FieldType.vector</span></tt> to a <tt class="docutils literal"><span class="pre">Format</span></tt> object, the indexing code will use the
<tt class="docutils literal"><span class="pre">Format</span></tt> object to store information about the terms in each document. Currently
by default Whoosh does not make use of term vectors at all, but they are
available to expert users who want to implement their own field types.</p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">Designing a schema</a><ul>
<li><a class="reference internal" href="#about-schemas-and-fields">About schemas and fields</a></li>
<li><a class="reference internal" href="#built-in-field-types">Built-in field types</a></li>
<li><a class="reference internal" href="#creating-a-schema">Creating a Schema</a></li>
<li><a class="reference internal" href="#modifying-the-schema-after-indexing">Modifying the schema after indexing</a></li>
<li><a class="reference internal" href="#dynamic-fields">Dynamic fields</a></li>
<li><a class="reference internal" href="#advanced-schema-setup">Advanced schema setup</a><ul>
<li><a class="reference internal" href="#field-boosts">Field boosts</a></li>
<li><a class="reference internal" href="#field-types">Field types</a></li>
<li><a class="reference internal" href="#formats">Formats</a></li>
<li><a class="reference internal" href="#vectors">Vectors</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="glossary.html"
title="previous chapter">Glossary</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="indexing.html"
title="next chapter">How to index documents</a></p>
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/schema.txt"
rel="nofollow">Show Source</a></li>
</ul>
<div id="searchbox" style="display: none">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<input type="text" name="q" />
<input type="submit" value="Go" />
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
<p class="searchtip" style="font-size: 90%">
Enter search terms or a module, class or function name.
</p>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="genindex.html" title="General Index"
>index</a></li>
<li class="right" >
<a href="py-modindex.html" title="Python Module Index"
>modules</a> |</li>
<li class="right" >
<a href="indexing.html" title="How to index documents"
>next</a> |</li>
<li class="right" >
<a href="glossary.html" title="Glossary"
>previous</a> |</li>
<li><a href="index.html">Whoosh 2.5.7 documentation</a> »</li>
</ul>
</div>
<div class="footer">
© Copyright 2007-2012 Matt Chaput.
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.2.2.
</div>
</body>
</html>
|