/usr/share/doc/python-pygments/unicode.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"
   "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
  <title>Unicode and Encodings &mdash; Pygments</title>
  <meta http-equiv="content-type" content="text/html; charset=utf-8">
  <style type="text/css">
    body {
    background-color: #f2f2f2;
    margin: 0;
    padding: 0;
    font-family: 'Georgia', serif;
    color: #111;
}

#content {
    background-color: white;
    padding: 20px;
    margin: 20px auto 20px auto;
    max-width: 800px;
    border: 4px solid #ddd;
}

h1 {
    font-weight: normal;
    font-size: 40px;
    color: #09839A;
}

h2 {
    font-weight: normal;
    font-size: 30px;
    color: #C73F00;
}

h1.heading {
    margin: 0 0 30px 0;
}

h2.subheading {
    margin: -30px 0 0 45px;
}

h3 {
    margin-top: 30px;
}

table.docutils {
    border-collapse: collapse;
    border: 2px solid #aaa;
    margin: 0.5em 1.5em 0.5em 1.5em;
}

table.docutils td {
    padding: 2px;
    border: 1px solid #ddd;
}

p, li, dd, dt, blockquote {
    font-size: 15px;
    color: #333;
}

p {
    line-height: 150%;
    margin-bottom: 0;
    margin-top: 10px;
}

hr {
    border-top: 1px solid #ccc;
    border-bottom: 0;
    border-right: 0;
    border-left: 0;
    margin-bottom: 10px;
    margin-top: 20px;
}

dl {
    margin-left: 10px;
}

li, dt {
    margin-top: 5px;
}

dt {
    font-weight: bold;
}

th {
    text-align: left;
}

a {
    color: #990000;
}

a:hover {
    color: #c73f00;
}

pre {
    background-color: #f9f9f9;
    border-top: 1px solid #ccc;
    border-bottom: 1px solid #ccc;
    padding: 5px;
    font-size: 13px;
    font-family: Bitstream Vera Sans Mono,monospace;
}

tt {
    font-size: 13px;
    font-family: Bitstream Vera Sans Mono,monospace;
    color: black;
    padding: 1px 2px 1px 2px;
    background-color: #f0f0f0;
}

cite {
    /* abusing <cite>, it's generated by ReST for `x` */
    font-size: 13px;
    font-family: Bitstream Vera Sans Mono,monospace;
    font-weight: bold;
    font-style: normal;
}

#backlink {
    float: right;
    font-size: 11px;
    color: #888;
}

div.toc {
    margin: 0 0 10px 0;
}

div.toc h2 {
    font-size: 20px;
}
.syntax .hll { background-color: #ffffcc }
.syntax  { background: #ffffff; }
.syntax .c { color: #888888 } /* Comment */
.syntax .err { color: #a61717; background-color: #e3d2d2 } /* Error */
.syntax .k { color: #008800; font-weight: bold } /* Keyword */
.syntax .cm { color: #888888 } /* Comment.Multiline */
.syntax .cp { color: #cc0000; font-weight: bold } /* Comment.Preproc */
.syntax .c1 { color: #888888 } /* Comment.Single */
.syntax .cs { color: #cc0000; font-weight: bold; background-color: #fff0f0 } /* Comment.Special */
.syntax .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */
.syntax .ge { font-style: italic } /* Generic.Emph */
.syntax .gr { color: #aa0000 } /* Generic.Error */
.syntax .gh { color: #303030 } /* Generic.Heading */
.syntax .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */
.syntax .go { color: #888888 } /* Generic.Output */
.syntax .gp { color: #555555 } /* Generic.Prompt */
.syntax .gs { font-weight: bold } /* Generic.Strong */
.syntax .gu { color: #606060 } /* Generic.Subheading */
.syntax .gt { color: #aa0000 } /* Generic.Traceback */
.syntax .kc { color: #008800; font-weight: bold } /* Keyword.Constant */
.syntax .kd { color: #008800; font-weight: bold } /* Keyword.Declaration */
.syntax .kn { color: #008800; font-weight: bold } /* Keyword.Namespace */
.syntax .kp { color: #008800 } /* Keyword.Pseudo */
.syntax .kr { color: #008800; font-weight: bold } /* Keyword.Reserved */
.syntax .kt { color: #888888; font-weight: bold } /* Keyword.Type */
.syntax .m { color: #0000DD; font-weight: bold } /* Literal.Number */
.syntax .s { color: #dd2200; background-color: #fff0f0 } /* Literal.String */
.syntax .na { color: #336699 } /* Name.Attribute */
.syntax .nb { color: #003388 } /* Name.Builtin */
.syntax .nc { color: #bb0066; font-weight: bold } /* Name.Class */
.syntax .no { color: #003366; font-weight: bold } /* Name.Constant */
.syntax .nd { color: #555555 } /* Name.Decorator */
.syntax .ne { color: #bb0066; font-weight: bold } /* Name.Exception */
.syntax .nf { color: #0066bb; font-weight: bold } /* Name.Function */
.syntax .nl { color: #336699; font-style: italic } /* Name.Label */
.syntax .nn { color: #bb0066; font-weight: bold } /* Name.Namespace */
.syntax .py { color: #336699; font-weight: bold } /* Name.Property */
.syntax .nt { color: #bb0066; font-weight: bold } /* Name.Tag */
.syntax .nv { color: #336699 } /* Name.Variable */
.syntax .ow { color: #008800 } /* Operator.Word */
.syntax .w { color: #bbbbbb } /* Text.Whitespace */
.syntax .mf { color: #0000DD; font-weight: bold } /* Literal.Number.Float */
.syntax .mh { color: #0000DD; font-weight: bold } /* Literal.Number.Hex */
.syntax .mi { color: #0000DD; font-weight: bold } /* Literal.Number.Integer */
.syntax .mo { color: #0000DD; font-weight: bold } /* Literal.Number.Oct */
.syntax .sb { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Backtick */
.syntax .sc { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Char */
.syntax .sd { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Doc */
.syntax .s2 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Double */
.syntax .se { color: #0044dd; background-color: #fff0f0 } /* Literal.String.Escape */
.syntax .sh { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Heredoc */
.syntax .si { color: #3333bb; background-color: #fff0f0 } /* Literal.String.Interpol */
.syntax .sx { color: #22bb22; background-color: #f0fff0 } /* Literal.String.Other */
.syntax .sr { color: #008800; background-color: #fff0ff } /* Literal.String.Regex */
.syntax .s1 { color: #dd2200; background-color: #fff0f0 } /* Literal.String.Single */
.syntax .ss { color: #aa6600; background-color: #fff0f0 } /* Literal.String.Symbol */
.syntax .bp { color: #003388 } /* Name.Builtin.Pseudo */
.syntax .vc { color: #336699 } /* Name.Variable.Class */
.syntax .vg { color: #dd7700 } /* Name.Variable.Global */
.syntax .vi { color: #3333bb } /* Name.Variable.Instance */
.syntax .il { color: #0000DD; font-weight: bold } /* Literal.Number.Integer.Long */
  </style>
</head>
<body>
  <div id="content">
    <h1 class="heading">Pygments</h1>
    <h2 class="subheading">Unicode and Encodings</h2>
    
      <a id="backlink" href="index.html">&laquo; Back To Index</a>
    
    
    <p>Since Pygments 0.6, all lexers use unicode strings internally. Because of that
you might encounter the occasional <cite>UnicodeDecodeError</cite> if you pass strings with the
wrong encoding.</p>
<p>Per default all lexers have their input encoding set to <cite>latin1</cite>.
If you pass a lexer a string object (not unicode), it tries to decode the data
using this encoding.
You can override the encoding using the <cite>encoding</cite> lexer option. If you have the
<a class="reference external" href="http://chardet.feedparser.org/">chardet</a> library installed and set the encoding to <tt class="docutils literal">chardet</tt> if will ananlyse
the text and use the encoding it thinks is the right one automatically:</p>
<div class="syntax"><pre><span class="kn">from</span> <span class="nn">pygments.lexers</span> <span class="kn">import</span> <span class="n">PythonLexer</span>
<span class="n">lexer</span> <span class="o">=</span> <span class="n">PythonLexer</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s">&#39;chardet&#39;</span><span class="p">)</span>
</pre></div>
<p>The best way is to pass Pygments unicode objects. In that case you can't get
unexpected output.</p>
<p>The formatters now send Unicode objects to the stream if you don't set the
output encoding. You can do so by passing the formatters an <cite>encoding</cite> option:</p>
<div class="syntax"><pre><span class="kn">from</span> <span class="nn">pygments.formatters</span> <span class="kn">import</span> <span class="n">HtmlFormatter</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">HtmlFormatter</span><span class="p">(</span><span class="n">encoding</span><span class="o">=</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
</pre></div>
<p><strong>You will have to set this option if you have non-ASCII characters in the
source and the output stream does not accept Unicode written to it!</strong>
This is the case for all regular files and for terminals.</p>
<p>Note: The Terminal formatter tries to be smart: if its output stream has an
<cite>encoding</cite> attribute, and you haven't set the option, it will encode any
Unicode string with this encoding before writing it. This is the case for
<cite>sys.stdout</cite>, for example. The other formatters don't have that behavior.</p>
<p>Another note: If you call Pygments via the command line (<cite>pygmentize</cite>),
encoding is handled differently, see <a class="reference external" href="./cmdline.html">the command line docs</a>.</p>
<p><em>New in Pygments 0.7</em>: the formatters now also accept an <cite>outencoding</cite> option
which will override the <cite>encoding</cite> option if given. This makes it possible to
use a single options dict with lexers and formatters, and still have different
input and output encodings.</p>

  </div>
</body>
<!-- generated on: 2011-11-07 19:48:13.982650
     file id: unicode -->
</html>
python-pygments 1.4+dfsg-2 / usr / share / doc / python-pygments / unicode.html
