/usr/share/doc/recode-doc/Universal.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<!-- Created by GNU Texinfo 6.0, http://www.gnu.org/software/texinfo/ -->
<head>
<title>The recode reference manual: Universal</title>

<meta name="description" content="The recode reference manual: Universal">
<meta name="keywords" content="The recode reference manual: Universal">
<meta name="resource-type" content="document">
<meta name="distribution" content="global">
<meta name="Generator" content="makeinfo">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link href="index.html#Top" rel="start" title="Top">
<link href="Concept-Index.html#Concept-Index" rel="index" title="Concept Index">
<link href="Charset-and-Surface-Index.html#SEC_Contents" rel="contents" title="Table of Contents">
<link href="index.html#Top" rel="up" title="Top">
<link href="libiconv.html#libiconv" rel="next" title="libiconv">
<link href="Library.html#Errors" rel="prev" title="Errors">
<style type="text/css">
<!--
a.summary-letter {text-decoration: none}
blockquote.indentedblock {margin-right: 0em}
blockquote.smallindentedblock {margin-right: 0em; font-size: smaller}
blockquote.smallquotation {font-size: smaller}
div.display {margin-left: 3.2em}
div.example {margin-left: 3.2em}
div.lisp {margin-left: 3.2em}
div.smalldisplay {margin-left: 3.2em}
div.smallexample {margin-left: 3.2em}
div.smalllisp {margin-left: 3.2em}
kbd {font-style: oblique}
pre.display {font-family: inherit}
pre.format {font-family: inherit}
pre.menu-comment {font-family: serif}
pre.menu-preformatted {font-family: serif}
pre.smalldisplay {font-family: inherit; font-size: smaller}
pre.smallexample {font-size: smaller}
pre.smallformat {font-family: inherit; font-size: smaller}
pre.smalllisp {font-size: smaller}
span.nocodebreak {white-space: nowrap}
span.nolinebreak {white-space: nowrap}
span.roman {font-family: serif; font-weight: normal}
span.sansserif {font-family: sans-serif; font-weight: normal}
ul.no-bullet {list-style: none}
-->
</style>


</head>

<body lang="en">
<a name="Universal"></a>
<div class="header">
<p>
Next: <a href="libiconv.html#libiconv" accesskey="n" rel="next">libiconv</a>, Previous: <a href="Library.html#Library" accesskey="p" rel="prev">Library</a>, Up: <a href="index.html#Top" accesskey="u" rel="up">Top</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="The-universal-charset"></a>
<h2 class="chapter">5 The universal charset</h2>

<a name="index-ISO-10646"></a>
<p>Standard ISO&nbsp;10646<!-- /@w --> defines a universal character set, intended to encompass
in the long run all languages written on this planet.  It is based on
wide characters, and offer possibilities for two billion characters
(<em>2^31</em>).
</p>
<a name="index-UCS"></a>
<p>This charset was to become available in <code>recode</code> under the name
<code>UCS</code>, with many external surfaces for it.  But in the current
version, only surfaces of <code>UCS</code> are offered, each presented as a
genuine charset rather than a surface.  Such surfaces are only meaningful
for the <code>UCS</code> charset, so it is not that useful to draw a line
between the surfaces and the only charset to which they may apply.
</p>
<a name="index-UTF_002d1"></a>
<p><code>UCS</code> stands for Universal Character Set.  <code>UCS-2</code> and
<code>UCS-4</code> are fixed length encodings, using two or four bytes per
character respectively.  <code>UTF</code> stands for <code>UCS</code> Transformation
Format, and are variable length encodings dedicated to <code>UCS</code>.
<code>UTF-1</code> was based on ISO&nbsp;2022<!-- /@w -->, it did not succeed<a name="DOCF9" href="#FOOT9"><sup>9</sup></a>.  <code>UTF-2</code>
replaced it, it has been called <code>UTF-FSS</code> (File System Safe) in
Unicode or Plan9 context, but is better known today as <code>UTF-8</code>.
To complete the picture, there is <code>UTF-16</code> based on 16 bits bytes,
and <code>UTF-7</code> which is meant for transmissions limited to 7-bit bytes.
Most often, one might see <code>UTF-8</code> used for external storage, and
<code>UCS-2</code> used for internal storage.
</p>
<p>When <code>recode</code> is producing any representation of <code>UCS</code>,
it uses the replacement character <code>U+FFFD</code> for any <em>valid</em>
character which is not representable in the goal charset<a name="DOCF10" href="#FOOT10"><sup>10</sup></a>.
This happens, for example, when <code>UCS-2</code> is not capable to echo a
wide <code>UCS-4</code> character, or for a similar reason, an <code>UTF-8</code>
sequence using more than three bytes.  The replacement character is
meant to represent an existing character.  So, it is never produced to
represent an invalid sequence or ill-formed character in the input text.
In such cases, <code>recode</code> just gets rid of the noise, while taking note
of the error in its usual ways.
</p>
<p>Even if <code>UTF-8</code> is an encoding, really, it is the encoding of a single
character set, and nothing else.  It is useful to distinguish between an
encoding (a <em>surface</em> within <code>recode</code>) and a charset, but only
when the surface may be applied to several charsets.  Specifying a charset
is a bit simpler than specifying a surface in a <code>recode</code> request.
There would not be a practical advantage at imposing a more complex syntax
to <code>recode</code> users, when it is simple to assimilate <code>UTF-8</code> to
a charset.  Similar considerations apply for <code>UCS-2</code>, <code>UCS-4</code>,
<code>UTF-16</code> and <code>UTF-7</code>.  These are all considered to be charsets.
</p>
<table class="menu" border="0" cellspacing="0">
<tr><td align="left" valign="top">&bull; <a href="#UCS_002d2" accesskey="1">UCS-2</a>:</td><td>&nbsp;&nbsp;</td><td align="left" valign="top">Universal Character Set, 2 bytes
</td></tr>
<tr><td align="left" valign="top">&bull; <a href="#UCS_002d4" accesskey="2">UCS-4</a>:</td><td>&nbsp;&nbsp;</td><td align="left" valign="top">Universal Character Set, 4 bytes
</td></tr>
<tr><td align="left" valign="top">&bull; <a href="#UTF_002d7" accesskey="3">UTF-7</a>:</td><td>&nbsp;&nbsp;</td><td align="left" valign="top">Universal Transformation Format, 7 bits
</td></tr>
<tr><td align="left" valign="top">&bull; <a href="#UTF_002d8" accesskey="4">UTF-8</a>:</td><td>&nbsp;&nbsp;</td><td align="left" valign="top">Universal Transformation Format, 8 bits
</td></tr>
<tr><td align="left" valign="top">&bull; <a href="#UTF_002d16" accesskey="5">UTF-16</a>:</td><td>&nbsp;&nbsp;</td><td align="left" valign="top">Universal Transformation Format, 16 bits
</td></tr>
<tr><td align="left" valign="top">&bull; <a href="#count_002dcharacters" accesskey="6">count-characters</a>:</td><td>&nbsp;&nbsp;</td><td align="left" valign="top">Frequency count of characters
</td></tr>
<tr><td align="left" valign="top">&bull; <a href="#dump_002dwith_002dnames" accesskey="7">dump-with-names</a>:</td><td>&nbsp;&nbsp;</td><td align="left" valign="top">Fully interpreted UCS dump
</td></tr>
</table>

<hr>
<a name="UCS_002d2"></a>
<div class="header">
<p>
Next: <a href="#UCS_002d4" accesskey="n" rel="next">UCS-4</a>, Previous: <a href="#Universal" accesskey="p" rel="prev">Universal</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="Universal-Character-Set_002c-2-bytes"></a>
<h3 class="section">5.1 Universal Character Set, 2 bytes</h3>

<a name="index-UCS_002d2"></a>
<a name="index-Unicode"></a>
<p>One surface of <code>UCS</code> is usable for the subset defined by its first
sixty thousand characters (in fact, <em>31 * 2^11</em> codes), and uses
exactly two bytes per character.  It is a mere dump of the internal
memory representation which is <em>natural</em> for this subset and as such,
conveys with it endianness problems.
</p>
<a name="index-byte-order-mark"></a>
<p>A non-empty <code>UCS-2</code> file normally begins with a so called <em>byte
order mark</em>, having value <code>0xFEFF</code>.  The value <code>0xFFFE</code> is not an
<code>UCS</code> character, so if this value is seen at the beginning of a file,
<code>recode</code> reacts by swapping all pairs of bytes.  The library also
properly reacts to other occurrences of <code>0xFEFF</code> or <code>0xFFFE</code>
elsewhere than at the beginning, because concatenation of <code>UCS-2</code>
files should stay a simple matter, but it might trigger a diagnostic
about non canonical input.
</p>
<p>By default, when producing an <code>UCS-2</code> file, <code>recode</code> always
outputs the high order byte before the low order byte.  But this could be
easily overridden through the <code>21-Permutation</code> surface
(see <a href="Surfaces.html#Permutations">Permutations</a>).  For example, the command:
</p>
<div class="example">
<pre class="example">recode u8..u2/21 &lt; <var>input</var> &gt; <var>output</var>
</pre></div>

<p>asks for an <code>UTF-8</code> to <code>UCS-2</code> conversion, with swapped byte
output.
</p>
<a name="index-ISO_002d10646_002dUCS_002d2_002c-and-aliases"></a>
<a name="index-BMP"></a>
<a name="index-rune"></a>
<a name="index-u2"></a>
<p>Use <code>UCS-2</code> as a genuine charset.  This charset is available in
<code>recode</code> under the name <code>ISO-10646-UCS-2</code>.  Accepted aliases
are <code>UCS-2</code>, <code>BMP</code>, <code>rune</code> and <code>u2</code>.
</p>
<a name="index-combined_002dUCS_002d2"></a>
<a name="index-combining-characters"></a>
<p>The <code>recode</code> library is able to combine <code>UCS-2</code> some sequences
of codes into single code characters, to represent a few diacriticized
characters, ligatures or diphtongs which have been included to ease
mapping with other existing charsets.  It is also able to explode
such single code characters into the corresponding sequence of codes.
The request syntax for triggering such operations is rudimentary and
temporary.  The <code>combined-UCS-2</code> pseudo character set is a special
form of <code>UCS-2</code> in which known combinings have been replaced by the
simpler code.  Using <code>combined-UCS-2</code> instead of <code>UCS-2</code> in an
<em>after</em> position of a request forces a combining step, while using
<code>combined-UCS-2</code> instead of <code>UCS-2</code> in a <em>before</em> position
of a request forces an exploding step.  For the time being, one has to
resort to advanced request syntax to achieve other effects.  For example:
</p>
<div class="example">
<pre class="example">recode u8..co,u2..u8 &lt; <var>input</var> &gt; <var>output</var>
</pre></div>

<p>copies an <code>UTF-8</code> <var>input</var> over <var>output</var>, still to be in
<code>UTF-8</code>, yet merging combining characters into single codes whenever
possible.
</p>
<hr>
<a name="UCS_002d4"></a>
<div class="header">
<p>
Next: <a href="#UTF_002d7" accesskey="n" rel="next">UTF-7</a>, Previous: <a href="#UCS_002d2" accesskey="p" rel="prev">UCS-2</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="Universal-Character-Set_002c-4-bytes"></a>
<h3 class="section">5.2 Universal Character Set, 4 bytes</h3>

<a name="index-UCS_002d4"></a>
<p>Another surface of <code>UCS</code> uses exactly four bytes per character, and is
a mere dump of the internal memory representation which is <em>natural</em>
for the whole charset and as such, conveys with it endianness problems.
</p>
<a name="index-ISO_002d10646_002dUCS_002d4_002c-and-aliases"></a>
<a name="index-ISO_005f10646"></a>
<a name="index-10646"></a>
<a name="index-u4"></a>
<p>Use it as a genuine charset.  This charset is available in <code>recode</code>
under the name <code>ISO-10646-UCS-4</code>.  Accepted aliases are <code>UCS</code>,
<code>UCS-4</code>, <code>ISO_10646</code>, <code>10646</code> and <code>u4</code>.
</p>
<hr>
<a name="UTF_002d7"></a>
<div class="header">
<p>
Next: <a href="#UTF_002d8" accesskey="n" rel="next">UTF-8</a>, Previous: <a href="#UCS_002d4" accesskey="p" rel="prev">UCS-4</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="Universal-Transformation-Format_002c-7-bits"></a>
<h3 class="section">5.3 Universal Transformation Format, 7 bits</h3>

<a name="index-UTF_002d7"></a>
<p><code>UTF-7</code> comes from IETF rather than ISO, and is described by RFC&nbsp;2152<!-- /@w -->, in the MIME series.  The <code>UTF-7</code> encoding is meant to fit
<code>UCS-2</code> over channels limited to seven bits per byte.  It proceeds
from a mix between the spirit of <code>Quoted-Printable</code> and methods of
<code>Base64</code>, adapted to Unicode contexts.
</p>
<a name="index-UNICODE_002d1_002d1_002dUTF_002d7_002c-and-aliases"></a>
<a name="index-TF_002d7"></a>
<a name="index-u7"></a>
<p>This charset is available in <code>recode</code> under the name
<code>UNICODE-1-1-UTF-7</code>.  Accepted aliases are <code>UTF-7</code>, <code>TF-7</code>
and <code>u7</code>.
</p>
<hr>
<a name="UTF_002d8"></a>
<div class="header">
<p>
Next: <a href="#UTF_002d16" accesskey="n" rel="next">UTF-16</a>, Previous: <a href="#UTF_002d7" accesskey="p" rel="prev">UTF-7</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="Universal-Transformation-Format_002c-8-bits"></a>
<h3 class="section">5.4 Universal Transformation Format, 8 bits</h3>

<a name="index-UTF_002d8"></a>
<p>Even if <code>UTF-8</code> does not originally come from IETF, there is now
RFC&nbsp;2279<!-- /@w --> to describe it.  In letters sent on 1995-01-21 and 1995-04-20,
Markus Kuhn writes:
</p>
<blockquote>
<p><code>UTF-8</code> is an <code>ASCII</code> compatible multi-byte encoding of the ISO&nbsp;10646<!-- /@w --> universal character set (<code>UCS</code>).  <code>UCS</code> is a 31-bit superset
of all other character set standards.  The first 256 characters of <code>UCS</code>
are identical to those of ISO&nbsp;<span class="nolinebreak">8859-1</span><!-- /@w --> (<span class="nolinebreak">Latin-1</span><!-- /@w -->).  The <code>UCS-2</code>
encoding of UCS is a sequence of bigendian 16-bit words, the <code>UCS-4</code>
encoding is a sequence of bigendian 32-bit words.  The <code>UCS-2</code> subset
of ISO&nbsp;10646<!-- /@w --> is also known as &ldquo;Unicode&rdquo;.  As both <code>UCS-2</code>
and <code>UCS-4</code> require heavy modifications to traditional <code>ASCII</code>
oriented system designs (e.g. Unix), the <code>UTF-8</code> encoding has been
designed for these applications.
</p>
<p>In <code>UTF-8</code>, only <code>ASCII</code> characters are encoded using bytes
below 128.  All other non-ASCII characters are encoded as multi-byte
sequences consisting only of bytes in the range 128-253.  This avoids
critical bytes like <kbd>NUL</kbd> and <kbd>/</kbd> in <code>UTF-8</code> strings, which
makes the <code>UTF-8</code> encoding suitable for being handled by the standard
C string library and being used in Unix file names.  Other properties
include the preserved lexical sorting order and that <code>UTF-8</code> allows
easy self-synchronisation of software receiving <code>UTF-8</code> strings.
</p></blockquote>

<p><code>UTF-8</code> is the most common external surface of <code>UCS</code>, each
character uses from one to six bytes, and is able to encode all <em>2^31</em>
characters of the <code>UCS</code>.  It is implemented as a charset, with the
following properties:
</p>
<ul>
<li> Strict 7-bit <code>ASCII</code> is completely invariant under <code>UTF-8</code>,
and those are the only one-byte characters.  <code>UCS</code> values and
<code>ASCII</code> values coincide.  No multi-byte characters ever contain bytes
less than 128.  <code>NUL</code> <em>is</em> <code>NUL</code>.  A multi-byte character
always starts with a byte of 192 or more, and is always followed by a
number of bytes between 128 to 191.  That means that you may read at
random on disk or memory, and easily discover the start of the current,
next or previous character.  You can count, skip or extract characters
with this only knowledge.

</li><li> If you read the first byte of a multi-byte character in binary, it contains
many &lsquo;<samp>1</samp>&rsquo; bits in successions starting with the most significant one
(from the left), at least two.  The length of this &lsquo;<samp>1</samp>&rsquo; sequence equals
the byte size of the character.  All succeeding bytes start by &lsquo;<samp>10</samp>&rsquo;.
This is a lot of redundancy, making it fairly easy to guess that a file
is valid <code>UTF-8</code>, or to safely state that it is not.

</li><li> In a multi-byte character, if you remove all leading &lsquo;<samp>1</samp>&rsquo; bits of the
first byte of a multi-byte character, and the initial &lsquo;<samp>10</samp>&rsquo; bits of
all remaining bytes (so keeping 6 bits per byte for those), the remaining
bits concatenated are the UCS value.
</li></ul>

<p>These properties also have a few nice consequences:
</p>
<ul>
<li> Conversion to/from values is algorithmically simple, and reasonably speedy.

</li><li> A sequence of <var>N</var> bytes can hold characters needing up to 2 + 5<var>N</var>
bits in their <code>UCS</code> representation.  Here, <var>N</var> is a number between
1 and 6.  So, <code>UTF-8</code> is most economical when mapping ASCII (1 byte),
followed by <code>UCS-2</code> (1 to 3 bytes) and <code>UCS-4</code> (1 to 6 bytes).

</li><li> The lexicographic sorting order of <code>UCS</code> strings is preserved.

</li><li> Bytes with value 254 or 255 never appear, and because of that, these are
sometimes used when escape mechanisms are needed.
</li></ul>

<p>In some case, when little processing is done on a lot of strings, one may
choose for efficiency reasons to handle <code>UTF-8</code> strings directly even
if variable length, as it is easy to get start of characters.  Character
insertion or replacement might require moving the remainder of the string
in either direction.  In most cases, it is faster and easier to convert
from <code>UTF-8</code> to <code>UCS-2</code> or <code>UCS-4</code> prior to processing.
</p>
<a name="index-UTF_002d8_002c-aliases"></a>
<a name="index-UTF_002dFSS"></a>
<a name="index-FSS_005fUTF"></a>
<a name="index-TF_002d8"></a>
<a name="index-u8"></a>
<p>This charset is available in <code>recode</code> under the name <code>UTF-8</code>.
Accepted aliases are <code>UTF-2</code>, <code>UTF-FSS</code>, <code>FSS_UTF</code>,
<code>TF-8</code> and <code>u8</code>.
</p>
<hr>
<a name="UTF_002d16"></a>
<div class="header">
<p>
Next: <a href="#count_002dcharacters" accesskey="n" rel="next">count-characters</a>, Previous: <a href="#UTF_002d8" accesskey="p" rel="prev">UTF-8</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="Universal-Transformation-Format_002c-16-bits"></a>
<h3 class="section">5.5 Universal Transformation Format, 16 bits</h3>

<a name="index-UTF_002d16_002c-and-aliases"></a>
<p>Another external surface of <code>UCS</code> is also variable length, each
character using either two or four bytes.  It is usable for the subset
defined by the first million characters (<em>17 * 2^16</em>) of <code>UCS</code>.
</p>
<p>Martin J. D&uuml;rst writes (to <a href="comp.std.internat">comp.std.internat</a>, on 1995-03-28):
</p>
<blockquote>
<p><code>UTF-16</code> is another method that reserves two times 1024 codepoints in
Unicode and uses them to index around one million additional characters.
<code>UTF-16</code> is a little bit like former multibyte codes, but quite
not so, as both the first and the second 16-bit code clearly show what
they are.  The idea is that one million codepoints should be enough for
all the rare Chinese ideograms and historical scripts that do not fit
into the Base Multilingual Plane of ISO&nbsp;10646<!-- /@w --> (with just about 63,000
positions available, now that 2,000 are gone).
</p></blockquote>

<a name="index-Unicode_002c-an-alias-for-UTF_002d16"></a>
<a name="index-TF_002d16"></a>
<a name="index-u6"></a>
<p>This charset is available in <code>recode</code> under the name <code>UTF-16</code>.
Accepted aliases are <code>Unicode</code>, <code>TF-16</code> and <code>u6</code>.
</p>
<hr>
<a name="count_002dcharacters"></a>
<div class="header">
<p>
Next: <a href="#dump_002dwith_002dnames" accesskey="n" rel="next">dump-with-names</a>, Previous: <a href="#UTF_002d16" accesskey="p" rel="prev">UTF-16</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="Frequency-count-of-characters"></a>
<h3 class="section">5.6 Frequency count of characters</h3>

<a name="index-count_002dcharacters"></a>
<a name="index-counting-characters"></a>
<p>A device may be used to obtain a list of characters in a file, and how many
times each character appears.  Each count is followed by the <code>UCS-2</code>
value of the character and, when known, the RFC&nbsp;1345<!-- /@w --> mnemonic for that
character.
</p>
<p>This charset is available in <code>recode</code> under the name
<code>count-characters</code>.
</p>
<p>This <code>count</code> feature has been implemented as a charset.  This may
change in some later version, as it would sometimes be convenient to count
original bytes, instead of their <code>UCS-2</code> equivalent.
</p>
<hr>
<a name="dump_002dwith_002dnames"></a>
<div class="header">
<p>
Previous: <a href="#count_002dcharacters" accesskey="p" rel="prev">count-characters</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>
<a name="Fully-interpreted-UCS-dump"></a>
<h3 class="section">5.7 Fully interpreted UCS dump</h3>

<a name="index-dump_002dwith_002dnames"></a>
<a name="index-dumping-characters_002c-with-description"></a>
<a name="index-character-streams_002c-description"></a>
<a name="index-description-of-individual-characters"></a>
<p>Another device may be used to get fully interpreted dumps of an <code>UCS-2</code>
stream of characters, with one <code>UCS-2</code> character displayed on a full
output line.  Each line receives the RFC&nbsp;1345<!-- /@w --> mnemonic for the character
if it exists, the <code>UCS-2</code> value of the character, and a descriptive
comment for that character.  As each input character produces its own
output line, beware that the output file from this conversion may be much,
much bigger than the input file.
</p>
<p>This charset is available in <code>recode</code> under the name
<code>dump-with-names</code>.
</p>
<p>This <code>dump-with-names</code> feature has been implemented as a charset rather
than a surface.  This is surely debatable.  The current implementation
allows for dumping charsets other than <code>UCS-2</code>.  For example, the
command &lsquo;<samp>recode&nbsp;l2..full&nbsp;&lt;&nbsp;<var>input</var></samp>&rsquo;<!-- /@w --> implies a necessary
conversion from <code>Latin-2</code> to <code>UCS-2</code>, as <code>dump-with-names</code>
is only connected out from <code>UCS-2</code>.  In such cases, <code>recode</code>
does not display the original <code>Latin-2</code> codes in the dump, only the
corresponding <code>UCS-2</code> values.  To give a simpler example, the command
</p>
<div class="example">
<pre class="example">echo 'Hello, world!' | recode us..dump
</pre></div>

<p>produces the following output:
</p>
<div class="example">
<pre class="example">UCS2   Mne   Description

0048   H     latin capital letter h
0065   e     latin small letter e
006C   l     latin small letter l
006C   l     latin small letter l
006F   o     latin small letter o
002C   ,     comma
0020   SP    space
0077   w     latin small letter w
006F   o     latin small letter o
0072   r     latin small letter r
006C   l     latin small letter l
0064   d     latin small letter d
0021   !     exclamation mark
000A   LF    line feed (lf)
</pre></div>

<p>The descriptive comment is given in English and <code>ASCII</code>,
yet if the English description is not available but a French one is, then
the French description is given instead, using <code>Latin-1</code>.  However,
if the <code>LANGUAGE</code> or <code>LANG</code> environment variable begins with
the letters &lsquo;<samp>fr</samp>&rsquo;, then listing preference goes to French when both
descriptions are available.
</p>
<p>Here is another example.  To get the long description of the code 237 in
<span class="nolinebreak">Latin-5</span><!-- /@w --> table, one may use the following command.
</p>
<div class="example">
<pre class="example">echo -n 237 | recode l5/d..dump
</pre></div>

<p>If your <code>echo</code> does not grok &lsquo;<samp>-n</samp>&rsquo;, use &lsquo;<samp>echo 237\c</samp>&rsquo; instead.
Here is how to see what Unicode <code>U+03C6</code> means, while getting rid of
the title lines.
</p>
<div class="example">
<pre class="example">echo -n 0x03C6 | recode u2/x2..dump | tail +3
</pre></div>

<div class="footnote">
<hr>
<h4 class="footnotes-heading">Footnotes</h4>

<h3><a name="FOOT9" href="#DOCF9">(9)</a></h3>
<p>It is not
probable that <code>recode</code> will ever support <code>UTF-1</code>.</p>
<h3><a name="FOOT10" href="#DOCF10">(10)</a></h3>
<p>This
is when the goal charset allows for 16-bits.  For shorter charsets,
the &lsquo;<samp>--strict</samp>&rsquo; (&lsquo;<samp>-s</samp>&rsquo;) option decides what happens: either the
character is dropped, or a reversible mapping is produced on the fly.</p>
</div>
<hr>
<div class="header">
<p>
Previous: <a href="#count_002dcharacters" accesskey="p" rel="prev">count-characters</a>, Up: <a href="#Universal" accesskey="u" rel="up">Universal</a> &nbsp; [<a href="Charset-and-Surface-Index.html#SEC_Contents" title="Table of contents" rel="contents">Contents</a>][<a href="Concept-Index.html#Concept-Index" title="Index" rel="index">Index</a>]</p>
</div>



</body>
</html>
recode-doc 3.6-22 / usr / share / doc / recode-doc / Universal.html
