encoding.html 19.4 KB
Newer Older
1
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd">
2
3
<html>
<head>
4
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
5
<link rel="SHORTCUT ICON" href="/favicon.ico">
6
<style type="text/css"><!--
7
8
9
10
11
TD {font-family: Verdana,Arial,Helvetica}
BODY {font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em}
H1 {font-family: Verdana,Arial,Helvetica}
H2 {font-family: Verdana,Arial,Helvetica}
H3 {font-family: Verdana,Arial,Helvetica}
12
13
14
A:link, A:visited, A:active { text-decoration: underline }
--></style>
<title>Encodings support</title>
15
</head>
16
17
18
<body bgcolor="#8b7765" text="#000000" link="#000000" vlink="#000000">
<table border="0" width="100%" cellpadding="5" cellspacing="0" align="center"><tr>
<td width="180">
19
<a href="http://www.gnome.org/"><img src="gnome2.png" alt="Gnome2 Logo"></a><a href="http://www.w3.org/Status"><img src="w3c.png" alt="W3C Logo"></a><a href="http://www.redhat.com/"><img src="redhat.gif" alt="Red Hat Logo"></a><div align="left"><a href="http://xmlsoft.org/"><img src="Libxml2-Logo-180x168.gif" alt="Made with Libxml2 Logo"></a></div>
20
21
22
23
24
25
26
27
28
29
</td>
<td><table border="0" width="90%" cellpadding="2" cellspacing="0" align="center" bgcolor="#000000"><tr><td><table width="100%" border="0" cellspacing="1" cellpadding="3" bgcolor="#fffacd"><tr><td align="center">
<h1>The XML C library for Gnome</h1>
<h2>Encodings support</h2>
</td></tr></table></td></tr></table></td>
</tr></table>
<table border="0" cellpadding="4" cellspacing="0" width="100%" align="center"><tr><td bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="2" width="100%"><tr>
<td valign="top" width="200" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td>
<table width="100%" border="0" cellspacing="1" cellpadding="3">
<tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Main Menu</b></center></td></tr>
30
31
32
33
34
<tr><td bgcolor="#fffacd">
<form action="search.php" enctype="application/x-www-form-urlencoded" method="GET">
<input name="query" type="TEXT" size="20" value=""><input name="submit" type="submit" value="Search ...">
</form>
<ul>
35
36
37
38
39
40
41
42
<li><a href="index.html">Home</a></li>
<li><a href="intro.html">Introduction</a></li>
<li><a href="FAQ.html">FAQ</a></li>
<li><a href="docs.html">Documentation</a></li>
<li><a href="bugs.html">Reporting bugs and getting help</a></li>
<li><a href="help.html">How to help</a></li>
<li><a href="downloads.html">Downloads</a></li>
<li><a href="news.html">News</a></li>
43
<li><a href="XMLinfo.html">XML</a></li>
44
<li><a href="XSLT.html">XSLT</a></li>
45
<li><a href="python.html">Python and bindings</a></li>
46
47
48
49
50
51
52
53
54
55
56
57
<li><a href="architecture.html">libxml architecture</a></li>
<li><a href="tree.html">The tree output</a></li>
<li><a href="interface.html">The SAX interface</a></li>
<li><a href="xmldtd.html">Validation &amp; DTDs</a></li>
<li><a href="xmlmem.html">Memory Management</a></li>
<li><a href="encoding.html">Encodings support</a></li>
<li><a href="xmlio.html">I/O Interfaces</a></li>
<li><a href="catalog.html">Catalog support</a></li>
<li><a href="library.html">The parser interfaces</a></li>
<li><a href="entities.html">Entities or no entities</a></li>
<li><a href="namespaces.html">Namespaces</a></li>
<li><a href="upgrade.html">Upgrading 1.x code</a></li>
58
<li><a href="threads.html">Thread safety</a></li>
59
60
61
<li><a href="DOM.html">DOM Principles</a></li>
<li><a href="example.html">A real example</a></li>
<li><a href="contribs.html">Contributions</a></li>
62
<li><a href="xmlreader.html">The Reader Interface</a></li>
63
<li><a href="tutorial/index.html">Tutorial</a></li>
64
<li><a href="guidelines.html">XML Guidelines</a></li>
65
66
67
<li>
<a href="xml.html">flat page</a>, <a href="site.xsl">stylesheet</a>
</li>
68
69
</ul>
</td></tr>
70
71
</table>
<table width="100%" border="0" cellspacing="1" cellpadding="3">
72
<tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>Related links</b></center></td></tr>
73
<tr><td bgcolor="#fffacd"><ul>
74
75
<li><a href="http://mail.gnome.org/archives/xml/">Mail archive</a></li>
<li><a href="http://xmlsoft.org/XSLT/">XSLT libxslt</a></li>
76
<li><a href="http://phd.cs.unibo.it/gdome2/">DOM gdome2</a></li>
77
<li><a href="http://www.aleksey.com/xmlsec/">XML-DSig xmlsec</a></li>
78
<li><a href="ftp://xmlsoft.org/">FTP</a></li>
79
<li><a href="http://www.zlatkovic.com/projects/libxml/">Windows binaries</a></li>
Daniel Veillard's avatar
Daniel Veillard committed
80
<li><a href="http://garypennington.net/libxml2/">Solaris binaries</a></li>
81
<li><a href="http://www.zveno.com/open_source/libxml2xslt.html">MacOsX binaries</a></li>
82
<li><a href="http://sourceforge.net/projects/libxml2-pas/">Pascal bindings</a></li>
83
<li><a href="http://bugzilla.gnome.org/buglist.cgi?product=libxml&amp;product=libxml2">Bug Tracker</a></li>
84
85
86
87
88
89
90
91
92
93
</ul></td></tr>
</table>
<table width="100%" border="0" cellspacing="1" cellpadding="3">
<tr><td colspan="1" bgcolor="#eecfa1" align="center"><center><b>API Indexes</b></center></td></tr>
<tr><td bgcolor="#fffacd"><ul>
<li><a href="APIchunk0.html">Alphabetic</a></li>
<li><a href="APIconstructors.html">Constructors</a></li>
<li><a href="APIfunctions.html">Functions/Types</a></li>
<li><a href="APIfiles.html">Modules</a></li>
<li><a href="APIsymbols.html">Symbols</a></li>
94
95
96
97
</ul></td></tr>
</table>
</td></tr></table></td>
<td valign="top" bgcolor="#8b7765"><table border="0" cellspacing="0" cellpadding="1" width="100%"><tr><td><table border="0" cellspacing="0" cellpadding="1" width="100%" bgcolor="#000000"><tr><td><table border="0" cellpadding="3" cellspacing="1" width="100%"><tr><td bgcolor="#fffacd">
98
99
<p>Table of Content:</p>
<ol>
100
101
<li><a href="encoding.html#What">What does internationalization support
    mean ?</a></li>
102
  <li><a href="encoding.html#internal">The internal encoding, how and
103
  why</a></li>
104
105
106
  <li><a href="encoding.html#implemente">How is it implemented ?</a></li>
  <li><a href="encoding.html#Default">Default supported encodings</a></li>
  <li><a href="encoding.html#extend">How to extend the existing
107
  support</a></li>
108
</ol>
109
<h3><a name="What">What does internationalization support mean ?</a></h3>
110
111
112
<p>If you are not really familiar with Internationalization (usual shorcut is
I18N) , Unicode, characters and glyphs, I suggest you read a <a href="http://www.tbray.org/ongoing/When/200x/2003/04/06/Unicode">presentation</a>
by Tim Bray on Unicode and why you should care about it.</p>
113
114
115
<p>XML was designed from the start to allow the support of any character set
by using Unicode. Any conformant XML parser has to support the UTF-8 and
UTF-16 default encodings which can both express the full unicode ranges. UTF8
116
117
is a variable length encoding whose greatest points are to reuse the same
encoding for ASCII and to save space for Western encodings, but it is a bit
118
more complex to handle in practice. UTF-16 use 2 bytes per characters (and
119
120
121
sometimes combines two pairs), it makes implementation easier, but looks a
bit overkill for Western languages encoding. Moreover the XML specification
allows document to be encoded in other encodings at the condition that they
122
are clearly labeled as such. For example the following is a wellformed XML
123
document encoded in ISO-8859 1 and using accentuated letter that we French
124
likes for both markup and content:</p>
125
<pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;ISO-8859-1&quot;?&gt;
126
&lt;trs&gt;l&lt;/trs&gt;</pre>
127
<p>Having internationalization support in libxml means the following:</p>
128
<ul>
129
<li>the document is properly parsed</li>
130
131
132
133
  <li>informations about it's encoding are saved</li>
  <li>it can be modified</li>
  <li>it can be saved in its original encoding</li>
  <li>it can also be saved in another encoding supported by libxml (for
134
135
136
137
138
139
    example straight UTF8 or even an ASCII form)</li>
</ul>
<p>Another very important point is that the whole libxml API, with the
exception of a few routines to read with a specific encoding or save to a
specific encoding, is completely agnostic about the original encoding of the
document.</p>
140
<p>It should be noted too that the HTML parser embedded in libxml now obey
141
142
the same rules too, the following document will be (as of 2.2.2) handled  in
an internationalized fashion by libxml too:</p>
143
144
145
<pre>&lt;!DOCTYPE HTML PUBLIC &quot;-//W3C//DTD HTML 4.0 Transitional//EN&quot;
                      &quot;http://www.w3.org/TR/REC-html40/loose.dtd&quot;&gt;
&lt;html lang=&quot;fr&quot;&gt;
146
&lt;head&gt;
147
  &lt;META HTTP-EQUIV=&quot;Content-Type&quot; CONTENT=&quot;text/html; charset=ISO-8859-1&quot;&gt;
148
149
150
151
&lt;/head&gt;
&lt;body&gt;
&lt;p&gt;W3C cre des standards pour le Web.&lt;/body&gt;
&lt;/html&gt;</pre>
152
<h3><a name="internal">The internal encoding, how and why</a></h3>
153
154
155
156
<p>One of the core decision was to force all documents to be converted to a
default internal encoding, and that encoding to be UTF-8, here are the
rationale for those choices:</p>
<ul>
157
<li>keeping the native encoding in the internal form would force the libxml
158
    users (or the code associated) to be fully aware of the encoding of the
159
160
    original document, for examples when adding a text node to a document,
    the content would have to be provided in the document encoding, i.e. the
161
162
163
    client code would have to check it before hand, make sure it's conformant
    to the encoding, etc ... Very hard in practice, though in some specific
    cases this may make sense.</li>
164
  <li>the second decision was which encoding. From the XML spec only UTF8 and
165
    UTF16 really makes sense as being the two only encodings for which there
166
    is mandatory support. UCS-4 (32 bits fixed size encoding) could be
167
168
169
170
    considered an intelligent choice too since it's a direct Unicode mapping
    support. I selected UTF-8 on the basis of efficiency and compatibility
    with surrounding software:
    <ul>
171
<li>UTF-8 while a bit more complex to convert from/to (i.e. slightly
172
173
174
175
176
177
178
179
180
        more costly to import and export CPU wise) is also far more compact
        than UTF-16 (and UCS-4) for a majority of the documents I see it used
        for right now (RPM RDF catalogs, advogato data, various configuration
        file formats, etc.) and the key point for today's computer
        architecture is efficient uses of caches. If one nearly double the
        memory requirement to store the same amount of data, this will trash
        caches (main memory/external caches/internal caches) and my take is
        that this harms the system far more than the CPU requirements needed
        for the conversion to UTF-8</li>
181
      <li>Most of libxml version 1 users were using it with straight ASCII
182
183
184
        most of the time, doing the conversion with an internal encoding
        requiring all their code to be rewritten was a serious show-stopper
        for using UTF-16 or UCS-4.</li>
185
      <li>UTF-8 is being used as the de-facto internal encoding standard for
186
187
        related code like the <a href="http://www.pango.org/">pango</a>
        upcoming Gnome text widget, and a lot of Unix code (yep another place
188
189
        where Unix programmer base takes a different approach from Microsoft
        - they are using UTF-16)</li>
190
    </ul>
191
</li>
192
193
194
</ul>
<p>What does this mean in practice for the libxml user:</p>
<ul>
195
196
197
<li>xmlChar, the libxml data type is a byte, those bytes must be assembled
    as UTF-8 valid strings. The proper way to terminate an xmlChar * string
    is simply to append 0 byte, as usual.</li>
198
  <li>One just need to make sure that when using chars outside the ASCII set,
199
200
    the values has been properly converted to UTF-8</li>
</ul>
201
<h3><a name="implemente">How is it implemented ?</a></h3>
202
203
204
205
206
<p>Let's describe how all this works within libxml, basically the I18N
(internationalization) support get triggered only during I/O operation, i.e.
when reading a document or saving one. Let's look first at the reading
sequence:</p>
<ol>
207
<li>when a document is processed, we usually don't know the encoding, a
208
209
    simple heuristic allows to detect UTF-18 and UCS-4 from whose where the
    ASCII range (0-0x7F) maps with ASCII</li>
210
  <li>the xml declaration if available is parsed, including the encoding
211
212
    declaration. At that point, if the autodetected encoding is different
    from the one declared a call to xmlSwitchEncoding() is issued.</li>
213
  <li>If there is no encoding declaration, then the input has to be in either
214
215
216
217
218
219
220
221
222
223
    UTF-8 or UTF-16, if it is not then at some point when processing the
    input, the converter/checker of UTF-8 form will raise an encoding error.
    You may end-up with a garbled document, or no document at all ! Example:
    <pre>~/XML -&gt; ./xmllint err.xml 
err.xml:1: error: Input is not proper UTF-8, indicate encoding !
&lt;trs&gt;l&lt;/trs&gt;
   ^
err.xml:1: error: Bytes: 0xE8 0x73 0x3E 0x6C
&lt;trs&gt;l&lt;/trs&gt;
   ^</pre>
224
225
  </li>
  <li>xmlSwitchEncoding() does an encoding name lookup, canonicalize it, and
226
227
228
229
230
231
    then search the default registered encoding converters for that encoding.
    If it's not within the default set and iconv() support has been compiled
    it, it will ask iconv for such an encoder. If this fails then the parser
    will report an error and stops processing:
    <pre>~/XML -&gt; ./xmllint err2.xml 
err2.xml:1: error: Unsupported encoding UnsupportedEnc
232
&lt;?xml version=&quot;1.0&quot; encoding=&quot;UnsupportedEnc&quot;?&gt;
233
                                             ^</pre>
234
235
  </li>
  <li>From that point the encoder processes progressively the input (it is
236
237
238
239
240
241
    plugged as a front-end to the I/O module) for that entity. It captures
    and convert on-the-fly the document to be parsed to UTF-8. The parser
    itself just does UTF-8 checking of this input and process it
    transparently. The only difference is that the encoding information has
    been added to the parsing context (more precisely to the input
    corresponding to this entity).</li>
242
  <li>The result (when using DOM) is an internal form completely in UTF-8
243
    with just an encoding information on the document node.</li>
244
</ol>
245
246
<p>Ok then what happens when saving the document (assuming you
collected/built an xmlDoc DOM like structure) ? It depends on the function
247
248
249
250
called, xmlSaveFile() will just try to save in the original encoding, while
xmlSaveFileTo() and xmlSaveFileEnc() can optionally save to a given
encoding:</p>
<ol>
251
<li>if no encoding is given, libxml will look for an encoding value
252
253
254
    associated to the document and if it exists will try to save to that
    encoding,
    <p>otherwise everything is written in the internal form, i.e. UTF-8</p>
255
256
  </li>
  <li>so if an encoding was specified, either at the API level or on the
257
    document, libxml will again canonicalize the encoding name, lookup for a
258
259
    converter in the registered set or through iconv. If not found the
    function will return an error code</li>
260
  <li>the converter is placed before the I/O buffer layer, as another kind of
261
262
263
    buffer, then libxml will simply push the UTF-8 serialization to through
    that buffer, which will then progressively be converted and pushed onto
    the I/O layer.</li>
264
  <li>It is possible that the converter code fails on some input, for example
265
    trying to push an UTF-8 encoded Chinese character through the UTF-8 to
266
    ISO-8859-1 converter won't work. Since the encoders are progressive they
267
268
269
    will just report the error and the number of bytes converted, at that
    point libxml will decode the offending character, remove it from the
    buffer and replace it with the associated charRef encoding &amp;#123; and
270
    resume the conversion. This guarantees that any document will be saved
271
    without losses (except for markup names where this is not legal, this is
272
    a problem in the current version, in practice avoid using non-ascii
273
    characters for tags or attributes names  @@). A special &quot;ascii&quot; encoding
274
275
    name is used to save documents to a pure ascii form can be used when
    portability is really crucial</li>
276
277
278
</ol>
<p>Here is a few examples based on the same test document:</p>
<pre>~/XML -&gt; ./xmllint isolat1 
279
&lt;?xml version=&quot;1.0&quot; encoding=&quot;ISO-8859-1&quot;?&gt;
280
281
&lt;trs&gt;l&lt;/trs&gt;
~/XML -&gt; ./xmllint --encode UTF-8 isolat1 
282
&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
283
284
&lt;très&gt;l &lt;/très&gt;
~/XML -&gt; </pre>
285
<p>The same processing is applied (and reuse most of the code) for HTML I18N
286
processing. Looking up and modifying the content encoding is a bit more
287
288
difficult since it is located in a &lt;meta&gt; tag under the &lt;head&gt;,
so a couple of functions htmlGetMetaEncoding() and htmlSetMetaEncoding() have
289
been provided. The parser also attempts to switch encoding on the fly when
290
291
292
293
294
detecting such a tag on input. Except for that the processing is the same
(and again reuses the same code).</p>
<h3><a name="Default">Default supported encodings</a></h3>
<p>libxml has a set of default converters for the following encodings
(located in encoding.c):</p>
295
<ol>
296
<li>UTF-8 is supported by default (null handlers)</li>
297
298
299
300
  <li>UTF-16, both little and big endian</li>
  <li>ISO-Latin-1 (ISO-8859-1) covering most western languages</li>
  <li>ASCII, useful mostly for saving</li>
  <li>HTML, a specific handler for the conversion of UTF-8 to ASCII with HTML
301
302
    predefined entities like &amp;copy; for the Copyright sign.</li>
</ol>
303
304
<p>More over when compiled on an Unix platform with iconv support the full
set of encodings supported by iconv can be instantly be used by libxml. On a
305
306
307
308
309
310
311
312
313
314
linux machine with glibc-2.1 the list of supported encodings and aliases fill
3 full pages, and include UCS-4, the full set of ISO-Latin encodings, and the
various Japanese ones.</p>
<h4>Encoding aliases</h4>
<p>From 2.2.3, libxml has support to register encoding names aliases. The
goal is to be able to parse document whose encoding is supported but where
the name differs (for example from the default set of names accepted by
iconv). The following functions allow to register and handle new aliases for
existing encodings. Once registered libxml will automatically lookup the
aliases when handling a document:</p>
315
<ul>
316
<li>int xmlAddEncodingAlias(const char *name, const char *alias);</li>
317
318
319
  <li>int xmlDelEncodingAlias(const char *alias);</li>
  <li>const char * xmlGetEncodingAlias(const char *alias);</li>
  <li>void xmlCleanupEncodingAliases(void);</li>
320
</ul>
321
<h3><a name="extend">How to extend the existing support</a></h3>
322
323
324
325
326
327
328
329
330
<p>Well adding support for new encoding, or overriding one of the encoders
(assuming it is buggy) should not be hard, just write an input and output
conversion routines to/from UTF-8, and register them using
xmlNewCharEncodingHandler(name, xxxToUTF8, UTF8Toxxx),  and they will be
called automatically if the parser(s) encounter such an encoding name
(register it uppercase, this will help). The description of the encoders,
their arguments and expected return values are described in the encoding.h
header.</p>
<p>A quick note on the topic of subverting the parser to use a different
331
332
333
334
335
336
337
internal encoding than UTF-8, in some case people will absolutely want to
keep the internal encoding different, I think it's still possible (but the
encoding must be compliant with ASCII on the same subrange) though I didn't
tried it. The key is to override the default conversion routines (by
registering null encoders/decoders for your charsets), and bypass the UTF-8
checking of the parser by setting the parser context charset
(ctxt-&gt;charset) to something different than XML_CHAR_ENCODING_UTF8, but
338
there is no guarantee that this will work. You may also have some troubles
339
saving back.</p>
340
341
342
<p>Basically proper I18N support is important, this requires at least
libxml-2.0.0, but a lot of features and corrections are really available only
starting 2.2.</p>
343
<p><a href="bugs.html">Daniel Veillard</a></p>
344
345
</td></tr></table></td></tr></table></td></tr></table></td>
</tr></table></td></tr></table>
346
347
</body>
</html>