Our source code contains a copyright at the top of every CSS file...
/* Copyright © ... */
Every time CSS files are loaded by the Firefox Style Editor, a special character is inserted before the copyright symbol...
/* Copyright © ... */
It adds an additional special character each time the file is loaded. I do not believe this is limited to Firefox, but that's what I use at the moment for CSS dynamic styling. It's annoying to have to delete this char every time and occasionally it gets into commits and pushed.
Question: How can the special character insertion be prevented?
Instead of using copyright symbol itself, try to use its numerical number:
©
My advice is to open the files in Notepad++ and check the detected encoding, as displayed under the Encoding menu. I expect that it will read:
Encode in UTF-8
If so, apply Convert to UTF-8-BOM. It will prepend 3 magic bytes to your text file, making the UTF-8 encoding explicit. Save the files and see if it works.
Explanation
The reason for this  to appear, is that some tool is not detecting the encoding correctly and assumes it is ANSI (a.k.a. Windows-1252) or ISO 8859-1. Those one-bytes encodings and UTF-8 are very much alike for normal English texts and code files. The standard ASCII set is encoded in exactly the same way. Only special characters, like in your case, the copyright symbol, are encoded differently, using two, three of four bytes, rather than one.
Now, the copyright symbol has bytes 0xC2 0xA9 or 11000010 10101001 in UTF-8 encoding, and byte 0xA9 in ANSI encoding.
The latin capital letter A with circumflex has byte 0xC2 or 11000010 in ANSI encoding.
When 11000010 10101001 is encountered and interpreted as UTF-8, the first three bits, of the first byte, 110, in combination with the first two bits of the second byte , 10, indicate the start of a two-byte UTF-8 character. So this is the correct UTF-8 encoding of the copyright symbol.
If, however, 11000010 10101001 is encountered and interpreted as ANSI, two separate characters are seen, Â and ©.
I think it is no coincidence that the second byte of the UTF-8 encoding of © is the same as the one-byte ANSI encoding. It looks like the Latin-1 supplement is inserted in UTF-8 at exactly the same order as it has in ANSI and with the same offset, leaving the second bytes equal. E.g. a UTF-8 encoded
µ
would show up as
µ
if wrongly interpreted as ANSI.
Maybe, this was done to preserve some information about the original characters, if an encoding error were made.
Check if you have set a correct charset-meta-tag in your html head
<meta charSet="UTF-8"/>
Related
I have a character appearing over the wire that has a hex value and octal value \xb1 and \261.
This is what my header looks like:
From: "\261Central Station <sip#...>"
Looking at the ASCII table the character in the picture is "±":
What I don't understand:
If I try to test the same by passing "±Central Station" in the header I see it converted to "\xC2\xB1". Why?
How can I have "\xB1" or "\261" appearing over the wire instead of "\xC2\xB1".
e. If I try to print "\xB1" or "\261" I never see "±" being printed. But if I print "\u00b1" it prints the desired character, I'm assuming because "\u00b1" is the Unicode format.
From the page you linked to:
The extended ASCII codes (character code 128-255)
There are several different variations of the 8-bit ASCII table. The table below is according to ISO 8859-1, also called ISO Latin-1.
That's worth reading twice. The character codes 128–255 aren't ASCII (ASCII is a 7-bit encoding and ends at 127).
Assuming that you're correct that the character in question is ± (it's likely, but not guaranteed), your text could be encoded ISO 8850-1 or, as #muistooshort kindly pointed out in the comments, any of a number of other ISO 8859-X or CP-12XX (Windows-12XX) encodings. We do know, however, that the text isn't (valid) UTF-8, because 0xb1 on its own isn't a valid UTF-8 character.
If you're lucky, whatever client is sending this text specified the encoding in the Content-Type header.
As to your questions:
If I try to test the same by passing ±Central Station in header I see it get converted to \xC2\xB1. Why?
The text you're passing is in UTF-8, and the bytes that represent ± in UTF-8 are 0xC2 0xB1.
How can I have \xB1 or \261 appearing over the wire instead of \xC2\xB1?
We have no idea how you're testing this, so we can't answer this question. In general, though: Either send the text encoded as ISO 8859-1 (Encoding::ISO_8859_1 in Ruby), or whatever encoding the original text was in, or as raw bytes (Encoding::ASCII_8BIT or Encoding::BINARY, which are aliases for each other).
If I try to print \xB1 or \261 I never see ± being printed. But if I print \u00b1 it prints the desired character. (I'm assuming because \u00b1 is the unicode format but I will love If some can explain this in detail.)
That's not a question, but the reason is that \xB1 (\261) is not a valid UTF-8 character. Some interfaces will print � for invalid characters; others will simply elide them. \u00b1, on the other hand, is a valid Unicode code point, which Ruby knows how to represent in UTF-8.
Brief aside: UTF-8 (like UTF-16 and UTF-32) is a character encoding specified by the Unicode standard. U+00B1 is the Unicode code point for ±, and 0xC2 0xB1 are the bytes that represent that code point in UTF-8. In Ruby we can represent UTF-8 characters using either the Unicode code point (\u00b1) or the UTF-8 bytes (in hex: \xC2\xB1; or octal: \302\261, although I don't recommend the latter since fewer Rubyists are familiar with it).
Character encoding is a big topic, well beyond the scope of a Stack Overflow answer. For a good primer, read Joel Spolsky's "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)", and for more details on how character encoding works in Ruby read Yehuda Katz's "Encodings, Unabridged". Reading both will take you less than 30 minutes and will save you hundreds of hours of pain in the future.
I'm handling some file encoding stuff. When I learn BOM, it says The UTF-8 representation of the BOM is the byte sequence 0xEF,0xBB,0xBF, then I find the Code page layout which is a table that contains many character encoding information. What I am curious is that if there are some rules for the BOM bytes sequence, I mean, why don't use 0xEE,0xFF,0xBB or any other bytes sequence to represent UTF-8? Thanks in advance.
The BOM is specific to Unicode UTF (Unicode Transformation Format) encodings. It is the Unicode character U+FEFF ZERO WIDTH NO-BREAK SPACE encoded to a specific byte sequence according to the rules defined in the specific UTF that it is encoded in, same as for any other Unicode codepoint. What makes the BOM special is that it is the first encoded codepoint at the front of the encoded text, so you can discover which UTF was used to encode the text, if not specified out-of-band through other means.
The BOM for UTF-8 is EF BB BF, for UTF-16LE is FF FE, for UTF-32LE is FF FE 00 00, etc. They are all just different representations of the same Unicode codepoint U+FEFF.
Other encodings, like Windows-1252, which you link to, do not use a BOM and cannot encode that particular character, so there is no alternative "Windows-1252 encoding" of a BOM.
I close it and reopen it I get some letters and strange characters, especially where words have accent.
look at an example:
Este exto es la creación de mà probia autorÃa
google translation
What you see is a byte representation of your string, which is UTF-8. UTF-8 is multibyte encoding, that means that some characters (eg. those with accents) are saved as several bytes, usually starting with Ã.
Your application probably doesn't understand that the string is UTF-8 and prints it as byte sequence. You should use a different text editor which will be able to display your UTF-8 text correctly.
I am encountering a weird problem regarding the encoding of my files.
I have a site which is multilingual; Users can set this viá a dropdown on the site itself, the default value being German.
When the user logs in, some settings are being set depending on the language (charset, codepage and LCID). At this point I also want to point out, that all my files are ANSI-encoded.
Recently, I had to make some changes.
So I fire up Visual Studio 2010, edit the files in question and upload them to my server using Filezilla.
And now, all of a sudden, the German umlauts (Ää, Öö, Üü, ß) are being displayed incorrectly (something like ä) - but only on the files I opened with VS2010.
I checked the charset on the site itself and also displaying it with Response.CharSet and it was ISO-8859-1, which is correct.
So I tried some converting with notepad++, but no success.
I know that setting the charset to UTF-8 would solve this problem, but a) the charset is set from a database-value and b) it kind of messes things up in other languages.
You are displaying a utf-8 encoded file with a iso-8859-1 view. Usually you want to see just one character, but why do you see two instead of one? This is because in utf-8 a german small 'a' letter with 'two dots' is a 2-byte sequence with utf-8 (0xC3 and 0xA4). If this gets NOT displayed as utf-8 but as iso-8859-1 encoding - which means one byte one character - you'll get that what you have mentioned. You'll get the startbyte 0xC3 as a single iso-8859-1 character and the following byte 0xA4 as as a single iso-8859-1 character. In utf-8 this 2-byte sequence must become decoded by extracting the payload bits of the startbyte and the following byte like this:
Startbyte: 11000011
Following: 10100100
So 110 of the startbyte must get stripped off, so 11 is left.
So 10 of the following byte must get stripped off, so 100100 is left.
Chained together this becomes 11100100 which is decimal 228 which should be equal to the german character 'a with two dots' unicode codepoint.
I recommend to let the encoding as it is, utf-8. It is just the encoding of your viewer/editor that should display utf-8 encoded files as utf-8 and not as iso-8859-1. Configure your viewer/editor with utf-8. In other words, configure the viewer's/editor's encoding according to the encoding of the file's content (which is in your case utf-8 and NOT iso-8859-1).
To convert your files or check them for a certain encoding, just use madedit. madedit has a built-in hex-editor which wraps a rectangle around utf-8 sequences, displaying just one character on the right side (the encoded codepoint). It's easy to identify single-byte characters and/or 2/3/4-byte sequences within utf-8 encoded files. It also wraps a rectangle around the 3-byte utf-8 BOM (if any).
Encoding problems have several failure points:
Check template file encoding
Check response encoding
Check database encoding
Check that they are coherent to what you want to output.
Also note that Notepad++ has a "Encode as..." and a "Convert to..."
1st one reads file as encoding specified and 2nd reads file and writes it back to selected encoding (changing file)
I got a file full of those codes, and I want to "translate" it into normal chars (a whole file, I mean). How can I do it?
Thank you very much in advance.
Looks like you originally had a UTF-8 file which has been interpreted as an 8 bit encoding (e.g. ISO-8859-15) and entity-encoded. I say this because the sequence C3A9 looks like a pretty plausible UTF-8 encoding sequence.
You will need to first entity-decode it, then you'll have a UTF-8 encoding again. You could then use something like iconv to convert to an encoding of your choosing.
To work through your example:
à © would be decoded as the byte sequence 0xC3A9
0xC3A9 = 11000011 10101001 in binary
the leading 110 in the first octet tells us this could be interpreted as a UTF-8 two byte sequence. As the second octet starts with 10, we're looking at something we can interpret as UTF-8. To do that, we take the last 5 bits of the first octet, and the last 6 bits of the second octet...
So, interpreted as UTF8 it's 00011101001 = E9 = é (LATIN SMALL LETTER E WITH ACUTE)
You mention wanting to handle this with PHP, something like this might do it for you:
//to load from a file, use
//$file=file_get_contents("/path/to/filename.txt");
//example below uses a literal string to demonstrate technique...
$file="&Précédent is a French word";
$utf8=html_entity_decode($file);
$iso8859=utf8_decode($utf8);
//$utf8 contains "Précédent is a French word" in UTF-8
//$iso8859 contains "Précédent is a French word" in ISO-8859