LESSCHARSET=utf-8 less doesn't seem to work - utf-8

Question

I'm trying to view a UTF-8 text file/stream in less, and even if I invoke it like this:
cat file | LESSCHARSET=utf-8 less
the non-ASCII compatible UTF-8 characters don't display correctly. Instead, their hex values appear highlighted in brackets, e.g. <F4>.
The reading the same text in vim with UTF-8 encoding poses no problems. So I'm thinking something is wrong with the way I'm invoking less.
My locale output is the following
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
My less version is the one installed by XCode on OSX Leopard:
$ less --version | sed 's/^/ /'
less 394
Copyright (C) 1984-2005 Mark Nudelman
less comes with NO WARRANTY, to the extent permitted by law.
For information about the terms of redistribution,
see the file named README in the less distribution.
Homepage: http://www.greenwoodsoftware.com/less
locale -a | grep US | sed 's/^/ /' outputs the following:
en_AU.US-ASCII
en_CA.US-ASCII
en_GB.US-ASCII
en_NZ.US-ASCII
en_US
en_US.ISO8859-1
en_US.ISO8859-15
en_US.US-ASCII
en_US.UTF-8

Answer 1

What does the locale command output? Is it a UTF-8 locale?
Are you sure your terminal is set to display UTF-8?
Does echo -e '\xe2\x82\xac' produce the € (euro) sign?
Is the locale that you have set even installed on the system? Is it
present in the list that locale -a outputs?
What version of less are you using? (Run less --version to find out.)
Really, really old versions did not even support LESSCHARSET. This
is less likely to be the case, because I have a Debian "sarge" system with
less version 382, and it does not even need LESSCHARSET if the locale is
set correctly.

Answer 2

My guess is that your file isn't UTF8 but rather ISO8859. (Is the <F4> character supposed to be a 'ô'?)
Start an xterm with LANG=en_US.ISO-8859-1 xterm. Then verify the locale (the output of locale should be something like en_US.ISO-8859-1). Then use less to view the file. Does it display correctly?
Note that it isn't enough to just use LESSCHARSET=iso8859 without starting a new terminal. LESSCHARSET tells less to think that the terminal can interpret iso8859, but your terminal probably displays UTF8, since the euro sign displays correctly. But as \xf4 isn't a valid utf8 character, the terminal will probably show something like '�'.

Answer 3

On Mac OS a charset name have to be in upper case:
bash-4.4$ less --version
less 458 (POSIX regular expressions)
Copyright (C) 1984-2012 Mark Nudelman
bash-4.4$ LESSCHARSET=cp1251 less
invalid charset name
bash-4.4$ LESSCHARSET=CP1251 less
Missing filename ("less --help" for help)
Here I found a list of charsets:
{ "ascii", NULL, "8bcccbcc18b95.b" },
{ "utf-8", &utf_mode, "8bcccbcc18b95.b126.bb" },
{ "iso8859", NULL, "8bcccbcc18b95.33b." },
{ "latin3", NULL, "8bcccbcc18b95.33b5.b8.b15.b4.b12.b18.b12.b." },
{ "arabic", NULL, "8bcccbcc18b95.33b.3b.7b2.13b.3b.b26.5b19.b" },
{ "greek", NULL, "8bcccbcc18b95.33b4.2b4.b3.b35.b44.b" },
{ "greek2005", NULL, "8bcccbcc18b95.33b14.b35.b44.b" },
{ "hebrew", NULL, "8bcccbcc18b95.33b.b29.32b28.2b2.b" },
{ "koi8-r", NULL, "8bcccbcc18b95.b." },
{ "KOI8-T", NULL, "8bcccbcc18b95.b8.b6.b8.b.b.5b7.3b4.b4.b3.b.b.3b." },
{ "georgianps", NULL, "8bcccbcc18b95.3b11.4b12.2b." },
{ "tcvn", NULL, "b..b...bcccbccbbb7.8b95.b48.5b." },
{ "TIS-620", NULL, "8bcccbcc18b95.b.4b.11b7.8b." },
{ "next", NULL, "8bcccbcc18b95.bb125.bb" },
{ "dos", NULL, "8bcccbcc12bc5b95.b." },
{ "windows-1251", NULL, "8bcccbcc12bc5b95.b24.b." },
{ "windows-1252", NULL, "8bcccbcc12bc5b95.b.b11.b.2b12.b." },
{ "windows-1255", NULL, "8bcccbcc12bc5b95.b.b8.b.5b9.b.4b." },
{ "ebcdic", NULL, "5bc6bcc7bcc41b.9b7.9b5.b..8b6.10b6.b9.7b9.8b8.17b3.3b9.7b9.8b8.6b10.b.b.b." },
{ "IBM-1047", NULL, "4cbcbc3b9cbccbccbb4c6bcc5b3cbbc4bc4bccbc191.b" },
{ NULL, NULL, NULL }
and a list of corresponding aliases for them:
{ "UTF-8", "utf-8" },
{ "ANSI_X3.4-1968", "ascii" },
{ "US-ASCII", "ascii" },
{ "latin1", "iso8859" },
{ "ISO-8859-1", "iso8859" },
{ "latin9", "iso8859" },
{ "ISO-8859-15", "iso8859" },
{ "latin2", "iso8859" },
{ "ISO-8859-2", "iso8859" },
{ "ISO-8859-3", "latin3" },
{ "latin4", "iso8859" },
{ "ISO-8859-4", "iso8859" },
{ "cyrillic", "iso8859" },
{ "ISO-8859-5", "iso8859" },
{ "ISO-8859-6", "arabic" },
{ "ISO-8859-7", "greek" },
{ "IBM9005", "greek2005" },
{ "ISO-8859-8", "hebrew" },
{ "latin5", "iso8859" },
{ "ISO-8859-9", "iso8859" },
{ "latin6", "iso8859" },
{ "ISO-8859-10", "iso8859" },
{ "latin7", "iso8859" },
{ "ISO-8859-13", "iso8859" },
{ "latin8", "iso8859" },
{ "ISO-8859-14", "iso8859" },
{ "latin10", "iso8859" },
{ "ISO-8859-16", "iso8859" },
{ "IBM437", "dos" },
{ "EBCDIC-US", "ebcdic" },
{ "IBM1047", "IBM-1047" },
{ "KOI8-R", "koi8-r" },
{ "KOI8-U", "koi8-r" },
{ "GEORGIAN-PS", "georgianps" },
{ "TCVN5712-1", "tcvn" },
{ "NEXTSTEP", "next" },
{ "windows", "windows-1252" }, /* backward compatibility */
{ "CP1251", "windows-1251" },
{ "CP1252", "windows-1252" },
{ "CP1255", "windows-1255" },
{ NULL, NULL }

Answer 4

Try the command file file.txt.  If, for example, the output is "ISO-8859 English text" then change the encoding of the file from ISO-8859 to UTF-8 via the command iconv -f ISO-8859-1 -t UTF-8 -o testfile.txt file.txt.  If less testfile.txt displays correctly, finish with mv testfile.txt file.txt.

Answer 5

less -r displays the file correctly for me.
-r or --raw-control-chars
Causes "raw" control characters to be displayed. The default is
to display control characters using the caret notation; for
example, a control-A (octal 001) is displayed as "^A". Warning:
when the -r option is used, less cannot keep track of the actual
appearance of the screen (since this depends on how the screen
responds to each type of control character). Thus, various dis-
play problems may result, such as long lines being split in the
wrong place.

Answer 6

Worked for me:
f='path/to/file/filename.extension'; LESSCHARSET=`file -b --mime-encoding ${f}|tr '[:lower:]' '[:upper:]'` less ${f}