Sort a table with UTF-8 encoded values alphabetically - sorting

Question

I am storing dictionary entries in a Lua table, using it as an array. I want to sort the entries from Lua, so that I can add new ones without having to move to the correct position myself (which gets quite tedious soon). However, I am facing several problems:
Many words contain non-ASCII characters, which makes the built-in comparison operator for strings unsuitable for the task (for instance, it makes amputar come before ámbito).
There are words from various languages (all Western, though), namely Spanish, German and English. The problem here is that, probably, different languages have different notions of the alphabetical order. Since the main language is Spanish, I would like to use its rules, although I'm unsure as to whether that will work with characters not contained in the Spanish alphabet.
Some words contain capital letters, or, even worse, start with them. For example, all German nouns start with upper-case letters. By the built-in comparison operator, the capital letters come before their lower-case siblings, which is not my desired behaviour; I would like upper-case letters to be treated exactly as their lower-case counterparts.
Take, for example, the following table:
local entries =
{
'amputar',
'Volksgeist',
'ámbito'
}
Those entries should be ordered like this:
ámbito
amputar
Volksgeist
However, with my current code, the output is wrong:
local function compare_utf8_strings( o1 , o2 )
-- Using the built-in non-UTF-8-aware non-locale-aware string comparison operator
return o1 < o2
end
table.sort( entries , function ( a , b ) return compare_utf8_strings( a , b ) end )
for i, entry in ipairs(entries) do
print( entry )
end
That outputs:
Volksgeist
amputar
ámbito
Could you please take the following code, and hack it to fulfill my requirements?
local entries =
{
'amputar',
'Volksgeist',
'ámbito'
}
local function compare_utf8_strings( o1 , o2 )
-- Hack here, please, accomplishing my requirements
end
table.sort( entries , function ( a , b ) return compare_utf8_strings( a , b ) end )
for i, entry in ipairs(entries) do
print( entry )
end
It should output this:
ámbito
amputar
Volksgeist
As an additional requirement, this Lua code is all inside LuaTeX, which currently supports 5.2 version of the language. As for external libraries, I guess it's possible to use them.
I am a novice in the Lua camp, so, please, forgive any error I have made, and feel free to notify it, so I fix it.

Answer 1

After some time searching to no avail, I found this article by Joseph Wright. Although it touched my issue, it didn't provide a clear solution to follow. I asked him, and it turned out that there's currently no direct way to do what I want. He pointed out, however, that slnunicode comes built-in with LuaTeX (albeit it will be replaced in the future).
I developed a 'crude' solution using the facilities provided in the LuaTeX environment. It isn't elegant, but it works, and it doesn't pull any external dependencies. About its efficiency, I have not perceived any difference in the document build time.
-- Make the facilities available
unicode = require( 'unicode' )
utf8 = unicode.utf8
--[[
Each character's position in this array-like table determines its 'priority'.
Several characters in the same slot have the same 'priority'.
]]
local alphabet =
{
-- The space is here because of other requirements of my project
{ ' ' },
{ 'a', 'á', 'à', 'ä' },
{ 'b' },
{ 'c' },
{ 'd' },
{ 'e', 'é', 'è', 'ë' },
{ 'f' },
{ 'g' },
{ 'h' },
{ 'i', 'í', 'ì', 'ï' },
{ 'j' },
{ 'k' },
{ 'l' },
{ 'm' },
{ 'n' },
{ 'ñ' },
{ 'o', 'ó', 'ò', 'ö' },
{ 'p' },
{ 'q' },
{ 'r' },
{ 's' },
{ 't' },
{ 'u', 'ú', 'ù', 'ü' },
{ 'v' },
{ 'w' },
{ 'x' },
{ 'y' },
{ 'z' }
}
-- Looks up the character `character´ in the alphabet and returns its 'priority'
local function get_pos_in_alphabet( character )
for i, alphabet_entry in ipairs(alphabet) do
for _, alphabet_char in ipairs(alphabet_entry) do
if character == alphabet_char then
return i
end
end
end
--[[
If it isn't in the alphabet, abort: it's better than silently outputting some
random garbage, and, thanks to the message, allows to add the character to
the table.
]]
assert( false , "'" .. character .. "' was not in alphabet" )
end
-- Returns the characters in the UTF-8-encoded string `s´ in an array-like table
local function get_utf8_string_characters( s )
--[[
I saw this variable being used in several code snippets around the Web, but
it isn't provided in my LuaTeX environment; I use this form of initialization
to be safe if it's defined in the future.
]]
utf8.charpattern = utf8.charpattern or "([%z\1-\127\194-\244][\128-\191]*)"
local characters = {}
for character in s:gmatch(utf8.charpattern) do
table.insert( characters , character )
end
return characters
end
local function compare_utf8_strings( _o1 , _o2 )
--[[
`o1_chars´ and `o2_chars´ are array-like tables containing all of the
characters of each string, which are all made lower-case using the
slnunicode facilities that come built-in with LuaTeX.
]]
local o1_chars = get_utf8_string_characters( utf8.lower(_o1) )
local o2_chars = get_utf8_string_characters( utf8.lower(_o2) )
local o1_len = utf8.len(o1)
local o2_len = utf8.len(o2)
for i = 1, math.min( o1_len , o2_len ) do
o1_pos = get_pos_in_alphabet( o1_chars[i] )
o2_pos = get_pos_in_alphabet( o2_chars[i] )
if o1_pos > o2_pos then
return false
elseif o1_pos < o2_pos then
return true
end
end
return o1_len < o2_len
end
I cannot integrate this solution in the question's framework because my test environment, the ZeroBrane Studio Lua IDE, doesn't come with slnunicode and I don't know how to add it.
That was it. If anyone has any doubt or would like further explanations, please, use the comments. I hope it's useful to someone else.