Converting UTF-8 iso-8859 bytes - String.to_charlist - utf-8

Question

I have run into an odd problem. I have a program that receives data in bytes from an external source, which I in return perform some business logic and send a reply. We have come into an odd issue with String.to_charlist.
Try this in iex:
String.to_charlist <<169, 99, 111, 114, 94, 51, 94, 51, 94, 66, 52, 57, 49, 49, 49, 32, 32, 49, 48, 51, 53, 94, 67, 79, 77, 80, 76, 69, 84, 69, 68, 94, 49, 49, 52, 50, 52, 53, 94, 75>>
This will throw the following error:
** (UnicodeConversionError) invalid encoding starting at <<169, 99, 111, 114, 94, 51, 94, 51, 94, 66, 52, 57, 49, 49, 49, 32, 32, 49, 48, 51, 53, 94, 67, 79, 77, 80, 76, 69, 84, 69, 68, 94, 49, 49, 52, 50, 52, 53, 94, 75>>
(elixir) lib/string.ex:2035: String.to_charlist/1
Now if you strip off the first byte and in turn try:
String.to_charlist <<99, 111, 114, 94, 51, 94, 51, 94, 66, 52, 57, 49, 49, 49, 32, 32, 49, 48, 51, 53, 94, 67, 79, 77, 80, 76, 69, 84, 69, 68, 94, 49, 49, 52, 50, 52, 53, 94, 75>>
You get:
'cor^3^3^B49111 1035^COMPLETED^114245^K'
Is there a different way I should be converting these bytes to a string? I understand some characters like 169 may not be displayable, but what is the recommended way of handling such?
Thanks for your help.
I did find this:
<<169 :: utf8, 0>>
Returns <<194, 169, 0>> which added the 194. So if you paste this into iex the output appears to be correct.
<<194, 169, 99, 111, 114, 94, 51, 94, 51, 94, 66, 52, 57, 49, 49, 49, 32, 32, 49, 48, 51, 53, 94, 67, 79, 77, 80, 76, 69, 84, 69, 68, 94, 49, 49, 52, 50, 52, 53, 94, 75>>
Do I need to write a function that loops through the bytes and calls <> then reduces on the bytes returned (except for the 0 concat)?

Answer 1

String.to_charlist/1 is redundant in the first place:
String.to_charlist <<99, 111, 114, 94, 51, 94, 51, 94>>
works only because utf8 and latin1 share the same codepoints in the interval 1–127. The below would be enough to get a perfectly valid binary:
<<99, 111, 114, 94, 51, 94, 51, 94>>
#⇒ "cor^3^3^"
Unfortunately, what you receive is not in utf8 encoding and Elixir has no built-in tools to convert binaries between encoding. You cannot just drop meaningful symbols.
I would suggest using codepagex package for that conversion:
Codepagex.from_string(<<99, 111, 114, 94, 51, 94, 51, 94>>, :iso_8859_1)
#⇒ "cor^3^3^"
Other way round would be to use erlang’s unicode.characters_to_binary/2:
:unicode.characters_to_binary(
<<169, 99, 111, 114, 94, 51, 94, 51, 94>>, :latin1, :utf8
)
#⇒ "©cor^3^3^"
Another solution using Kernel.SpecialForms.for/1 comprehension:
to_string(for <<c :: 8 <- <<169, 99, 111, 114, 94, 51, 94, 51, 94>> >>, do: c)
#⇒ "©cor^3^3^"