Converting a UTF-16LE Elixir bitstring into an Elixir String - utf-8

Given an Elixir bitstring encoded in UTF-16LE:
<<68, 0, 101, 0, 118, 0, 97, 0, 115, 0, 116, 0, 97, 0, 116, 0, 111, 0, 114, 0, 0, 0>>
how can I get this converted into a readable Elixir String (it spells out "Devastator")? The closest I've gotten is transforming the above into a list of the Unicode codepoints (["0044", "0065", ...]) and trying to prepend the \u escape sequence to them, but Elixir throws an error since it's an invalid sequence. I'm out of ideas.

The simplest way is using functions from the :unicode module:
:unicode.characters_to_binary(utf16binary, {:utf16, :little})
For example
<<68, 0, 101, 0, 118, 0, 97, 0, 115, 0, 116, 0, 97, 0, 116, 0, 111, 0, 114, 0, 0, 0>>
|> :unicode.characters_to_binary({:utf16, :little})
|> IO.puts
#=> Devastator
(there's a null byte at the very end, so the binary display instead of string will be used in the shell, and depending on OS it may print some extra representation for the null byte)

You can make use of Elixir's pattern matching, specifically <<codepoint::utf16-little>>:
defmodule Convert do
def utf16le_to_utf8(binary), do: utf16le_to_utf8(binary, "")
defp utf16le_to_utf8(<<codepoint::utf16-little, rest::binary>>, acc) do
utf16le_to_utf8(rest, <<acc::binary, codepoint::utf8>>)
end
defp utf16le_to_utf8("", acc), do: acc
end
<<68, 0, 101, 0, 118, 0, 97, 0, 115, 0, 116, 0, 97, 0, 116, 0, 111, 0, 114, 0, 0, 0>>
|> Convert.utf16le_to_utf8
|> IO.puts
<<192, 3, 114, 0, 178, 0>>
|> Convert.utf16le_to_utf8
|> IO.puts
Output:
Devastator
πr²

Related

how to extract the raw value in a subdocument in mongo's libbson c library?

consider the mongo document: {"key1": "value1", "key2": {"subkey1": "subvalue1"}}, this can be encoded (using python bson module like: bson.encode(DATA)) to the following uint8_t array: [56, 0, 0, 0, 2, 107, 101, 121, 49, 0, 7, 0, 0, 0, 118, 97, 108, 117, 101, 49, 0, 3, 107, 101, 121, 50, 0, 28, 0, 0, 0, 2, 115, 117, 98, 107, 101, 121, 49, 0, 10, 0, 0, 0, 115, 117, 98, 118, 97, 108, 117, 101, 49, 0, 0, 0]. Now, I use this array to initialize the bson_t struct and use the iterator to find the subdocument for key2.
Here I want to create another bson_t document from this. I tried using the bson_iter_document() method but it gives me precondition failed: document error. Maybe i'm not using it right. Is there another way to do it properly?
void test_bson(){
uint8_t raw[] = {56, 0, 0, 0, 2, 107, 101, 121, 49, 0, 7, 0, 0, 0, 118, 97, 108, 117, 101, 49, 0, 3, 107, 101, 121, 50, 0, 28, 0, 0, 0, 2, 115, 117, 98, 107, 101, 121, 49, 0, 10, 0, 0, 0, 115, 117, 98, 118, 97, 108, 117, 101, 49, 0, 0, 0};
bson_t *bson;
bson = bson_new_from_data(raw, 56);
bson_iter_t iter;
bson_iter_init(&iter, bson);
if (bson_iter_find(&iter, "key2")){
printf("found. %d\n", bson_iter_type(&iter));
const uint8_t **subdocument;
uint32_t subdoclen = 56;
bson_iter_document (&iter, &subdoclen, subdocument);
}
bson_free(bson);
}

javascript: from a websocket I receive messages as zlib deflate: how to read OR "unflate" OR "deflate" (not inflate)

This question is about converting a gzip deflate message from a websocket message and convert it to array OR raw text that I can apply JSON.parse on it...
*** to be also clear: In this question : i use a websocket from a crypto exchange.... but the question is about the received message NOT about crypto exchange
in the documentation they say "please use zlib deflate"
HERE THE JAVASCRIPT
digifinexopen = '{"id":12312,"method":"trades.subscribe","params":["btc_usdt"]}';
digifinex_market_ws = new WebSocket("wss://openapi.digifinex.com/ws/v1/");
digifinex_market_ws.binaryType = "arraybuffer";
digifinex_market_ws.onmessage = event => digifinex_trades(event.data);
digifinex_market_ws.onopen = event => digifinex_market_ws.send(digifinexopen);
function fu_bitmex_trades (jsonx) { console.log(jsonx); }
I have this in the log
object=>[[Int8Array]]: Int8Array(1129) 0 … 99]
0: 120
1: -38
I tried with <script src="https://cdnjs.cloudflare.com/ajax/libs/pako/2.0.4/pako.min.js" ...></script>
if I do pako.deflate(jsonx);
I get
object=> Uint8Array(78) [120, 156, 1, 67, 0, 188, 255, 120, 218, 4, 192, 177, 13, 196, 32, 12, 133, 225, 93, 254, 154, 6, 174, 243, 54, 39, 66, 17, 201, 74, 36, 63, 187, 66, 236, 158, 111, 179, 34, 222, 192, 158, 114, 111, 196, 82, 121, 98, 27, 229, 63, 75, 24, 170, 57, 151, 196, 105, 220, 23, 214, 199, 175, 143, 243, 5, 0, 0, 255, 255, 32, 108, 18, 108, 62, 68, 31,
If I add decoder = new TextDecoder("utf8"); and log(decoder.decode(jsonx)); I get
string=> x�E��xڜ��n\7����'5���*
���$pƋ Ȼ�*�֋�g��#����|�����������������v\�//�_������������
but, HOW TO RETREIVE the array or raw data that I could json.parse ????
If I decompress your data twice, I get:
{"error":null,"result":{"status":"success"},"id":12312}
It looks like you compressed instead of decompressed. Use pako.inflate().

Why won't ruby-mcrypt accept an array as a key?

Hello I am having trouble encrypting using an array as the key and the value with the ruby-mcrypt gem. The gem lets me use an array for the key fine, cipher = Mcrypt.new("rijndael-256", :ecb, secret) works. But it will give me an error when I try to encrypt. I've tried many things but no luck. Does anyone know if Mcrypt just doesn't like encrypting with an array?
require 'mcrypt'
def encrypt(plain, secret)
cipher = Mcrypt.new("rijndael-256", :ecb, secret)
cipher.padding = :zeros
encrypted = cipher.encrypt(plain)
p encrypted
encrypted.unpack("H*").first.to_s.upcase
end
array_to_encrypt = [16, 0, 0, 0, 50, 48, 49, 55, 47, 48, 50, 47, 48, 55, 32, 50, 50, 58, 52, 54, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
key_array = [65, 66, 67, 68, 49, 50, 51, 52, 70, 71, 72, 73, 53, 54, 55, 56]
result = encrypt(array_to_encrypt, key_array)
p "RESULT IS #{result}"
The output is as follows:
Mcrypt::RuntimeError: Could not initialize mcrypt: Key length is not legal.
I traced this error to here in the ruby-mcrypt gem but don't understand it enough to figure out why I am getting the error message. Any help or insights would be amazing. Thanks!
The library doesn't support arrays. You'll need to use Strings instead:
def binary(byte_array)
byte_array.pack('C*')
end
array_to_encrypt = [16, 0, 0, 0, 50, 48, 49, 55, 47, 48, 50, 47, 48, 55, 32, 50, 50, 58, 52, 54, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
key_array = [65, 66, 67, 68, 49, 50, 51, 52, 70, 71, 72, 73, 53, 54, 55, 56]
result = encrypt(binary(array_to_encrypt), binary(key_array))
p "RESULT IS #{result}"

Print ASCII table without loop

I just had an interview and I have been asked a question: How would you print all the ASCII table characters without using a loop. The language doesn't matter.
The only method for doing so that comes to my mind is by using recursion instead of loops. An algorithm for doing this will be something like:
void printASCII(int i){
if(i == 128)
return;
print(i + " " + ((char)i) + "\n");
printASCII(i + 1);
}
You should call the previous function using:
printASCII(0);
This will print the complete ASCII table, where each line contains the index followed by a space and the actual ASCII character.
I don't think you can find any other way to do so, specially that it clearly says:
The language doesn't matter
This usually means that the question is about an algorithmic idea, rather than being specific for any language.
Two other approaches that weren't mentioned:
The obvious:
#include <stdio.h>
int main() {
printf("%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c%c", 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127);
}
The power of two tree (probably intended by the interviewer):
#include <stdio.h>
int c;
#define a128 a64; a64;
#define a64 a32; a32;
#define a32 a16; a16;
#define a16 a8; a8;
#define a8 a4; a4;
#define a4 a2; a2;
#define a2 a; a;
#define a printf("%c", c++);
int main() {
c = 0;
a128
}

How to initialize __m128i array statically in gcc?

I am porting some SSE optimization code from Windows to Linux. And I found that the following code, which works well in MSVC, won't work in GCC.
The code is to initialize an array of __m128i. Each __mi28i contains 16 int8_t. It does compile with gcc but the result is not as expected.
Actually, as gcc defines __m128i as long long int, the code will initialize an array like:
long long int coeffs_ssse3[4] = {64, 83, 64, 36}.
I googled and was told that "The only portable way to initialize a vector is to use _mm_set_XXX intrinsics." However, I want to know is there any other way to initialize the __m128i array? Better statically, and don't need to modify the following code much (since I have tons of code in the following format). Any suggestion is appreciated.
static const __m128i coeffs_ssse3[4] =
{
{ 64, 0, 64, 0, 64, 0, 64, 0, 64, 0, 64, 0, 64, 0, 64, 0},
{ 83, 0, 36, 0,-36,-1,-83,-1, 83, 0, 36, 0,-36,-1,-83, -1},
{ 64, 0,-64,-1,-64,-1, 64, 0, 64, 0,-64,-1,-64,-1, 64, 0},
{ 36, 0,-83,-1, 83, 0,-36,-1, 36, 0,-83,-1, 83, 0,-36,-1}
};
It seems that gcc doesn't treat the __m128* types as being candidates for aggregate initialization. Since they aren't standard types, this behavior will vary from compiler to compiler. One approach would be to declare the array as an aligned array of 8-bit integers, then just cast a pointer to it:
static const int8_t coeffs[64] __attribute__((aligned(16))) =
{
64, 0, 64, 0, 64, 0, 64, 0, 64, 0, 64, 0, 64, 0, 64, 0,
83, 0, 36, 0,-36,-1,-83,-1, 83, 0, 36, 0,-36,-1,-83, -1,
64, 0,-64,-1,-64,-1, 64, 0, 64, 0,-64,-1,-64,-1, 64, 0,
36, 0,-83,-1, 83, 0,-36,-1, 36, 0,-83,-1, 83, 0,-36,-1
};
static const __m128i *coeffs_ssse3 = (__m128i *) coeffs;
However, I don't think this syntax (__attribute__((aligned(x)))) is supported by Visual Studio, so you would need some #ifdef trickery in there to use the right directives to achieve the alignment that you want on all of your target platforms.

Resources