A script I am making scans a 5-character code and assigns it a number based on the contents of characters within the code. The code is a randomly-generated number/letter combination. For example 7D3B5 or HH42B where any position can be any one of (26 + 10) characters.
Now, the issue I am having is I would like to figure out the number from 1-(36^5) based on the code. For example:
00000 = 0
00001 = 1
00002 = 2
0000A = 10
0000B = 11
0000Z = 36
00010 = 37
00011 = 38
So on and so forth until the final possible code which is:
ZZZZZ = 60466176 (36^5)
What I need to work out is a formula to figure out, let's say G47DU in its number form, using the examples below.
Something like this?
function getCount(s){
if (!isNaN(s))
return Number(s);
return s.charCodeAt(0) - 55;
}
function f(str){
let result = 0;
for (let i=0; i<str.length; i++)
result += Math.pow(36, str.length - i - 1) * getCount(str[i]);
return result;
}
var strs = [
'00000',
'00001',
'00002',
'0000A',
'0000B',
'0000Z',
'00010',
'00011',
'ZZZZZ'
];
for (str of strs)
console.log(str, f(str));
You are trying to create a base 36 numeric system. Since there are 5 'digits' each digit being 0 to Z, the value can go from 0 to 36^5. (If we are comparing this with hexadecimal system, in hexadecimal each 'digit' goes from 0 to F). Now to convert this to decimal, you could try use the same method used to convert from hex or binary etc... system to the decimal system.
It will be something like d4 * (36 ^ 4) + d3 * (36 ^ 3) + d2 * (36 ^ 2) + d1 * (36 ^ 1) + d0 * (36 ^ 0)
Note: Here 36 is the total number of symbols.
d0, d1, d2, d3, d4 can range from 0 to 35 in decimal (Important: Not 0 to 36).
Also, you can extend this for any number of digits or symbols and you can implement operations like addition, subtraction etc in this system itself as well. (It will be fun to implement that. :) ) But it will be easier to convert it to decimal do the operations and convert it back though.
I want to convert a fixed-length, say 50 character long randomized string into a 64 bit integer and be able to convert it back to original text given the 64 bit integer.
Does an algorithm exist for this? I want to go with encoding/decoding rather than hashing/reverse lookup.
just sumarization of the comments...
1:1 mapping between string and number requires enough characters and bits to store your data. Assuming 26 char alphabet only:
64bit -> 2^64 // possible numbers in 64 bits
1char -> 26 // possible characters per 1 char
so in order to get the number of chars fitting into 64 bit integer
chars = floor( 64 / (log(26)/log(2)) )
= floor( 64 / 4.7004397181410921603968126542567)
= floor( 13.6 )
= 13
if you want to know how many bits you need for 50 chars:
bits = ceil( 50 / (log(2)/log(26)) )
= ceil( 50 / 0.21274605355336315360618778415321
= ceil( 235.02198590705460801984063271284 )
= 236
Now if you want to encode 13 char (a..z) from text into 64 bit unsigned integer x:
char text[13] = "bla bla bla b";
unsigned int x,m,i;
for (i=0,x=0,m=1;i<13;i++,m*=26)
x += ((unsigned int)(text[i]-'a'))*m;
And decoding back:
for (i=0;i<13;i++)
{
text[i] = (x%26)+'a';
x /= 26;
}
As you can see its the same as converting between numbers in different bases...
In case you want to have faster dec/enc at the cost of text size you can ceil the number of bits per single character to 5 meaning floor(64/5) = 12 chars and use bits operations instead (each character would be 5 bits in the number)...
char text[12] = "bla bla bla ";
unsigned int x,i;
for (i=0,x=0,i<12;i++)
{
x <<= 5;
x |= text[i]-'a';
}
for (i=0;i<12;i++)
{
text[11-i] = (x&31)+'a';
x >>= 5;
}
However if you have any additional knowledge about the characters its possible to implement compression but only in cases where entropy allows it... for more info google RLE,Huffman encoding...
In the golang color package, there is a method to get r,g,b,a values from an RGBA object:
func (c RGBA) RGBA() (r, g, b, a uint32) {
r = uint32(c.R)
r |= r << 8
g = uint32(c.G)
g |= g << 8
b = uint32(c.B)
b |= b << 8
a = uint32(c.A)
a |= a << 8
return
}
If I were to implement this simple function, I would just write this
func (c RGBA) RGBA() (r, g, b, a uint32) {
r = uint32(c.R)
g = uint32(c.G)
b = uint32(c.B)
a = uint32(c.A)
return
}
What's the reason r |= r << 8 is used?
From the the excellent "The Go image package" blogpost:
[...] the channels have a 16-bit effective range: 100% red is represented by
RGBA returning an r of 65535, not 255, so that converting from CMYK or
YCbCr is not as lossy. Third, the type returned is uint32, even though
the maximum value is 65535, to guarantee that multiplying two values
together won't overflow.
and
Note that the R field of an RGBA is an 8-bit alpha-premultiplied color in the range [0, 255]. RGBA satisfies the Color interface by multiplying that value by 0x101 to generate a 16-bit alpha-premultiplied color in the range [0, 65535]
So if we look at the bit representation of a color with the value c.R = 10101010 then this operation
r = uint32(c.R)
r |= r << 8
effectively copies the first byte to the second byte.
00000000000000000000000010101010 (r)
| 00000000000000001010101000000000 (r << 8)
--------------------------------------
00000000000000001010101010101010 (r |= r << 8)
This is equivalent to a multiplication with the factor 0x101 and distributes all 256 possible values evenly across the range [0, 65535].
The color.RGBA type implements the RGBA method to satisfy the color.Color interface:
type Color interface {
// RGBA returns the alpha-premultiplied red, green, blue and alpha values
// for the color. Each value ranges within [0, 0xffff], but is represented
// by a uint32 so that multiplying by a blend factor up to 0xffff will not
// overflow.
//
// An alpha-premultiplied color component c has been scaled by alpha (a),
// so has valid values 0 <= c <= a.
RGBA() (r, g, b, a uint32)
}
Now the RGBA type represents the colour channels with the uint8 type, giving a range of [0, 0xff]. Simply converting these values to uint32 would not extend the range up to [0, 0xffff].
An appropriate conversion would be something like:
r = uint32((float64(c.R) / 0xff) * 0xffff)
However, they want to avoid the floating point arithmetic. Luckily 0xffff / 0xff is 0x0101, so we can simplify the expression (ignoring the type conversions for now):
r = c.R * 0x0101
= c.R * 0x0100 + c.R
= (c.R << 8) + c.R # multiply by power of 2 is equivalent to shift
= (c.R << 8) | c.R # equivalent, since bottom 8 bits of first operand are 0
And that's essentially what the code in the standard library is doing.
Converting a value in the range 0 to 255 (an 8-bit RGB component) to a value in the range 0 to 65535 (a 16-bit RGB component) would be done by multiplying the 8-bit value by 65535/255; 65535/255 is exactly 257, which is hex 101, so multiplying a one-byte by 65535/255 can be done by shifting that byte value left 8 bits and ORing it with the original value.
(There's nothing Go-specific about this; similar tricks are done elsewhere, in other languages, when converting 8-bit RGB/RGBA components to 16-bit RGB/RGBA components.)
To convert from 8- to 16-bits per RGB component, copy the byte into the high byte of the 16-bit value. e.g., 0x03 becomes 0x0303, 0xFE becomes 0xFEFE, so that the 8-bit values 0 through 255 (0xFF) produce 16-bit values 0 to 65,535 (0xFFFF) with an even distribution of values.
I'm attempting to recreate a PKCS #5 Padding algorithm I've found written in python.
The main line I'm struggling to recreate is this
return data + (chr(pad_count) * pad_count).encode('utf-8')
which essentially repeats pad_count (an integer, between 1 and 16), as a char, between 1 and 16 times. I'm having trouble getting a similar result in Go.
For example, pad_count of 11 will return the string
\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b\x0b
The closeset I've come is this:
b := make([]byte, 2)
binary.LittleEndian.PutUint16(b, uint16(padCount))
fmt.Println("Pad: ", padCount, "Hex: ", hex.EncodeToString(b))
which will return:
Pad: 11 Hex: 0b00
This is pretty close, and obviously I could take a substring, and add the \x myself, but is there a better way to go about this? Also if I substring, I feel there is no guarantee that would work for all the combinations.
As James Henstridge already mentioned the formatting you want (\x0b...) is not something that's required but rather python's representation of non-printable characters. See for yourself:
>>> chr(3)
'\x03'
What you have to do is defined in RFC2898:
[...] where the padding string PS consists of 8-(||M|| mod 8) octets
each with value 8-(||M|| mod 8). The padding string PS will
satisfy one of the following statements:
PS = 01, if ||M|| mod 8 = 7 ;
PS = 02 02, if ||M|| mod 8 = 6 ;
...
PS = 08 08 08 08 08 08 08 08, if ||M|| mod 8 = 0.
This means that you do not need uint16 but uint8 (since an octet has only 8 bits) and you also do not need to format your bytes the way python does. So the only thing you have to do is to use bytes.Repeat:
bytes.Repeat(paddingChar, paddingCount)
func pad(input []byte, pad_count int) []byte {
out := make([]byte, len(input) + int(pad_count))
copy(out, input)
for i := 0; i < pad_count; i++ {
out[len(input) + i] = byte(pad_count)
}
return out
}
I’ve got my hands on a 16-bit rgb565 image (specifically, an Android framebuffer dump), and I would like to convert it to 24-bit rgb888 for viewing on a normal monitor.
The question is, how does one convert a 5- or 6-bit channel to 8 bits? The obvious answer is to shift it. I started out by writing this:
puts("P6 320 480 255");
uint16_t buf;
while (read(0, &buf, sizeof buf)) {
unsigned char red = (buf & 0xf800) >> 11;
unsigned char green = (buf & 0x07e0) >> 5;
unsigned char blue = buf & 0x001f;
putchar(red << 3);
putchar(green << 2);
putchar(blue << 3);
}
However, this doesn’t have one property I would like, which is for 0xffff to map to 0xffffff, instead of 0xf8fcf8. I need to expand the value in some way, but I’m not sure how that should work.
The Android SDK comes with a tool called ddms (Dalvik Debug Monitor) that takes screen captures. As far as I can tell from reading the code, it implements the same logic; yet its screenshots are coming out different, and white is mapping to white.
Here’s the raw framebuffer, the smart conversion by ddms, and the dumb conversion by the above algorithm. Note that the latter is slightly darker and greener.
(By the way, this conversion is implemented in ffmpeg, but it’s just performing the dumb conversion listed above, leaving the LSBs at all zero.)
I guess I have two questions:
What’s the most sensible way to convert rgb565 to rgb888?
How is DDMS converting its screenshots?
You want to map each of these from a 5/6 bit space to an 8 bit space.
5 bits = 32 values
6 bits = 64 values
8 bits = 256 values
The code you're using is taking the naive approach that x5 * 256/32 = x8 where 256/32 = 8 and multiplying by 8 is left shift 3 but, as you say, this doesn't necessarily fill the new number space "correctly". 5 to 8 for max value is 31 to 255 and therein lies your clue to the solution.
x8 = 255/31 * x5
x8 = 255/63 * x6
where x5, x6 and x8 are 5, 6 and 8 bit values respectively.
Now there is a question about the best way to implement this. It does involve division and with integer division you will lose any remainder result (round down basically) so the best solution is probably to do floating point arithmetic and then round half up back to an integer.
This can be sped up considerably by simply using this formula to generate a lookup table for each of the 5 and 6 bit conversions.
My few cents:
If you care about precise mapping, yet fast algorithm you can consider this:
R8 = ( R5 * 527 + 23 ) >> 6;
G8 = ( G6 * 259 + 33 ) >> 6;
B8 = ( B5 * 527 + 23 ) >> 6;
It uses only: MUL, ADD and SHR -> so it is pretty fast!
From the other side it is compatible in 100% to floating point mapping with proper rounding:
// R8 = (int) floor( R5 * 255.0 / 31.0 + 0.5);
// G8 = (int) floor( G6 * 255.0 / 63.0 + 0.5);
// B8 = (int) floor( R5 * 255.0 / 31.0 + 0.5);
Some extra cents:
If you are interested in 888 to 565 conversion, this works very well too:
R5 = ( R8 * 249 + 1014 ) >> 11;
G6 = ( G8 * 253 + 505 ) >> 10;
B5 = ( B8 * 249 + 1014 ) >> 11;
Constants were found using brute force search with somę early rejections to speed thing up a bit.
You could shift and then or with the most significant bits; i.e.
Red 10101 becomes 10101000 | 101 => 10101101
12345 12345--- 123 12345123
This has the property you seek, but it's not the most linear mapping of values from one space to the other. It's fast, though. :)
Cletus' answer is more complete and probably better. :)
iOS vImage Conversion
The iOS Accelerate Framework documents the following algorithm for the vImageConvert_RGB565toARGB8888 function:
Pixel8 alpha = alpha
Pixel8 red = (5bitRedChannel * 255 + 15) / 31
Pixel8 green = (6bitGreenChannel * 255 + 31) / 63
Pixel8 blue = (5bitBlueChannel * 255 + 15) / 31
For a one-off conversion this will be fast enough, but if you want to process many frames you want to use something like the iOS vImage conversion or implement this yourself using NEON intrinsics.
From ARMs Community Forum Tutorial
First, we will look at converting RGB565 to RGB888. We assume there are eight 16-bit pixels in register q0, and we would like to separate reds, greens and blues into 8-bit elements across three registers d2 to d4.
vshr.u8 q1, q0, #3 # shift red elements right by three bits,
# discarding the green bits at the bottom of
# the red 8-bit elements.
vshrn.i16 d2, q1, #5 # shift red elements right and narrow,
# discarding the blue and green bits.
vshrn.i16 d3, q0, #5 # shift green elements right and narrow,
# discarding the blue bits and some red bits
# due to narrowing.
vshl.i8 d3, d3, #2 # shift green elements left, discarding the
# remaining red bits, and placing green bits
# in the correct place.
vshl.i16 q0, q0, #3 # shift blue elements left to most-significant
# bits of 8-bit color channel.
vmovn.i16 d4, q0 # remove remaining red and green bits by
# narrowing to 8 bits.
The effects of each instruction are described in the comments above, but in summary, the operation performed on each channel is:
Remove color data for adjacent channels using shifts to push the bits off either end of the element.
Use a second shift to position the color data in the most-significant bits of each element, and narrow to reduce element size from 16 to eight bits.
Note the use of element sizes in this sequence to address 8 and 16 bit elements, in order to achieve some of the masking operations.
A small problem
You may notice that, if you use the code above to convert to RGB888 format, your whites aren't quite white. This is because, for each channel, the lowest two or three bits are zero, rather than one; a white represented in RGB565 as (0x1F, 0x3F, 0x1F) becomes (0xF8, 0xFC, 0xF8) in RGB888. This can be fixed using shift with insert to place some of the most-significant bits into the lower bits.
For an Android specific example I found a YUV-to-RGB conversion written in intrinsics.
Try this:
red5 = (buf & 0xF800) >> 11;
red8 = (red5 << 3) | (red5 >> 2);
This will map all zeros into all zeros, all 1's into all 1's, and everything in between into everything in between. You can make it more efficient by shifting the bits into place in one step:
redmask = (buf & 0xF800);
rgb888 = (redmask << 8) | ((redmask<<3)&0x070000) | /* green, blue */
Do likewise for green and blue (for 6 bits, shift left 2 and right 4 respectively in the top method).
The general solution is to treat the numbers as binary fractions - thus, the 6 bit number 63/63 is the same as the 8 bit number 255/255. You can calculate this using floating point math initially, then compute a lookup table, as other posters suggest. This also has the advantage of being more intuitive than bit-bashing solutions. :)
There is an error jleedev !!!
unsigned char green = (buf & 0x07c0) >> 5;
unsigned char blue = buf & 0x003f;
the good code
unsigned char green = (buf & 0x07e0) >> 5;
unsigned char blue = buf & 0x001f;
Cheers,
Andy
I used the following and got good results. Turned out my Logitek cam was 16bit RGB555 and using the following to convert to 24bit RGB888 allowed me to save as a jpeg using the smaller animals ijg: Thanks for the hint found here on stackoverflow.
// Convert a 16 bit inbuf array to a 24 bit outbuf array
BOOL JpegFile::ByteConvert(BYTE* inbuf, BYTE* outbuf, UINT width, UINT height)
{ UINT row_cnt, pix_cnt;
ULONG off1 = 0, off2 = 0;
BYTE tbi1, tbi2, R5, G5, B5, R8, G8, B8;
if (inbuf==NULL)
return FALSE;
for (row_cnt = 0; row_cnt <= height; row_cnt++)
{ off1 = row_cnt * width * 2;
off2 = row_cnt * width * 3;
for(pix_cnt=0; pix_cnt < width; pix_cnt++)
{ tbi1 = inbuf[off1 + (pix_cnt * 2)];
tbi2 = inbuf[off1 + (pix_cnt * 2) + 1];
B5 = tbi1 & 0x1F;
G5 = (((tbi1 & 0xE0) >> 5) | ((tbi2 & 0x03) << 3)) & 0x1F;
R5 = (tbi2 >> 2) & 0x1F;
R8 = ( R5 * 527 + 23 ) >> 6;
G8 = ( G5 * 527 + 23 ) >> 6;
B8 = ( B5 * 527 + 23 ) >> 6;
outbuf[off2 + (pix_cnt * 3)] = R8;
outbuf[off2 + (pix_cnt * 3) + 1] = G8;
outbuf[off2 + (pix_cnt * 3) + 2] = B8;
}
}
return TRUE;
}
Here's the code:
namespace convert565888
{
inline uvec4_t const _c0{ { { 527u, 259u, 527u, 1u } } };
inline uvec4_t const _c1{ { { 23u, 33u, 23u, 0u } } };
} // end ns
uvec4_v const __vectorcall rgb565_to_888(uvec4_v const rgba) {
return(uvec4_v(_mm_srli_epi32(_mm_add_epi32(_mm_mullo_epi32(rgba.v,
uvec4_v(convert565888::_c0).v), uvec4_v(convert565888::_c1).v), 6)));
}
and for rgb 888 to 565 conversion:
namespace convert888565
{
inline uvec4_t const _c0{ { { 249u, 509u, 249u, 1u } } };
inline uvec4_t const _c1{ { { 1014u, 253u, 1014u, 0u } } };
} // end ns
uvec4_v const __vectorcall rgb888_to_565(uvec4_v const rgba) {
return(uvec4_v(_mm_srli_epi32(_mm_add_epi32(_mm_mullo_epi32(rgba.v,
uvec4_v(convert888565::_c0).v), uvec4_v(convert888565::_c1).v), 11)));
}
for the explanation of where all these numbers come from, specifically how I calculated the optimal multiplier and bias for green:
Desmos graph -
https://www.desmos.com/calculator/3grykboay1
The graph isn't the greatest but it shows the actual value vs. error -- play around with the interactive sliders to see how different values affect the output. This graph also applies to calculating the red and blue values aswell. Typically green is shifted by 10bits, red and blue 11bits.
In order for this to work with intrinsic _mm_srli_epi32 / _mm_srl_epi32 requires all components to be shifted by the same amount. So everything is shifted by 11 bits (rgb888_to_565) in this version, however, the green component is scaled to compensate for this change. Fortunately, it scales perfectly!
I had this difficulty too, and the most faithful way I found was to replace the 16-bit value with the original 24-bit value. Now the ILI9341 screen color is visually compatible with Notebook screen. I thought of just using the 24-bit color table, but then the display routines would have to be converted to 565, and that would make the program even slower.
If the color palette is fixed as in my case, it might be the most viable option. I tried to make use of the 3 MSB adding with the 3 LSB, but it wasn't very good.
The colors I used on the ILI9341 display I got from this website (Note: I choose the 24-bit color 888 and get the 16-bit color 565, on this website there's no way to do otherwise):
http://www.barth-dev.de/online/rgb565-color-picker/
For example, I read the pixel color of the ILI9341 display and save it to a USB Disk, in a file, in BMP format. As the display operates with 16-bit or 18-bit, I have no way to retrieve 24-bit information directly from the GRAM memory.
#define BLACK_565 0x0000
#define BLUE_565 0x001F
#define RED_565 0xF800
#define GREEN_565 0x07E0
#define CYAN_565 0x07FF
#define MAGENTA_565 0xF81F
#define YELLOW_565 0xFFE0
#define WHITE_565 0xFFFF
#define LIGHTGREY_565 0xC618
#define ORANGE_565 0xFD20
#define GREY_565 0x8410
#define DARKGREY_565 0x2104
#define DARKBLUE_565 0x0010
#define DARKGREEN_565 0x03E0
#define DARKCYAN_565 0x03EF
#define DARKYELLOW_565 0x8C40
#define BLUESKY_565 0x047F
#define BROWN_565 0xC408
#define BLACK_888 0x000000
#define BLUE_888 0x0000FF
#define RED_888 0xFF0000
#define GREEN_888 0x04FF00
#define CYAN_888 0x00FFFB
#define MAGENTA_888 0xFF00FA
#define YELLOW_888 0xFBFF00
#define WHITE_888 0xFFFFFF
#define LIGHTGREY_888 0xC6C3C6
#define ORANGE_888 0xFFA500
#define GREY_888 0x808080
#define DARKGREY_888 0x202020
#define DARKBLUE_888 0x000080
#define DARKGREEN_888 0x007D00
#define DARKCYAN_888 0x007D7B
#define DARKYELLOW_888 0x898A00
#define BLUESKY_888 0x008CFF
#define BROWN_888 0xC08240
I did the test (using an STM32F407 uC) with an IF statement, but it can also be done with Select Case, or another form of comparison.
uint16_t buff1; // pixel color value read from GRAM
uint8_t buff2[3];
uint32_t color_buff; // to save to USB disk
if (buff1 == BLUE_565) color_buff = BLUE_888;
else if (buff1 == RED_565) color_buff = RED_888;
else if (buff1 == GREEN_565) color_buff = GREEN_888;
else if (buff1 == CYAN_565) color_buff = CYAN_888;
else if (buff1 == MAGENTA_565) color_buff = MAGENTA_888;
else if (buff1 == YELLOW_565) color_buff = YELLOW_888;
else if (buff1 == WHITE_565) color_buff = WHITE_888;
else if (buff1 == LIGHTGREY_565) color_buff = LIGHTGREY_888;
else if (buff1 == ORANGE_565) color_buff = ORANGE_888;
else if (buff1 == GREY_565) color_buff = GREY_888;
else if (buff1 == DARKGREY_565) color_buff = DARKGREY_888;
else if (buff1 == DARKBLUE_565) color_buff = DARKBLUE_888;
else if (buff1 == DARKCYAN_565) color_buff = DARKCYAN_888;
else if (buff1 == DARKYELLOW_565) color_buff = DARKYELLOW_888;
else if (buff1 == BLUESKY_565) color_buff = BLUESKY_888;
else if (buff1 == BROWN_565) color_buff = BROWN_888;
else color_buff = BLACK;
RGB separation for saving to 8-bit variables:
buff2[0] = color_buff; // Blue
buff2[1] = color_buff >> 8; // Green
buff2[2] = color_buff >> 16; // Red