Indexing pixels in a monochrome FreeType glyph buffer - freetype

I want to translate a monochrome FreeType glyph to an RGBA unsigned byte OpenGL texture. The colour of the texture at pixel (x, y) would be (255, 255, alpha), where
alpha = glyph->bitmap.buffer[pixelIndex(x, y)] * 255
I load my glyph using
FT_Load_Char(face, glyphChar, FT_LOAD_RENDER | FT_LOAD_MONOCHROME | FT_LOAD_TARGET_MONO)
The target texture has dimensions of glyph->bitmap.width * glyph->bitmap.rows. I've been able to index a greyscale glyph (loaded using just FT_Load_Char(face, glyphChar, FT_LOAD_RENDER)) with
glyph->bitmap.buffer[(glyph->bitmap.width * y) + x]
This does not appear work on a monochrome buffer though and the characters in my final texture are scrambled.
What is the correct way to get the value of pixel (x, y) in a monochrome glyph buffer?

Based on this thread I started on Gamedev.net, I've come up with the following function to get the filled/empty state of the pixel at (x, y):
bool glyphBit(const FT_GlyphSlot &glyph, const int x, const int y)
{
int pitch = abs(glyph->bitmap.pitch);
unsigned char *row = &glyph->bitmap.buffer[pitch * y];
char cValue = row[x >> 3];
return (cValue & (128 >> (x & 7))) != 0;
}

I have a similiar question some time ago. So I would to try help you.
The target texture has dimensions of glyph->bitmap.width * glyph->bitmap.rows
This is very specific dimension for OpenGl. Would be better if you round this to power of two.
In common way you make cycle where you get every glyph. Then cycle for every row from 0 to glyph->bitmap.rows. Then cycle for every byte (unsigned char) in row from 0 to glyph->pitch. Where you get byte by handling glyph->bitmap.buffer[pitch * row + i] (i is index of inner cycle and row is index of outer). For example:
if(s[i] == ' ') left += 20; else
for (int row = 0; row < g->bitmap.rows; ++row) {
if(kerning)
for(int b = 0; b < pitch; b++){
if(data[left + 64*(strSize*(row + 64 - g->bitmap_top)) + b] + g->bitmap.buffer[pitch * row + b] < UCHAR_MAX)
data[left + 64*(strSize*(row + 64 - g->bitmap_top)) + b] += g->bitmap.buffer[pitch * row + b];
else
data[left + 64*(strSize*(row + 64 - g->bitmap_top)) + b] = UCHAR_MAX;
} else
std::memcpy(data + left + 64*(strSize*(row + 64 - g->bitmap_top)) , g->bitmap.buffer + pitch * row, pitch);
}
left += g->advance.x >> 6;
This code is relevant to an 8-bit bitmap (standart FT_Load_Char(face, glyphChar, FT_LOAD_RENDER)).
Now I tried to use the monochrome flag and it caused me trouble. So my answer is not a solution to your problem. If you just want to display the letter then you should see my question.

The following Python function unpacks a FT_LOAD_TARGET_MONO glyph bitmap into a more convenient representation where each byte in the buffer maps to one pixel.
I've got some more info on monochrome font rendering with Python and FreeType plus additional example code on my blog: http://dbader.org/blog/monochrome-font-rendering-with-freetype-and-python
def unpack_mono_bitmap(bitmap):
"""
Unpack a freetype FT_LOAD_TARGET_MONO glyph bitmap into a bytearray where each
pixel is represented by a single byte.
"""
# Allocate a bytearray of sufficient size to hold the glyph bitmap.
data = bytearray(bitmap.rows * bitmap.width)
# Iterate over every byte in the glyph bitmap. Note that we're not
# iterating over every pixel in the resulting unpacked bitmap --
# we're iterating over the packed bytes in the input bitmap.
for y in range(bitmap.rows):
for byte_index in range(bitmap.pitch):
# Read the byte that contains the packed pixel data.
byte_value = bitmap.buffer[y * bitmap.pitch + byte_index]
# We've processed this many bits (=pixels) so far. This determines
# where we'll read the next batch of pixels from.
num_bits_done = byte_index * 8
# Pre-compute where to write the pixels that we're going
# to unpack from the current byte in the glyph bitmap.
rowstart = y * bitmap.width + byte_index * 8
# Iterate over every bit (=pixel) that's still a part of the
# output bitmap. Sometimes we're only unpacking a fraction of a byte
# because glyphs may not always fit on a byte boundary. So we make sure
# to stop if we unpack past the current row of pixels.
for bit_index in range(min(8, bitmap.width - num_bits_done)):
# Unpack the next pixel from the current glyph byte.
bit = byte_value & (1 << (7 - bit_index))
# Write the pixel to the output bytearray. We ensure that `off`
# pixels have a value of 0 and `on` pixels have a value of 1.
data[rowstart + bit_index] = 1 if bit else 0
return data

Related

RAW 12 bits per pixel data format

I was analyzing a 12 bit per pixel, GRBG, Little Endian, 1920x1280 resolution raw image but I am confused how data or RGB pixels are stored. Image size is 4915200 bytes, when calculated 4915200/(1920x1280) = 2. That means each pixel takes 2 bytes and 4 bits in 2bytes are used for padding. I tried to edit image with Hex editor but I have no idea how pixels are stored in image. Please do share if you have any idea.
Image Link
That means each pixel takes 2 bytes and 4 bits in 2bytes are used for padding
Well, sort of. It means each sample is stored in two consecutive bytes, with 4 bits of padding. But in raw images, samples usually aren't pixels, not exactly. Raw images have not been demosaiced yet, they are raw after all. For GRGB, the Bayer pattern looks like this:
What's in the file, is a 1920x1280 grid of 12+4 bit samples, arranged in the same order as pixels would have been, but each sample has only one channel, namely the one that corresponds to its position in the Bayer pattern.
Additionally, the color space is probably linear, not Gamma-compressed. The color balance is unknown unless you reverse engineer it. A proper decoder would have a calibrated color matrix, but I don't have that.
I combined these two things and guessed a color balance to do a really basic decoding (with bad demosaicing, just to demonstrate that the above information is probably accurate):
Using this C# code:
Bitmap bm = new Bitmap(1920, 1280);
for (int y = 0; y < 1280; y += 2)
{
int i = y * 1920 * 2;
for (int x = 0; x < 1920; x += 2)
{
const int stride = 1920 * 2;
int d0 = data[i] + (data[i + 1] << 8);
int d1 = data[i + 2] + (data[i + 3] << 8);
int d2 = data[i + stride] + (data[i + stride + 1] << 8);
int d3 = data[i + stride + 2] + (data[i + stride + 3] << 8);
i += 4;
int r = Math.Min((int)(Math.Sqrt(d1) * 4.5), 255);
int b = Math.Min((int)(Math.Sqrt(d2) * 9), 255);
int g0 = Math.Min((int)(Math.Sqrt(d0) * 5), 255);
int g3 = Math.Min((int)(Math.Sqrt(d3) * 5), 255);
int g1 = Math.Min((int)(Math.Sqrt((d0 + d3) * 0.5) * 5), 255);
bm.SetPixel(x, y, Color.FromArgb(r, g0, b));
bm.SetPixel(x + 1, y, Color.FromArgb(r, g1, b));
bm.SetPixel(x, y + 1, Color.FromArgb(r, g1, b));
bm.SetPixel(x + 1, y + 1, Color.FromArgb(r, g3, b));
}
}
You can load your image into a Numpy array and reshape correctly like this:
import numpy as np
# Load image and reshape
img = np.fromfile('Image_12bpp_grbg_LittleEndian_1920x1280.raw',dtype=np.uint16).reshape((1280,1920))
print(img.shape)
(1280, 1920)
Then you can demosaic and scale to get a 16-bit PNG. Note that I don't know your calibration coefficients so I guessed:
#!/usr/bin/env python3
# Demosaicing Bayer Raw image
# https://stackoverflow.com/a/68823014/2836621
import cv2
import numpy as np
filename = 'Image_12bpp_grbg_LittleEndian_1920x1280.raw'
# Set width and height
w, h = 1920, 1280
# Read mosaiced image as GRGRGR...
# BGBGBG...
bayer = np.fromfile(filename, dtype=np.uint16).reshape((h,w))
# Extract g0, g1, b, r from mosaic
g0 = bayer[0::2, 0::2] # every second pixel down and across starting at 0,0
g1 = bayer[1::2, 1::2] # every second pixel down and across starting at 1,1
r = bayer[0::2, 1::2] # every second pixel down and across starting at 0,1
b = bayer[1::2, 0::2] # every second pixel down and across starting at 1,0
# Apply (guessed) color matrix for 16-bit PNG
R = np.sqrt(r) * 1200
B = np.sqrt(b) * 2300
G = np.sqrt((g0+g1)/2) * 1300 # very crude
# Stack into 3 channel
BGR16 = np.dstack((B,G,R)).astype(np.uint16)
# Save result as 16-bit PNG
cv2.imwrite('result.png', BGR16)
Keywords: Python, raw, image processing, Bayer, de-Bayer, mosaic, demosaic, de-mosaic, GBRG, 12-bit.

HEALPix with texture UV mapping

I found an implementation of the HEALpix algorithm this is the dokumentation
And the output looks very nice.
The following images show the latitude / longitude conversion to HEALpix areas.
The x-axe goes from 0 to 2 * pi. The y-axe goes from 0 to pi. The grey color represents the HEALpix pixel encoded in grey.
Nside = 1
Nside = 2
Nside = 4
Nside = 8
The different grey values are the IDs for the texture I have to use. That means, that each HEALpix pixel represents one texture. The missing part is the UV mapping within each of the HEALpix pixels like shown below:
nSide = 1 with UV mapping
Right now I am using the function:
void ang2pix_ring( const long nside, double theta, double phi, long *ipix)
Which gives me the correct texture ID. But I've no idea how to calculate the UV mapping for each HEALpix pixel.
Is there a way to calculate all four corners in lat/lon coordinates of a HEALpix pixel? Or even better a direct calculation to the UV coordinates?
BTW: I am using the RING scheme. But if the NESTED scheme is simpler to calculate I also would change to that.
After a lot of research I came to a solution for this problem:
First of all, I've changed the scheme to NESTED. With the NESTED scheme and a very high nSide value (8192), the returned value from the
void ang2pix_ring( const long nside, double theta, double phi, long *ipix)
function gives back a long value where the UV coordinates can be read out in the following way:
Bit 26 till 30 represents the level 0 (only the 12 HEALPix pixels).
By using higher levels, the Bits from 30 till 26 - (level * 2) represents the HEALPix pixels.
The leftover 26 - (level * 2) - 1 till bit 1 encode the UV texture-coordinates in the following way:
Each second odd bit shrink together represents the U coordinate and the even once represents the V coordinate.
To normalize these UV-coordinates the responding shrinked values need to be divided by the value of pow(2, (26 - level * 2) / 2).
Code says more than 1000 words:
unsigned long ignoreEverySecondBit(unsigned long value, bool odd, unsigned int countBits)
{
unsigned long result = 0;
unsigned long mask = odd == true ? 0b1 : 0b10;
countBits = countBits / 2;
for (int i = 0; i < countBits; ++i)
{
if ((value & mask) != 0)
{
result += std::pow(2, i);
}
mask = mask << 2;
}
return result;
}
//calculate the HEALPix values:
latLonToHealPixNESTED(nSide, theta, phi, &pix);
result.level = level;
result.texture = pix >> (26 - level * 2);
result.u = static_cast<float>(ignoreEverySecondBit(pix, true, 26 - level * 2));
result.v = static_cast<float>(ignoreEverySecondBit(pix, false, 26 - level * 2));
result.u = result.u / pow(2, (26 - level * 2) / 2);
result.v = result.v / pow(2, (26 - level * 2) / 2);
And of cause a few images to show the results. The blue value represents the textureID, the red value represents the U-coordinate and the green value represents the V-coordinate:
Level 0
Level 1
Level 2
Level 3
Level 4
I hope this solution will help others too.

How to get the correct `RGB` value of a `PNG` image?

Mapbox provides Global elevation data with height data encoded in PNG image. Height is decoded by height = -10000 + ((R * 256 * 256 + G * 256 + B) * 0.1). Details are in https://www.mapbox.com/blog/terrain-rgb/.
I want to import the height data to generate terrains in Unity3D.
Texture2D dem = (Texture2D)AssetDatabase.LoadAssetAtPath("Assets/dem/12/12_3417_1536.png", typeof(Texture2D));
for (int i = 0; i < width; i++)
for (int j = 0; j < height; j++)
{
Color c = dem.GetPixel(i, j);
float R = c.r*255;
float G = c.g*255;
float B = c.b*255;
array[i, j] = -10000 + ((R * 256 * 256 + G * 256 + B) * 0.1f);
}
Here I set a break point and the rgba value of the first pixel is RGBA(0.000, 0.592, 0.718, 1.000). c.r is 0. The height is incorrect as this point represent the height of somewhere on a mountain.
Then I open the image in Photoshop and get RGB of the first pixel: R=1,G=152,B=179.
I write a test program in C#.
System.Drawing.Bitmap bitmap = new System.Drawing.Bitmap("12_3417_1536.png");
Color a = bitmap.GetPixel(0, 0);
It shows Color a is (R,G,B,A)=(1,147,249,255)
Here is the image I test:
https://api.mapbox.com/v4/mapbox.terrain-rgb/12/3417/1536.pngraw?access_token=pk.eyJ1Ijoib2xlb3RpZ2VyIiwiYSI6ImZ2cllZQ3cifQ.2yDE9wUcfO_BLiinccfOKg
Why I got different RGBA value with different method? Which one is correct?
According to the comments below, different read order and compressed data in unity may result in different value of the rgba of pixel at (0,0).
Now I want to focus on----How to convert the rgba(0~1) to RGBA(0~255)?
r_ps=r_unity*255? But how can I explain r=0 in unity and r=1 in PS of pixel at (0,0)
?
Try disabling compression from the texture's import settings in Unity (No compression). Alternatively, if you fetch the data at runtime, you can use Texture.LoadBytes() to avoid compression artifacts.
I will assume you are using the same picture and that there aren't two 12_3417_1536.png files in separate folders.
Each of these functions has a different concept of which pixel is at (0,0). Not sure what you mean by "first" pixel when you tested with photoshop, but Texture coordinates in unity start at lower left corner.
When I tested the lower left corner pixel using paint, I got the same value as you did with photoshop. However, if you test the upper left corner, you get (1,147,249,255) which is the result bitmap.GetPixel returns.
The unity values that you're getting seem to be way off. Try calling dem.GetPixel(0,0) so that you're sure you're analyzing the simplest case.

How does this MATLAB code make an image into a binary image?

v = videoinput('winvideo', 1, 'YUY2_320x240');
s = serial('COM1', 'BaudRate', 9600);
fopen(s);
while(1)
h = getsnapshot(v);
rgb = ycbcr2rgb(h);
for i = 1:240
for j = 1:320
if rgb(i,j,1) > 140 && rgb(i,j,2) < 100 % use ur own conditions
bm(i, j) = 1;
else
bm(i, j) = 0;
end
end
end
This is the code i got from my senior regarding image processing using MATLAB. The above code is to convert the image to binary image, But in the code rgb(i, j, 1) > 140 I didn't understand that command. How to select that 140 and what does that rgb(i, j, 1) mean?
You have an RGB image rgb where the third dimension are the RGB color planes. Thus, rgb(i,j,1) is the red value at row i, column j.
By doing rgb(i,j,1)>140 it tests if this red value is greater than 140. The value 140 appears to be ad hoc, picked for a specific task.
The code is extremely inefficient as there is no need for a loop:
bm = rgb(:,:,1)>140 & rgb(:,:,2)<100;
Note the change from && to the element-wise operator &. Here I'm assuming that the size of rgb is 240x320x3.
Edit: The threshold values you choose completely depend on the task, but a common approach to automatic thresholding is is Otsu's method, graythresh. You can apply it to a single color plane to get a threshold:
redThresh = graythresh(rgb(:,:,1)) * 255;
Note that graythresh returns a value on [0,1], so you have to scale that by the data range.

Consistent ways of memorizing one dimensional information in 24 bit color

Say that I have a scalar information assigned to every pixel of a 2D rectangle R, e.g. a grayscale image, or a depth-map/bump-map.
Such a scalar information is canonically encoded in a 8 bit image, which allows 2^8=256 different tones. Conveniently, tones here have a pretty intuitive meaning, e.g. white=0, black=1, gray=somewhere between 0 and 1.
Once the image is saved, e.g. in .png, the tone t, 0 <= t <= 255, is encoded in the RGB color [t,t,t] (which wastes 16bit per pixel).
Question:
Say, that the resolution provided by the 8 bit grayscale is not enough for my purpose.
Are there established ways to losslessly encode a 24bit (1D) information to the RGB color space preserving some intuitive meaning of colors?
You might want to consider a Hilbert curve. This is an embedding of a one-dimensional curve into a higher dimensional (2, 3 or more) space.
Here's what it might look like in the case of mapping a 1d curve into a two-dimensional colour space. The white curve has 2^16 = 65,536 points, and is embedded into a 2^8 x 2^8 = 256 x 256 dimensional colour space. Any two neighbouring points on the curve are very similar.
It's possible to generalize this to embed a curve into three dimensions, though I haven't got the code to hand. I can make the Matlab code that generates this plot available if you like, though I'm not convinced it will be very helpful...
This is the color scale you end up with by following the Hilbert curve through the image. Not super intuitive, but it does cover all 65,536 colors.
Edit - here's the code
function [x,y] = d2xy(n,d)
# D2XY Embeds a point d into an n*n square (assuming n is a power of 2). For
# example, if n = 8 then we can embed the points d = 0:63 into it.
x = uint32(0);
y = uint32(0);
t = uint32(d);
n = uint32(n);
s = uint32(1);
while s < n
rx = bitand(1, idivide(t, 2));
ry = bitand(1, bitxor(t,rx));
[x,y] = rot(s,x,y,rx,ry);
x = x + s * rx;
y = y + s * ry;
t = idivide(t, 4);
s = s * 2;
end
end
function [x,y] = rot(n,x,y,rx,ry)
if ry == 0
if rx == 1
x = n-1-x;
y = n-1-y;
end
# Swap x and y
t = x; x = y; y = t;
end
end
b = zeros(65536, 2);
for d = 0:65535
[x,y] = d2xy(256, d);
b(d+1,1) = x;
b(d+1,2) = y;
end
plot(b(:,1), b(:,2)), xlim([-1,256]), ylim([-1,256]), axis square
HSV or HSI color space is the exactly what you are looking for:
http://www.mathworks.com/help/matlab/ref/rgb2hsv.html
HSI means hue, saturation and illumination or intensity. Hue gives the color frequency and it is quite intuitive to represent colors
I think you could quantize to 8, i.e. distribute intensity / 8 to as equal to 3 colors (you get a grey) and add the remaining bits (intensity % 8) to LSB of separed R,G,B. We can do with a local addition. Something like (untested but compiled)
int convert(int N) {
int Q = N / 8, // Q + Z == N
Z = N % 8;
int R = (Q >> 0) & 0xFF,
G = (Q >> 8) & 0xFF,
B = (Q >> 16) & 0xFF,
S = R + ((Z >> 0) & 1),
H = G + ((Z >> 1) & 1),
C = B + ((Z >> 2) & 1);
return (S << 0) + (H << 8) + (C << 16);
}

Resources