Decode values ignored in CGCreateImage - cocoa

I am creating a monochrome image with the following code:
CGColorSpaceRef cgColorSpace = CGColorSpaceCreateDeviceGray();
CGImageRef cgImage = CGImageCreate (width, height, 1, 1, rowBytes, colorSpace, 0, dataProvider, decodeValues, NO, kCGRenderingIntentDefault);
where decodeValues is an array of 2 CGFloat's, equal to {0,1}. This gives me a fine image, but apparently my data (which comes from a PDF image mask) is black-on-white instead of white-on-black. To invert the image, I tried to set the values of decodeValues to {1,0}, but this did not change anything at all. Actually, whatever nonsensical values I put into decodeValues, I get the same image.
Why is decodeValues ignored here? How do I invert black and white?

here's some code for creating and drawing a mono image. It's the same as yours but with more context (and without necessary cleanup):
size_t width = 200;
size_t height = 200;
size_t bitsPerComponent = 1;
size_t componentsPerPixel = 1;
size_t bitsPerPixel = bitsPerComponent * componentsPerPixel;
size_t bytesPerRow = (width * bitsPerPixel + 7)/8;
CGColorSpaceRef colorspace = CGColorSpaceCreateDeviceGray();
CGBitmapInfo bitmapInfo = kCGImageAlphaNone;
CGFloat decode[] = {0.0, 1.0};
size_t dataLength = bytesPerRow * height;
UInt32 *bitmap = malloc( dataLength );
memset( bitmap, 255, dataLength );
CGDataProviderRef dataProvider = CGDataProviderCreateWithData( NULL, bitmap, dataLength, NULL);
CGImageRef cgImage = CGImageCreate (
width,
height,
bitsPerComponent,
bitsPerPixel,
bytesPerRow,
colorspace,
bitmapInfo,
dataProvider,
decode,
false,
kCGRenderingIntentDefault
);
CGRect destRect = CGRectMake(0, 0, width, height);
CGContextDrawImage( context, destRect, cgImage );
If i change the decode array to CGFloat decode[] = {0.0, 0.0}; i always get a black image.
If you have tried that and it didn't have any effect (you say you get the same image whatever values you use), either: you aren't actually passing in those values but you think you are, or: somehow you aren't actually examining the output of CGImageCreate.

Related

Rendering Windows screenshot capture bitmap as DirectX texture

I'm making progress developing a '3d desktop' directx app that needs to display the current contents of a desktop window (e.g. "Calculator") as a 2D texture on a rectangular surface in directx (11). I'm sooo close but really struggling with the screenshot BMP -> Texture2D step. I do have screenshot->HBITMAP and DDSFile->rendered texture successfully working but can't complete the screenshot->rendered texture.
So far I have working the 'capture the window as a screenshot' bit:
RECT user_window_rectangle;
HWND user_window = FindWindow(NULL, TEXT("Calculator"));
GetClientRect(user_window, &user_window_rectangle);
HDC hdcScreen = GetDC(NULL);
HDC hdc = CreateCompatibleDC(hdcScreen);
UINT screenshot_width = user_window_rectangle.right - user_window_rectangle.left;
UINT screenshot_height = user_window_rectangle.bottom - user_window_rectangle.top;
hbmp = CreateCompatibleBitmap(hdcScreen, screenshot_width, screenshot_height);
SelectObject(hdc, hbmp);
PrintWindow(user_window, hdc, PW_CLIENTONLY);
At this point I have the window bitmap referenced by HBITMAP hbmp.
Also working is my code to render a DDS file as a texture on a directx/3d rectangle:
ID3D11Device *dev;
ID3D11DeviceContext *dev_context;
...
dev_context->PSSetShaderResources(0, 1, &shader_resource_view);
dev_context->PSSetSamplers(0, 1, &tex_sampler_state);
...
DirectX::TexMetadata tex_metadata;
DirectX::ScratchImage image;
hr = LoadFromDDSFile(L"Earth.dds", DirectX::DDS_FLAGS_NONE, &tex_metadata, image);
hr = CreateShaderResourceView(dev, image.GetImages(), image.GetImageCount(), tex_metadata, &shader_resource_view);
Pixel shader is:
Texture2D ObjTexture
SamplerState ObjSamplerState
float4 PShader(float4 pos : SV_POSITION, float4 color : COLOR, float2 tex : TEXCOORD) : SV_TARGET\
{
return ObjTexture.Sample( ObjSamplerState, tex );
}
The samplerstate (defaulting to linear) is:
D3D11_SAMPLER_DESC sampler_desc;
ZeroMemory(&sampler_desc, sizeof(sampler_desc));
sampler_desc.AddressU = D3D11_TEXTURE_ADDRESS_WRAP;
sampler_desc.AddressV = D3D11_TEXTURE_ADDRESS_WRAP;
sampler_desc.AddressW = D3D11_TEXTURE_ADDRESS_WRAP;
sampler_desc.MinLOD = 0;
sampler_desc.MaxLOD = D3D11_FLOAT32_MAX;
hr = dev->CreateSamplerState(&sampler_desc, &tex_sampler_state);
Question: how do I replace the LoadFromDDSFile bit with some equivalent that takes the HBITMAP from the windows screencapture and ends up with it on the graphics card as ObjTexture ?
Below is my best shot of bridging from the screenshot HBITMAP hbmp to the shader resource screenshot_texture, but it gives a memory access violation from the graphics driver (I think due to my "data.pSysmem = &bmp.bmBits", but no idea really):
GetObject(hbmp, sizeof(BITMAP), (LPSTR)&bmp)
D3D11_TEXTURE2D_DESC screenshot_desc = CD3D11_TEXTURE2D_DESC(DXGI_FORMAT_R8G8B8A8_UNORM, bmp.bmWidth, bmp.bmHeight, 1,
1,
D3D11_BIND_SHADER_RESOURCE
);
int bytes_per_pixel = 4;
D3D11_SUBRESOURCE_DATA data;
ZeroMemory(&data, sizeof(D3D11_SUBRESOURCE_DATA));
data.pSysMem = &bmp.bmBits; //pixel buffer
data.SysMemPitch = bytes_per_pixel * bmp.bmWidth;// line size in byte
data.SysMemSlicePitch = bytes_per_pixel * bmp.bmWidth * bmp.bmHeight;// total buffer size in byte
hr = dev->CreateTexture2D(
&screenshot_desc, //texture format
&data, // pixel buffer use to fill the texture
&screenshot_texture // created texture
);
:::::::::::::::::::::::::SOLUTION::::::::::::::::::::::::::::::::::::::::::
The main issue was trying to use &bmp.bmBits directly as a pixel buffer caused memory conflicts within the graphics driver - this was resolved by using 'malloc' to allocate an appropriately sized block of memory to store the pixel data. Thanks to Chuck Walbourn for helping with my poking around in the dark to work out how the pixel data is actually stored (it was actually 32 bits/pixel by default). It's still possible/likely some of code is relying on luck to read the pixel data correctly, but it's been improved with Chuck's input.
My basic technique was;
FindWindow to get the client window on the desktop
CreateCompatibleBitmap and SelectObject and PrintWindow to get a HBITMAP to the snapshot
malloc to allocate the correct amount of space for a (byte*)pixel buffer
GetDIBits to populate the (byte*)pixel buffer from the HBITMAP
CreateTexture2D to build the texture buffer
CreateShaderResourceView to map the texture to the graphics pixel shader
So working code to screenshot a windows desktop window and pass that as a texture to a direct3d app is:
RECT user_window_rectangle;
HWND user_window = FindWindow(NULL, TEXT("Calculator")); //the window can't be min
if (user_window == NULL)
{
MessageBoxA(NULL, "Can't find Calculator", "Camvas", MB_OK);
return;
}
GetClientRect(user_window, &user_window_rectangle);
//create
HDC hdcScreen = GetDC(NULL);
HDC hdc = CreateCompatibleDC(hdcScreen);
UINT screenshot_width = user_window_rectangle.right - user_window_rectangle.left;
UINT screenshot_height = user_window_rectangle.bottom - user_window_rectangle.top;
hbmp = CreateCompatibleBitmap(hdcScreen, screenshot_width, screenshot_height);
SelectObject(hdc, hbmp);
//Print to memory hdc
PrintWindow(user_window, hdc, PW_CLIENTONLY);
BITMAPINFOHEADER bmih;
ZeroMemory(&bmih, sizeof(BITMAPINFOHEADER));
bmih.biSize = sizeof(BITMAPINFOHEADER);
bmih.biPlanes = 1;
bmih.biBitCount = 32;
bmih.biWidth = screenshot_width;
bmih.biHeight = 0-screenshot_height;
bmih.biCompression = BI_RGB;
bmih.biSizeImage = 0;
int bytes_per_pixel = bmih.biBitCount / 8;
BYTE *pixels = (BYTE*)malloc(bytes_per_pixel * screenshot_width * screenshot_height);
BITMAPINFO bmi = { 0 };
bmi.bmiHeader = bmih;
int row_count = GetDIBits(hdc, hbmp, 0, screenshot_height, pixels, &bmi, DIB_RGB_COLORS);
D3D11_TEXTURE2D_DESC screenshot_desc = CD3D11_TEXTURE2D_DESC(
DXGI_FORMAT_B8G8R8A8_UNORM, // format
screenshot_width, // width
screenshot_height, // height
1, // arraySize
1, // mipLevels
D3D11_BIND_SHADER_RESOURCE, // bindFlags
D3D11_USAGE_DYNAMIC, // usage
D3D11_CPU_ACCESS_WRITE, // cpuaccessFlags
1, // sampleCount
0, // sampleQuality
0 // miscFlags
);
D3D11_SUBRESOURCE_DATA data;
ZeroMemory(&data, sizeof(D3D11_SUBRESOURCE_DATA));
data.pSysMem = pixels; // texArray; // &bmp.bmBits; //pixel buffer
data.SysMemPitch = bytes_per_pixel * screenshot_width;// line size in byte
data.SysMemSlicePitch = bytes_per_pixel * screenshot_width * screenshot_height;
hr = dev->CreateTexture2D(
&screenshot_desc, //texture format
&data, // pixel buffer use to fill the texture
&screenshot_texture // created texture
);
D3D11_SHADER_RESOURCE_VIEW_DESC srvDesc;
srvDesc.Format = screenshot_desc.Format;
srvDesc.ViewDimension = D3D11_SRV_DIMENSION_TEXTURE2D;
srvDesc.Texture2D.MostDetailedMip = 0;
srvDesc.Texture2D.MostDetailedMip = screenshot_desc.MipLevels;
dev->CreateShaderResourceView(screenshot_texture, NULL, &shader_resource_view);
You are making a lot of assumptions here that the BITMAP returned is actually in 32-bit RGBA form. It is likely not at all in that format, and in any case you need to validate the contents of bmPlanes to be 1 and bmBitsPixel to be 32 if you are assuming it is 4-bytes per pixel. You should read more about the BMP format.
BMPs uses BGRA order, so you can use DXGI_FORMAT_B8G8R8A8_UNORM for the case of bmBitsPixel being 32.
Secondly, you need to derive pitch from bmWidthBytes and not bmWidth.
data.pSysMem = &bmp.bmBits; //pixel buffer
data.SysMemPitch = bmp.bmWidthBytes;// line size in byte
data.SysMemSlicePitch = bmp.bmWidthBytes * bmp.bmHeight;// total buffer size in byte
If bmBitsPixel is 24, there is no DXGI format equivalent to that. You have to copy the data to a 32-bit format such as DXGI_FORMAT_B8G8R8X8_UNORM.
If bmBitsPixel is 15 or 16, you can use DXGI_FORMAT_B5G5R5A1_UNORM on a system with Direct3D 11.1, but remember that 16-bit DXGI formats are not always supported depending on the driver. Otherwise you'll have to convert this data to something else.
For bmBitsPixel values of 1, 2, 4, or 8 you have to convert them as there are no DXGI texture formats that are equivalent.
The main issue was trying to use &bmp.bmBits directly as a pixel buffer caused memory conflicts within the graphics driver - this was resolved by using 'malloc' to allocate an appropriately sized block of memory to store the pixel data. Thanks to Chuck Walbourn for helping with my poking around in the dark to work out how the pixel data is actually stored (it was actually 32 bits/pixel by default). It's still possible/likely some of code is relying on luck to read the pixel data correctly, but it's been improved with Chuck's input.
My basic technique was;
FindWindow to get the client window on the desktop
CreateCompatibleBitmap and SelectObject and PrintWindow to get a HBITMAP to the snapshot
malloc to allocate the correct amount of space for a (byte*)pixel buffer
GetDIBits to populate the (byte*)pixel buffer from the HBITMAP
CreateTexture2D to build the texture buffer
CreateShaderResourceView to map the texture to the graphics pixel shader
So working code to screenshot a windows desktop window and pass that as a texture to a direct3d app is:
RECT user_window_rectangle;
HWND user_window = FindWindow(NULL, TEXT("Calculator")); //the window can't be min
if (user_window == NULL)
{
MessageBoxA(NULL, "Can't find Calculator", "Camvas", MB_OK);
return;
}
GetClientRect(user_window, &user_window_rectangle);
//create
HDC hdcScreen = GetDC(NULL);
HDC hdc = CreateCompatibleDC(hdcScreen);
UINT screenshot_width = user_window_rectangle.right - user_window_rectangle.left;
UINT screenshot_height = user_window_rectangle.bottom - user_window_rectangle.top;
hbmp = CreateCompatibleBitmap(hdcScreen, screenshot_width, screenshot_height);
SelectObject(hdc, hbmp);
//Print to memory hdc
PrintWindow(user_window, hdc, PW_CLIENTONLY);
BITMAPINFOHEADER bmih;
ZeroMemory(&bmih, sizeof(BITMAPINFOHEADER));
bmih.biSize = sizeof(BITMAPINFOHEADER);
bmih.biPlanes = 1;
bmih.biBitCount = 32;
bmih.biWidth = screenshot_width;
bmih.biHeight = 0-screenshot_height;
bmih.biCompression = BI_RGB;
bmih.biSizeImage = 0;
int bytes_per_pixel = bmih.biBitCount / 8;
BYTE *pixels = (BYTE*)malloc(bytes_per_pixel * screenshot_width * screenshot_height);
BITMAPINFO bmi = { 0 };
bmi.bmiHeader = bmih;
int row_count = GetDIBits(hdc, hbmp, 0, screenshot_height, pixels, &bmi, DIB_RGB_COLORS);
D3D11_TEXTURE2D_DESC screenshot_desc = CD3D11_TEXTURE2D_DESC(
DXGI_FORMAT_B8G8R8A8_UNORM, // format
screenshot_width, // width
screenshot_height, // height
1, // arraySize
1, // mipLevels
D3D11_BIND_SHADER_RESOURCE, // bindFlags
D3D11_USAGE_DYNAMIC, // usage
D3D11_CPU_ACCESS_WRITE, // cpuaccessFlags
1, // sampleCount
0, // sampleQuality
0 // miscFlags
);
D3D11_SUBRESOURCE_DATA data;
ZeroMemory(&data, sizeof(D3D11_SUBRESOURCE_DATA));
data.pSysMem = pixels; // texArray; // &bmp.bmBits; //pixel buffer
data.SysMemPitch = bytes_per_pixel * screenshot_width;// line size in byte
data.SysMemSlicePitch = bytes_per_pixel * screenshot_width * screenshot_height;
hr = dev->CreateTexture2D(
&screenshot_desc, //texture format
&data, // pixel buffer use to fill the texture
&screenshot_texture // created texture
);
D3D11_SHADER_RESOURCE_VIEW_DESC srvDesc;
srvDesc.Format = screenshot_desc.Format;
srvDesc.ViewDimension = D3D11_SRV_DIMENSION_TEXTURE2D;
srvDesc.Texture2D.MostDetailedMip = 0;
srvDesc.Texture2D.MostDetailedMip = screenshot_desc.MipLevels;
dev->CreateShaderResourceView(screenshot_texture, NULL, &shader_resource_view);

How to iterate through all pixels of an UIImage?

Hey Guys i am currently trying to iterate through all pixels of an UIImage but the way i implemented it it takes sooo much time. So i thought it is the wrong way i implemented it.
This is my method how i get the RGBA Values of an Pixel :
+(NSArray*)getRGBAsFromImage:(UIImage*)image atX:(int)xx andY:(int)yy count:(int)count
{
// Initializing the result array
NSMutableArray *result = [NSMutableArray arrayWithCapacity:count];
// First get the image into your data buffer
CGImageRef imageRef = [image CGImage]; // creating an Instance of
NSUInteger width = CGImageGetWidth(imageRef); // Get width of our Image
NSUInteger height = CGImageGetHeight(imageRef); // Get height of our Image
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB(); // creating our colour Space
// Getting that raw Data out of an image
unsigned char *rawData = (unsigned char*) calloc(height * width * 4, sizeof(unsigned char));
NSUInteger bytesPerPixel = 4; // Bytes per pixel defined
NSUInteger bytesPerRow = bytesPerPixel * width; // Bytes per row
NSUInteger bitsPerComponent = 8; // Bytes per component
CGContextRef context = CGBitmapContextCreate(rawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace); // releasing the color space
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
CGContextRelease(context);
// Now your rawData contains the image data in the RGBA8888 pixel format.
int byteIndex = (bytesPerRow * yy) + xx * bytesPerPixel;
for (int ii = 0 ; ii < count ; ++ii)
{
CGFloat red = (rawData[byteIndex] * 1.0) / 255.0;
CGFloat green = (rawData[byteIndex + 1] * 1.0) / 255.0;
CGFloat blue = (rawData[byteIndex + 2] * 1.0) / 255.0;
CGFloat alpha = (rawData[byteIndex + 3] * 1.0) / 255.0;
byteIndex += 4;
UIColor *acolor = [UIColor colorWithRed:red green:green blue:blue alpha:alpha];
[result addObject:acolor];
}
free(rawData);
return result;
}
And this is the code how i parse through all the pixels :
for (NSUInteger y = 0 ; y < self.originalPictureWidth; y++) {
for (NSUInteger x = 0 ; x < self.originalPictureHeight; x++) {
NSArray * originalRGBA = [ComputerVisionHelperClass getRGBAsFromImage:self.originalPicture atX:(int)x andY:(int)y count:1];
NSArray * referenceRGBA = [ComputerVisionHelperClass getRGBAsFromImage:self.referencePicture atX:(int)referenceIndexX andY:(int)referenceIndexY count:1];
// Do something else ....
}
}
Is there a faster way of getting all RGBA values of an uiimage instance ?
For every pixel, you're generating a new copy of the image and then throwing it away. Yes, it would be much faster by just getting the data once and then processing on that byte array.
But it heavily depends on what is in "Do something else." There are many CoreImage and vImage functions that can do image processing very quickly, but you may need to approach the problem differently. It depends on what you're doing.

Taking snapshot of contents in CGL?

I want to create a image out of Core OpenGL context.
I used following code but it creates a black image. So I guess I cannot use glReadPixles there? Any other suggestions please?
int myDataLength = 480 * 480 * 4;
// allocate array and read pixels into it.
GLubyte *buffer = (GLubyte *) malloc(myDataLength);
glReadPixels(0, 0, 320, 480, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
// gl renders "upside down" so swap top to bottom into new array.
// there's gotta be a better way, but this works.
GLubyte *buffer2 = (GLubyte *) malloc(myDataLength);
for(int y = 0; y < 480; y++)
{
for(int x = 0; x < 320 * 4; x++)
{
buffer2[(479 - y) * 320 * 4 + x] = buffer[y * 4 * 320 + x];
}
}
// make data provider with data.
CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, buffer2, myDataLength, NULL);
// prep the ingredients
int bitsPerComponent = 8;
int bitsPerPixel = 32;
int bytesPerRow = 4 * 320;
CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault;
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
// make the cgimage
CGImageRef image= CGImageCreate(320, 480, bitsPerComponent, bitsPerPixel, bytesPerRow, colorSpaceRef, bitmapInfo, provider, NULL, false, renderingIntent);
//PRINT image... Its black!!!!!!
CGDataProviderRelease(provider);
free(buffer);
free(buffer2);
Before you do a glReadPixels call you must
set proper packing (see glPixelStorei reference page)
select the right buffer to read from with glReadBuffer (front after swapping, back before swapping, I recommend swap and read from front)

Get pixels and colours from NSImage

I have created an NSImage object, and ideally would like to determine how many of each pixels colour it contains. Is this possible?
This code renders the NSImage into a CGBitmapContext:
- (void)updateImageData {
if (!_image)
return;
// Dimensions - source image determines context size
NSSize imageSize = _image.size;
NSRect imageRect = NSMakeRect(0, 0, imageSize.width, imageSize.height);
// Create a context to hold the image data
CGColorSpaceRef colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
CGContextRef ctx = CGBitmapContextCreate(NULL,
imageSize.width,
imageSize.height,
8,
0,
colorSpace,
kCGImageAlphaPremultipliedLast);
// Wrap graphics context
NSGraphicsContext* gctx = [NSGraphicsContext graphicsContextWithCGContext:ctx flipped:NO];
// Make our bitmap context current and render the NSImage into it
[NSGraphicsContext setCurrentContext:gctx];
[_image drawInRect:imageRect];
// Calculate the histogram
[self computeHistogramFromBitmap:ctx];
// Clean up
[NSGraphicsContext setCurrentContext:nil];
CGContextRelease(ctx);
CGColorSpaceRelease(colorSpace);
}
Given a bitmap context, we can access the raw image data directly, and compute the histograms for each colour channel:
- (void)computeHistogramFromBitmap:(CGContextRef)bitmap {
// NB: Assumes RGBA 8bpp
size_t width = CGBitmapContextGetWidth(bitmap);
size_t height = CGBitmapContextGetHeight(bitmap);
uint32_t* pixel = (uint32_t*)CGBitmapContextGetData(bitmap);
for (unsigned y = 0; y < height; y++)
{
for (unsigned x = 0; x < width; x++)
{
uint32_t rgba = *pixel;
// Extract colour components
uint8_t red = (rgba & 0x000000ff) >> 0;
uint8_t green = (rgba & 0x0000ff00) >> 8;
uint8_t blue = (rgba & 0x00ff0000) >> 16;
// Accumulate each colour
_histogram[kRedChannel][red]++;
_histogram[kGreenChannel][green]++;
_histogram[kBlueChannel][blue]++;
// Next pixel!
pixel++;
}
}
}
#end
I've published a complete project, a Cocoa sample app, which includes the above.
https://github.com/gavinb/CocoaImageHistogram.git
I suggest creating your own bitmap context, wrapping it in a graphics context and setting that as the current context, telling the image to draw itself, and then accessing the pixel data behind the bitmap context directly.
This will be more code, but will save you both a trip through a TIFF representation and the creation of thousands or millions of NSColor objects. If you're working with images of any appreciable size, these expenses will add up quickly.
Get an NSBitmapImageRep from your NSImage. Then you can get access to the pixels.
NSImage* img = ...;
NSBitmapImageRep* raw_img = [NSBitmapImageRep imageRepWithData:[img TIFFRepresentation]];
NSColor* color = [raw_img colorAtX:0 y:0];
Look for "histogram" in the Core Image documentation.
Using colorAtX with NSBitmapImageRep does not always lead to the exact correct color.
I managed to get the correct color with this simple code:
[yourImage lockFocus]; // yourImage is just your NSImage variable
NSColor *pixelColor = NSReadPixel(NSMakePoint(1, 1)); // Or another point
[yourImage unlockFocus];
This maybe a more streamlined approach for some and reduce complexity of dropping into memory management.
https://github.com/koher/EasyImagy
Code sample
https://github.com/koher/EasyImagyCameraSample
import EasyImagy
let image = Image<RGBA<UInt8>>(nsImage: "test.png") // N.B. init with nsImage
print(image[x, y])
image[x, y] = RGBA(red: 255, green: 0, blue: 0, alpha: 127)
image[x, y] = RGBA(0xFF00007F) // red: 255, green: 0, blue: 0, alpha: 127
// Iterates over all pixels
for pixel in image {
// ...
}
//// Gets a pixel by subscripts Gets a pixel by
let pixel = image[x, y]
// Sets a pixel by subscripts
image[x, y] = RGBA(0xFF0000FF)
image[x, y].alpha = 127
// Safe get for a pixel
if let pixel = image.pixelAt(x: x, y: y) {
print(pixel.red)
print(pixel.green)
print(pixel.blue)
print(pixel.alpha)
print(pixel.gray) // (red + green + blue) / 3
print(pixel) // formatted like "#FF0000FF"
} else {
// `pixel` is safe: `nil` is returned when out of bounds
print("Out of bounds")
}

How to get pixel data from a UIImage (Cocoa Touch) or CGImage (Core Graphics)?

I have a UIImage (Cocoa Touch). From that, I'm happy to get a CGImage or anything else you'd like that's available. I'd like to write this function:
- (int)getRGBAFromImage:(UIImage *)image atX:(int)xx andY:(int)yy {
// [...]
// What do I want to read about to help
// me fill in this bit, here?
// [...]
int result = (red << 24) | (green << 16) | (blue << 8) | alpha;
return result;
}
FYI, I combined Keremk's answer with my original outline, cleaned-up the typos, generalized it to return an array of colors and got the whole thing to compile. Here is the result:
+ (NSArray*)getRGBAsFromImage:(UIImage*)image atX:(int)x andY:(int)y count:(int)count
{
NSMutableArray *result = [NSMutableArray arrayWithCapacity:count];
// First get the image into your data buffer
CGImageRef imageRef = [image CGImage];
NSUInteger width = CGImageGetWidth(imageRef);
NSUInteger height = CGImageGetHeight(imageRef);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
unsigned char *rawData = (unsigned char*) calloc(height * width * 4, sizeof(unsigned char));
NSUInteger bytesPerPixel = 4;
NSUInteger bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(rawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
CGContextRelease(context);
// Now your rawData contains the image data in the RGBA8888 pixel format.
NSUInteger byteIndex = (bytesPerRow * y) + x * bytesPerPixel;
for (int i = 0 ; i < count ; ++i)
{
CGFloat alpha = ((CGFloat) rawData[byteIndex + 3] ) / 255.0f;
CGFloat red = ((CGFloat) rawData[byteIndex] ) / alpha;
CGFloat green = ((CGFloat) rawData[byteIndex + 1] ) / alpha;
CGFloat blue = ((CGFloat) rawData[byteIndex + 2] ) / alpha;
byteIndex += bytesPerPixel;
UIColor *acolor = [UIColor colorWithRed:red green:green blue:blue alpha:alpha];
[result addObject:acolor];
}
free(rawData);
return result;
}
One way of doing it is to draw the image to a bitmap context that is backed by a given buffer for a given colorspace (in this case it is RGB): (note that this will copy the image data to that buffer, so you do want to cache it instead of doing this operation every time you need to get pixel values)
See below as a sample:
// First get the image into your data buffer
CGImageRef image = [myUIImage CGImage];
NSUInteger width = CGImageGetWidth(image);
NSUInteger height = CGImageGetHeight(image);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
unsigned char *rawData = malloc(height * width * 4);
NSUInteger bytesPerPixel = 4;
NSUInteger bytesPerRow = bytesPerPixel * width;
NSUInteger bitsPerComponent = 8;
CGContextRef context = CGBitmapContextCreate(rawData, width, height, bitsPerComponent, bytesPerRow, colorSpace, kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace);
CGContextDrawImage(context, CGRectMake(0, 0, width, height));
CGContextRelease(context);
// Now your rawData contains the image data in the RGBA8888 pixel format.
int byteIndex = (bytesPerRow * yy) + xx * bytesPerPixel;
red = rawData[byteIndex];
green = rawData[byteIndex + 1];
blue = rawData[byteIndex + 2];
alpha = rawData[byteIndex + 3];
Apple's Technical Q&A QA1509 shows the following simple approach:
CFDataRef CopyImagePixels(CGImageRef inImage)
{
return CGDataProviderCopyData(CGImageGetDataProvider(inImage));
}
Use CFDataGetBytePtr to get to the actual bytes (and various CGImageGet* methods to understand how to interpret them).
Couldn't believe that there is not one single correct answer here. No need to allocate pointers, and the unmultiplied values still need to be normalized. To cut to the chase, here is the correct version for Swift 4. For UIImage just use .cgImage.
extension CGImage {
func colors(at: [CGPoint]) -> [UIColor]? {
let colorSpace = CGColorSpaceCreateDeviceRGB()
let bytesPerPixel = 4
let bytesPerRow = bytesPerPixel * width
let bitsPerComponent = 8
let bitmapInfo: UInt32 = CGImageAlphaInfo.premultipliedLast.rawValue | CGBitmapInfo.byteOrder32Big.rawValue
guard let context = CGContext(data: nil, width: width, height: height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo),
let ptr = context.data?.assumingMemoryBound(to: UInt8.self) else {
return nil
}
context.draw(self, in: CGRect(x: 0, y: 0, width: width, height: height))
return at.map { p in
let i = bytesPerRow * Int(p.y) + bytesPerPixel * Int(p.x)
let a = CGFloat(ptr[i + 3]) / 255.0
let r = (CGFloat(ptr[i]) / a) / 255.0
let g = (CGFloat(ptr[i + 1]) / a) / 255.0
let b = (CGFloat(ptr[i + 2]) / a) / 255.0
return UIColor(red: r, green: g, blue: b, alpha: a)
}
}
}
The reason you have to draw/convert the image first into a buffer is because images can have several different formats. This step is required to convert it to a consistent format you can read.
NSString * path = [[NSBundle mainBundle] pathForResource:#"filename" ofType:#"jpg"];
UIImage * img = [[UIImage alloc]initWithContentsOfFile:path];
CGImageRef image = [img CGImage];
CFDataRef data = CGDataProviderCopyData(CGImageGetDataProvider(image));
const unsigned char * buffer = CFDataGetBytePtr(data);
Here is a SO thread where #Matt renders only the desired pixel into a 1x1 context by displacing the image so that the desired pixel aligns with the one pixel in the context.
Swift 5 version
The answers given here are either outdated or incorrect because they don't take into account the following:
The pixel size of the image can differ from its point size that is returned by image.size.width/image.size.height.
There can be various layouts used by pixel components in the image, such as BGRA, ABGR, ARGB etc. or may not have an alpha component at all, such as BGR and RGB. For example, UIView.drawHierarchy(in:afterScreenUpdates:) method can produce BGRA images.
Color components can be premultiplied by the alpha for all pixels in the image and need to be divided by alpha in order to restore the original color.
For memory optimization used by CGImage, the size of a pixel row in bytes can be greater than the mere multiplication of the pixel width by 4.
The code below is to provide a universal Swift 5 solution to get the UIColor of a pixel for all such special cases. The code is optimized for usability and clarity, not for performance.
public extension UIImage {
var pixelWidth: Int {
return cgImage?.width ?? 0
}
var pixelHeight: Int {
return cgImage?.height ?? 0
}
func pixelColor(x: Int, y: Int) -> UIColor {
assert(
0..<pixelWidth ~= x && 0..<pixelHeight ~= y,
"Pixel coordinates are out of bounds")
guard
let cgImage = cgImage,
let data = cgImage.dataProvider?.data,
let dataPtr = CFDataGetBytePtr(data),
let colorSpaceModel = cgImage.colorSpace?.model,
let componentLayout = cgImage.bitmapInfo.componentLayout
else {
assertionFailure("Could not get a pixel of an image")
return .clear
}
assert(
colorSpaceModel == .rgb,
"The only supported color space model is RGB")
assert(
cgImage.bitsPerPixel == 32 || cgImage.bitsPerPixel == 24,
"A pixel is expected to be either 4 or 3 bytes in size")
let bytesPerRow = cgImage.bytesPerRow
let bytesPerPixel = cgImage.bitsPerPixel/8
let pixelOffset = y*bytesPerRow + x*bytesPerPixel
if componentLayout.count == 4 {
let components = (
dataPtr[pixelOffset + 0],
dataPtr[pixelOffset + 1],
dataPtr[pixelOffset + 2],
dataPtr[pixelOffset + 3]
)
var alpha: UInt8 = 0
var red: UInt8 = 0
var green: UInt8 = 0
var blue: UInt8 = 0
switch componentLayout {
case .bgra:
alpha = components.3
red = components.2
green = components.1
blue = components.0
case .abgr:
alpha = components.0
red = components.3
green = components.2
blue = components.1
case .argb:
alpha = components.0
red = components.1
green = components.2
blue = components.3
case .rgba:
alpha = components.3
red = components.0
green = components.1
blue = components.2
default:
return .clear
}
// If chroma components are premultiplied by alpha and the alpha is `0`,
// keep the chroma components to their current values.
if cgImage.bitmapInfo.chromaIsPremultipliedByAlpha && alpha != 0 {
let invUnitAlpha = 255/CGFloat(alpha)
red = UInt8((CGFloat(red)*invUnitAlpha).rounded())
green = UInt8((CGFloat(green)*invUnitAlpha).rounded())
blue = UInt8((CGFloat(blue)*invUnitAlpha).rounded())
}
return .init(red: red, green: green, blue: blue, alpha: alpha)
} else if componentLayout.count == 3 {
let components = (
dataPtr[pixelOffset + 0],
dataPtr[pixelOffset + 1],
dataPtr[pixelOffset + 2]
)
var red: UInt8 = 0
var green: UInt8 = 0
var blue: UInt8 = 0
switch componentLayout {
case .bgr:
red = components.2
green = components.1
blue = components.0
case .rgb:
red = components.0
green = components.1
blue = components.2
default:
return .clear
}
return .init(red: red, green: green, blue: blue, alpha: UInt8(255))
} else {
assertionFailure("Unsupported number of pixel components")
return .clear
}
}
}
public extension UIColor {
convenience init(red: UInt8, green: UInt8, blue: UInt8, alpha: UInt8) {
self.init(
red: CGFloat(red)/255,
green: CGFloat(green)/255,
blue: CGFloat(blue)/255,
alpha: CGFloat(alpha)/255)
}
}
public extension CGBitmapInfo {
enum ComponentLayout {
case bgra
case abgr
case argb
case rgba
case bgr
case rgb
var count: Int {
switch self {
case .bgr, .rgb: return 3
default: return 4
}
}
}
var componentLayout: ComponentLayout? {
guard let alphaInfo = CGImageAlphaInfo(rawValue: rawValue & Self.alphaInfoMask.rawValue) else { return nil }
let isLittleEndian = contains(.byteOrder32Little)
if alphaInfo == .none {
return isLittleEndian ? .bgr : .rgb
}
let alphaIsFirst = alphaInfo == .premultipliedFirst || alphaInfo == .first || alphaInfo == .noneSkipFirst
if isLittleEndian {
return alphaIsFirst ? .bgra : .abgr
} else {
return alphaIsFirst ? .argb : .rgba
}
}
var chromaIsPremultipliedByAlpha: Bool {
let alphaInfo = CGImageAlphaInfo(rawValue: rawValue & Self.alphaInfoMask.rawValue)
return alphaInfo == .premultipliedFirst || alphaInfo == .premultipliedLast
}
}
UIImage is a wrapper the bytes are CGImage or CIImage
According the the Apple Reference on UIImage the object is immutable and you have no access to the backing bytes. While it is true that you can access the CGImage data if you populated the UIImage with a CGImage (explicitly or implicitly), it will return NULL if the UIImage is backed by a CIImage and vice-versa.
Image objects not provide direct access to their underlying image
data. However, you can retrieve the image data in other formats for
use in your app. Specifically, you can use the cgImage and ciImage
properties to retrieve versions of the image that are compatible with
Core Graphics and Core Image, respectively. You can also use the
UIImagePNGRepresentation(:) and UIImageJPEGRepresentation(:_:)
functions to generate an NSData object containing the image data in
either the PNG or JPEG format.
Common tricks to getting around this issue
As stated your options are
UIImagePNGRepresentation or JPEG
Determine if image has CGImage or CIImage backing data and get it there
Neither of these are particularly good tricks if you want output that isn't ARGB, PNG, or JPEG data and the data isn't already backed by CIImage.
My recommendation, try CIImage
While developing your project it might make more sense for you to avoid UIImage altogether and pick something else. UIImage, as a Obj-C image wrapper, is often backed by CGImage to the point where we take it for granted. CIImage tends to be a better wrapper format in that you can use a CIContext to get out the format you desire without needing to know how it was created. In your case, getting the bitmap would be a matter of calling
- render:toBitmap:rowBytes:bounds:format:colorSpace:
As an added bonus you can start doing nice manipulations to the image by chaining filters onto the image. This solves a lot of the issues where the image is upside down or needs to be rotated/scaled etc.
Building on Olie and Algal's answer, here is an updated answer for Swift 3
public func getRGBAs(fromImage image: UIImage, x: Int, y: Int, count: Int) -> [UIColor] {
var result = [UIColor]()
// First get the image into your data buffer
guard let cgImage = image.cgImage else {
print("CGContext creation failed")
return []
}
let width = cgImage.width
let height = cgImage.height
let colorSpace = CGColorSpaceCreateDeviceRGB()
let rawdata = calloc(height*width*4, MemoryLayout<CUnsignedChar>.size)
let bytesPerPixel = 4
let bytesPerRow = bytesPerPixel * width
let bitsPerComponent = 8
let bitmapInfo: UInt32 = CGImageAlphaInfo.premultipliedLast.rawValue | CGBitmapInfo.byteOrder32Big.rawValue
guard let context = CGContext(data: rawdata, width: width, height: height, bitsPerComponent: bitsPerComponent, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: bitmapInfo) else {
print("CGContext creation failed")
return result
}
context.draw(cgImage, in: CGRect(x: 0, y: 0, width: width, height: height))
// Now your rawData contains the image data in the RGBA8888 pixel format.
var byteIndex = bytesPerRow * y + bytesPerPixel * x
for _ in 0..<count {
let alpha = CGFloat(rawdata!.load(fromByteOffset: byteIndex + 3, as: UInt8.self)) / 255.0
let red = CGFloat(rawdata!.load(fromByteOffset: byteIndex, as: UInt8.self)) / alpha
let green = CGFloat(rawdata!.load(fromByteOffset: byteIndex + 1, as: UInt8.self)) / alpha
let blue = CGFloat(rawdata!.load(fromByteOffset: byteIndex + 2, as: UInt8.self)) / alpha
byteIndex += bytesPerPixel
let aColor = UIColor(red: red, green: green, blue: blue, alpha: alpha)
result.append(aColor)
}
free(rawdata)
return result
}
swift
To access the raw RGB values of an UIImage in Swift 5 use the underlying CGImage and its dataProvider:
import UIKit
let image = UIImage(named: "example.png")!
guard let cgImage = image.cgImage,
let data = cgImage.dataProvider?.data,
let bytes = CFDataGetBytePtr(data) else {
fatalError("Couldn't access image data")
}
assert(cgImage.colorSpace?.model == .rgb)
let bytesPerPixel = cgImage.bitsPerPixel / cgImage.bitsPerComponent
for y in 0 ..< cgImage.height {
for x in 0 ..< cgImage.width {
let offset = (y * cgImage.bytesPerRow) + (x * bytesPerPixel)
let components = (r: bytes[offset], g: bytes[offset + 1], b: bytes[offset + 2])
print("[x:\(x), y:\(y)] \(components)")
}
print("---")
}
https://www.ralfebert.de/ios/examples/image-processing/uiimage-raw-pixels/
Based on different answers but mainly on this, this works for what I need:
UIImage *image1 = ...; // The image from where you want a pixel data
int pixelX = ...; // The X coordinate of the pixel you want to retrieve
int pixelY = ...; // The Y coordinate of the pixel you want to retrieve
uint32_t pixel1; // Where the pixel data is to be stored
CGContextRef context1 = CGBitmapContextCreate(&pixel1, 1, 1, 8, 4, CGColorSpaceCreateDeviceRGB(), kCGImageAlphaNoneSkipFirst);
CGContextDrawImage(context1, CGRectMake(-pixelX, -pixelY, CGImageGetWidth(image1.CGImage), CGImageGetHeight(image1.CGImage)), image1.CGImage);
CGContextRelease(context1);
As a result of this lines, you will have a pixel in AARRGGBB format with alpha always set to FF in the 4 byte unsigned integer pixel1.

Resources