Getting into pixel data of NSImage - cocoa

I'm writing application that operates on black&white images. I'm doing it by passing a NSImage object into my method and then making NSBitmapImageRep from NSImage. All works but quite slow. Here's my code:
- (NSImage *)skeletonization: (NSImage *)image
{
int x = 0, y = 0;
NSUInteger pixelVariable = 0;
NSBitmapImageRep *bitmapImageRep = [[NSBitmapImageRep alloc] initWithData:[image TIFFRepresentation]];
[myHelpText setIntValue:[bitmapImageRep pixelsWide]];
[myHelpText2 setIntValue:[bitmapImageRep pixelsHigh]];
NSColor *black = [NSColor blackColor];
NSColor *white = [NSColor whiteColor];
[myColor set];
[myColor2 set];
for (x=0; x<=[bitmapImageRep pixelsWide]; x++) {
for (y=0; y<=[bitmapImageRep pixelsHigh]; y++) {
// This is only to see if it's working
[bitmapImageRep setColor:myColor atX:x y:y];
}
}
[myColor release];
[myColor2 release];
NSImage *producedImage = [[NSImage alloc] init];
[producedImage addRepresentation:bitmapImageRep];
[bitmapImageRep release];
return [producedImage autorelease];
}
So I tried to use CIImage but I don't know how to get into each pixel by (x,y) coordinates. That is really important.

Use the representations array property from NSImage, to get your NSBitmapImageRep. It should be faster than serializing your image to a TIFF and then back.
Use the bitmapData property of the NSBitmapImageRep to access the image bytes directly.
eg
unsigned char black = 0;
unsigned char white = 255;
NSBitmapImageRep* bitmapImageRep = [[image representations] firstObject];
// you will need to do checks here to determine the pixelformat of your bitmap data
unsigned char* imageData = [bitmapImageRep bitmapData];
int rowBytes = [bitmapImageRep bytesPerRow];
int bpp = [bitmapImageRep bitsPerPixel] / 8;
for (x=0; x<[bitmapImageRep pixelsWide]; x++) { // don't use <=
for (y=0; y<[bitmapImageRep pixelsHigh]; y++) {
*(imageData + y * rowBytes + x * bpp ) = black; // Red
*(imageData + y * rowBytes + x * bpp +1) = black; // Green
*(imageData + y * rowBytes + x * bpp +2) = black; // Blue
*(imageData + y * rowBytes + x * bpp +3) = 255; // Alpha
}
}
You will need to know what pixel format you are using in your images before you can go playing with its data, look at the bitsPerPixel property of NSBitmapImageRep to help determine if your image is in RGBA format.
You could be working with a gray scale image, or an RGB image, or possibly CMYK. And either convert the image to what you want first. Or handle the data in the loop differently.

Related

Xcode UI on background thread to render image

I'm rendering an image with text for one of my apps and has a noticeable impact on UI performance (can be as big as ~1 second freeze), so I am doing it on a background thread. Since the image has text, using UILabels and other UIViews makes it easy to lay everything out, and I render the view containing everything to an image.
However, I get a warning from Xcode saying that it's not allowed on the background thread because it uses UIKit. Why am I not allowed to call UIKit on the background thread even though my use case is completely self-contained and isolated from any rendering onscreen?
To help the code below make more sense, it draws an image that is a listing of several items, each of which consists of two small square images and the name of the item all in a row. The list can have several columns. The code has been tweaked slightly (mostly variable names) to avoid showing proprietary code, but does the same job.
My code:
NSArray<MyItem*>* items; // These are the items that I'm drawing. They
// get set before the following code is called.
// Processing code:
const CGFloat TITLE_FONT_SIZE = 50; // font size of the title
const CGFloat ITEM_FONT_SIZE = 25; // font size of the item names
const int OUTER_PADDING = 60; // padding from the edge of the image to the main content
const int ROW_PADDING = 13; // padding between rows
const int COL_PADDING = 100; // padding between columns
const int PADDING = 20; // padding between content items in a row
const int BOX_SIZE = 25; // how high/wide each image is
const int ROW_HEIGHT = BOX_SIZE; // pixel height of a line
const int COL_WIDTH = 500; // pixel width of a column (image1, image2, and name)
// compute the dimensions of the image
UILabel* titleLabel = [[UILabel alloc] init];
titleLabel.font = [UIFont systemFontOfSize:TITLE_FONT_SIZE];
titleLabel.text = #"My image";
[titleLabel sizeToFit];
titleLabel.frame = CGRectMake(OUTER_PADDING, OUTER_PADDING / 2, titleLabel.frame.size.width, titleLabel.frame.size.height);
const int MIN_NUM_COLS = 1 + ((titleLabel.frame.size.width - COL_WIDTH) / (COL_WIDTH + COL_PADDING));
const int NORMAL_NUM_COLS = (int)ceil(sqrt([items count] / (COL_WIDTH / (ROW_HEIGHT))));
const int NUM_COLS = (MIN_NUM_COLS > NORMAL_NUM_COLS ? MIN_NUM_COLS : NORMAL_NUM_COLS);
const int NUM_ROWS = (int)ceil([items count] / (float)NUM_COLS);
const int NUM_OVERFLOW_ROWS = [items count] % NUM_ROWS;
const int titleWidth = titleLabel.frame.size.width;
const int defaultWidth = (NUM_COLS * (COL_WIDTH + COL_PADDING)) - COL_PADDING;
const int pixelWidth = (2 * OUTER_PADDING) + (titleWidth > defaultWidth ? titleWidth : defaultWidth);
const int pixelHeight = (2 * OUTER_PADDING) + (TITLE_FONT_SIZE + PADDING) + (NUM_ROWS * (ROW_HEIGHT + ROW_PADDING)) - ROW_PADDING;
const int nbytes = 4 * pixelHeight * pixelWidth;
byte* data = (byte*)malloc(sizeof(byte) * nbytes);
memset(data, 255, nbytes);
CGContextRef context = CGBitmapContextCreate(data, pixelWidth, pixelHeight, 8, 4 * pixelWidth, CGColorSpaceCreateDeviceRGB(), kCGBitmapByteOrderDefault | kCGImageAlphaNoneSkipLast);
// --------------------------------------------------
// create a view heirarchy and then draw to our context
UIView* mainView = [[UIView alloc] init];
[mainView addSubview:titleLabel];
// setup all the views
int keyIndex = 0;
CGFloat x = OUTER_PADDING;
CGFloat starty = titleLabel.frame.origin.y + titleLabel.frame.size.height + PADDING;
for (int col = 0; col < NUM_COLS; col++)
{
int nrows = (col == NUM_COLS + 1 ? NUM_OVERFLOW_ROWS : NUM_ROWS);
CGFloat y = starty;
for (int row = 0; (row < nrows) && (keyIndex < [items count]); row++)
{
CGFloat tempx = x;
MyItem* item = [items objectAtIndex:keyIndex];
UIImageView* imageview1 = [[UIImageView alloc] initWithImage:item.image1];
imageview1.frame = CGRectMake(tempx, y, BOX_SIZE, BOX_SIZE);
[mainView addSubview:imageview1];
tempx += BOX_SIZE + PADDING;
UIImageView* imageview2 = [[UIImageView alloc] initWithImage:item.imageview2];
imageview2.frame = CGRectMake(tempx, y, BOX_SIZE, BOX_SIZE);
[mainView addSubview:imageview2];
tempx += BOX_SIZE + PADDING;
UILabel* label = [[UILabel alloc] init];
label.font = [UIFont systemFontOfSize:ITEM_FONT_SIZE];
label.text = item.name;
[label sizeToFit];
label.center = CGPointMake(tempx + (label.frame.size.width / 2), imageview2.center.y);
[mainView addSubview:label];
y += ROW_HEIGHT + ROW_PADDING;
keyIndex++;
}
x += COL_WIDTH + COL_PADDING;
}
// --------------------------------------------------
// draw everything to actually generate the image
CGContextConcatCTM(context, CGAffineTransformMake(1, 0, 0, -1, 0, pixelHeight));
[mainView.layer renderInContext:context];
CGImageRef cgimage = CGBitmapContextCreateImage(context);
myCoolImage = [UIImage imageWithCGImage:cgimage];
CGImageRelease(cgimage);
CGContextRelease(context);
free(data);
As we've established in comments, what you're doing is both illegitimate and slow.
Arranging and sizing UILabel and UIImageView objects is slow, and calling
CALayer renderInContext is really slow.
And it isn't how you draw.
Everything you're doing has its analogue in the actual drawing world (Quartz 2D), and if you did it that way, not only would it be legal in the background, it probably wouldn't even need to be in the background because it would be so much faster. So:
Every place you use a UILabel, you can achieve exactly the same effect by using NSAttributedString draw... commands.
Every place you use a UIImageView, you can achieve exactly the same effect by using UIImage draw... commands.
Any of us who does any extensive drawing has learned to create structured layouts of the type you're making by using actual drawing code, and now is your chance to learn to do it too.

How is transparency achieved in cocoa applications

I am trying to understand how is transparency actually implemented in cocoa applications. I was expecting the standard blending equation to be used i.e.
BlendedColour = alpha * layerColour + (1-alpha)*backgroundColour
However, I noticed that there is the slight difference in the blended colour expected if the above equation is used. To verify it, I did a small experiment as follows:
1.) Created a window, added a transparency of 0.8 to the window and grabbed a screenshot.
2.) I took a screenshot of the part of the screen where I am overlaying the window in step one without the window and overlayed the same image as in step 1, using the equation mentioned above. (I used openCV for that).
There is a slight difference in the colours for the two images, if you look closely. I wanted to understand what is causing the difference.
Resources:
1.) Images from Step 1 and Step2 respectively
2.) Code used in step 1
NSRect windowRect = {0,0,200,200};
m_NSWindow = [[NSWindow alloc] initWithContentRect:windowRect styleMask:NSBorderlessWindowMask backing:NSBackingStoreBuffered defer:NO];
[m_NSWindow setTitle:#"overlayWindow"];
[m_NSWindow makeKeyAndOrderFront:nil];
g_imageView = [[NSImageView alloc] initWithFrame:NSMakeRect(0,0,200,200)];
[m_NSWindow.contentView addSubview:g_imageView];
[m_NSWindow setOpaque:NO];
[m_NSWindow setAlphaValue:0.8];
NSBitmapImageRep* imageRep = [[NSBitmapImageRep alloc] initWithBitmapDataPlanes:nil
pixelsWide:200
pixelsHigh:200
bitsPerSample:8
samplesPerPixel:4
hasAlpha:YES
isPlanar:NO
colorSpaceName:NSDeviceRGBColorSpace
bitmapFormat:NSAlphaNonpremultipliedBitmapFormat
bytesPerRow:(200*4)
bitsPerPixel:32];
memcpy(imageRep.bitmapData,m_paintBuffer.data,160000);
NSSize imageSize = NSMakeSize(200,200);
NSImage* myImage = [[NSImage alloc] initWithSize: imageSize];
[myImage addRepresentation:imageRep];
[g_imageView setImage:myImage];
4.) Code for step 2
void overlayImage(const cv::Mat &background, const cv::Mat &foreground,
cv::Mat &output, cv::Point2i location)
{
background.copyTo(output);
// start at the row indicated by location, or at row 0 if location.y is negative.
for (int y = max(location.y, 0); y < background.rows; ++y)
{
int fY = y - location.y; // because of the translation
// we are done of we have processed all rows of the foreground image.
if (fY >= foreground.rows)
break;
// start at the column indicated by location,
// or at column 0 if location.x is negative.
for (int x = max(location.x, 0); x < background.cols; ++x)
{
int fX = x - location.x; // because of the translation.
// we are done with this row if the column is outside of the foreground image.
if (fX >= foreground.cols)
break;
// determine the opacity of the foregrond pixel, using its fourth (alpha) channel.
double opacity =
((double)foreground.data[fY * foreground.step + fX * foreground.channels() + 3])
/ 255.;
// and now combine the background and foreground pixel, using the opacity,
// but only if opacity > 0.
for (int c = 0; opacity > 0 && c < output.channels(); ++c)
{
unsigned char foregroundPx =
foreground.data[fY * foreground.step + fX * foreground.channels() + c];
unsigned char backgroundPx =
background.data[y * background.step + x * background.channels() + c];
output.data[y*output.step + output.channels()*x + c] =
backgroundPx * (1. - opacity) + foregroundPx * opacity;
}
}
}
}

How to iterate through all pixels of an UIImage?

Hey Guys i am currently trying to iterate through all pixels of an UIImage but the way i implemented it it takes sooo much time. So i thought it is the wrong way i implemented it.
This is my method how i get the RGBA Values of an Pixel :
+(NSArray*)getRGBAsFromImage:(UIImage*)image atX:(int)xx andY:(int)yy count:(int)count
{
// Initializing the result array
NSMutableArray *result = [NSMutableArray arrayWithCapacity:count];
// First get the image into your data buffer
CGImageRef imageRef = [image CGImage]; // creating an Instance of
NSUInteger width = CGImageGetWidth(imageRef); // Get width of our Image
NSUInteger height = CGImageGetHeight(imageRef); // Get height of our Image
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB(); // creating our colour Space
// Getting that raw Data out of an image
unsigned char *rawData = (unsigned char*) calloc(height * width * 4, sizeof(unsigned char));
NSUInteger bytesPerPixel = 4; // Bytes per pixel defined
NSUInteger bytesPerRow = bytesPerPixel * width; // Bytes per row
NSUInteger bitsPerComponent = 8; // Bytes per component
CGContextRef context = CGBitmapContextCreate(rawData, width, height,
bitsPerComponent, bytesPerRow, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGColorSpaceRelease(colorSpace); // releasing the color space
CGContextDrawImage(context, CGRectMake(0, 0, width, height), imageRef);
CGContextRelease(context);
// Now your rawData contains the image data in the RGBA8888 pixel format.
int byteIndex = (bytesPerRow * yy) + xx * bytesPerPixel;
for (int ii = 0 ; ii < count ; ++ii)
{
CGFloat red = (rawData[byteIndex] * 1.0) / 255.0;
CGFloat green = (rawData[byteIndex + 1] * 1.0) / 255.0;
CGFloat blue = (rawData[byteIndex + 2] * 1.0) / 255.0;
CGFloat alpha = (rawData[byteIndex + 3] * 1.0) / 255.0;
byteIndex += 4;
UIColor *acolor = [UIColor colorWithRed:red green:green blue:blue alpha:alpha];
[result addObject:acolor];
}
free(rawData);
return result;
}
And this is the code how i parse through all the pixels :
for (NSUInteger y = 0 ; y < self.originalPictureWidth; y++) {
for (NSUInteger x = 0 ; x < self.originalPictureHeight; x++) {
NSArray * originalRGBA = [ComputerVisionHelperClass getRGBAsFromImage:self.originalPicture atX:(int)x andY:(int)y count:1];
NSArray * referenceRGBA = [ComputerVisionHelperClass getRGBAsFromImage:self.referencePicture atX:(int)referenceIndexX andY:(int)referenceIndexY count:1];
// Do something else ....
}
}
Is there a faster way of getting all RGBA values of an uiimage instance ?
For every pixel, you're generating a new copy of the image and then throwing it away. Yes, it would be much faster by just getting the data once and then processing on that byte array.
But it heavily depends on what is in "Do something else." There are many CoreImage and vImage functions that can do image processing very quickly, but you may need to approach the problem differently. It depends on what you're doing.

Export Opengl ES video

XCode has the ability to capture Opengl ES frames from the iPad, and that's great! I would like to extend this functionality and capture an entire Opengl ES movie of my application. Is there a way for that?
if it's not possible using XCode, how can i do it without much effort and big changes on my code? thank you very much!
I use a very simple technique, which requires just a few lines of code.
You can capture each OGL frame into UIImage using this code:
- (UIImage*)captureScreen {
NSInteger dataLength = framebufferWidth * framebufferHeight * 4;
// Allocate array.
GLuint *buffer = (GLuint *) malloc(dataLength);
GLuint *resultsBuffer = (GLuint *)malloc(dataLength);
// Read data
glReadPixels(0, 0, framebufferWidth, framebufferHeight, GL_RGBA, GL_UNSIGNED_BYTE, buffer);
// Flip vertical
for(int y = 0; y < framebufferHeight; y++) {
for(int x = 0; x < framebufferWidth; x++) {
resultsBuffer[x + y * framebufferWidth] = buffer[x + (framebufferHeight - 1 - y) * framebufferWidth];
}
}
free(buffer);
// make data provider with data.
CGDataProviderRef provider = CGDataProviderCreateWithData(NULL, resultsBuffer, dataLength, releaseScreenshotData);
// prep the ingredients
const int bitsPerComponent = 8;
const int bitsPerPixel = 4 * bitsPerComponent;
const int bytesPerRow = 4 * framebufferWidth;
CGColorSpaceRef colorSpaceRef = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrderDefault;
CGColorRenderingIntent renderingIntent = kCGRenderingIntentDefault;
// make the cgimage
CGImageRef imageRef = CGImageCreate(framebufferWidth, framebufferHeight, bitsPerComponent, bitsPerPixel, bytesPerRow, colorSpaceRef, bitmapInfo, provider, NULL, NO, renderingIntent);
CGColorSpaceRelease(colorSpaceRef);
CGDataProviderRelease(provider);
// then make the UIImage from that
UIImage *image = [UIImage imageWithCGImage:imageRef];
CGImageRelease(imageRef);
return image;
}
Then you will capture each frame in your main loop:
- (void)onTimer {
// Compute and render new frame
[self update];
// Recording
if (recordingMode == RecordingModeMovie) {
recordingFrameNum++;
// Save frame
UIImage *image = [self captureScreen];
NSString *fileName = [NSString stringWithFormat:#"%d.jpg", (int)recordingFrameNum];
[UIImageJPEGRepresentation(image, 1.0) writeToFile:[basePath stringByAppendingPathComponent:fileName] atomically:NO];
}
}
At the end you will have tons of JPEG files which can be easily converted into a movie by
Time Lapse Assembler
If you want to have nice 30FPS movie, hard fix your calc steps to 1 / 30.0 sec per frame.

Get pixels and colours from NSImage

I have created an NSImage object, and ideally would like to determine how many of each pixels colour it contains. Is this possible?
This code renders the NSImage into a CGBitmapContext:
- (void)updateImageData {
if (!_image)
return;
// Dimensions - source image determines context size
NSSize imageSize = _image.size;
NSRect imageRect = NSMakeRect(0, 0, imageSize.width, imageSize.height);
// Create a context to hold the image data
CGColorSpaceRef colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
CGContextRef ctx = CGBitmapContextCreate(NULL,
imageSize.width,
imageSize.height,
8,
0,
colorSpace,
kCGImageAlphaPremultipliedLast);
// Wrap graphics context
NSGraphicsContext* gctx = [NSGraphicsContext graphicsContextWithCGContext:ctx flipped:NO];
// Make our bitmap context current and render the NSImage into it
[NSGraphicsContext setCurrentContext:gctx];
[_image drawInRect:imageRect];
// Calculate the histogram
[self computeHistogramFromBitmap:ctx];
// Clean up
[NSGraphicsContext setCurrentContext:nil];
CGContextRelease(ctx);
CGColorSpaceRelease(colorSpace);
}
Given a bitmap context, we can access the raw image data directly, and compute the histograms for each colour channel:
- (void)computeHistogramFromBitmap:(CGContextRef)bitmap {
// NB: Assumes RGBA 8bpp
size_t width = CGBitmapContextGetWidth(bitmap);
size_t height = CGBitmapContextGetHeight(bitmap);
uint32_t* pixel = (uint32_t*)CGBitmapContextGetData(bitmap);
for (unsigned y = 0; y < height; y++)
{
for (unsigned x = 0; x < width; x++)
{
uint32_t rgba = *pixel;
// Extract colour components
uint8_t red = (rgba & 0x000000ff) >> 0;
uint8_t green = (rgba & 0x0000ff00) >> 8;
uint8_t blue = (rgba & 0x00ff0000) >> 16;
// Accumulate each colour
_histogram[kRedChannel][red]++;
_histogram[kGreenChannel][green]++;
_histogram[kBlueChannel][blue]++;
// Next pixel!
pixel++;
}
}
}
#end
I've published a complete project, a Cocoa sample app, which includes the above.
https://github.com/gavinb/CocoaImageHistogram.git
I suggest creating your own bitmap context, wrapping it in a graphics context and setting that as the current context, telling the image to draw itself, and then accessing the pixel data behind the bitmap context directly.
This will be more code, but will save you both a trip through a TIFF representation and the creation of thousands or millions of NSColor objects. If you're working with images of any appreciable size, these expenses will add up quickly.
Get an NSBitmapImageRep from your NSImage. Then you can get access to the pixels.
NSImage* img = ...;
NSBitmapImageRep* raw_img = [NSBitmapImageRep imageRepWithData:[img TIFFRepresentation]];
NSColor* color = [raw_img colorAtX:0 y:0];
Look for "histogram" in the Core Image documentation.
Using colorAtX with NSBitmapImageRep does not always lead to the exact correct color.
I managed to get the correct color with this simple code:
[yourImage lockFocus]; // yourImage is just your NSImage variable
NSColor *pixelColor = NSReadPixel(NSMakePoint(1, 1)); // Or another point
[yourImage unlockFocus];
This maybe a more streamlined approach for some and reduce complexity of dropping into memory management.
https://github.com/koher/EasyImagy
Code sample
https://github.com/koher/EasyImagyCameraSample
import EasyImagy
let image = Image<RGBA<UInt8>>(nsImage: "test.png") // N.B. init with nsImage
print(image[x, y])
image[x, y] = RGBA(red: 255, green: 0, blue: 0, alpha: 127)
image[x, y] = RGBA(0xFF00007F) // red: 255, green: 0, blue: 0, alpha: 127
// Iterates over all pixels
for pixel in image {
// ...
}
//// Gets a pixel by subscripts Gets a pixel by
let pixel = image[x, y]
// Sets a pixel by subscripts
image[x, y] = RGBA(0xFF0000FF)
image[x, y].alpha = 127
// Safe get for a pixel
if let pixel = image.pixelAt(x: x, y: y) {
print(pixel.red)
print(pixel.green)
print(pixel.blue)
print(pixel.alpha)
print(pixel.gray) // (red + green + blue) / 3
print(pixel) // formatted like "#FF0000FF"
} else {
// `pixel` is safe: `nil` is returned when out of bounds
print("Out of bounds")
}

Resources