I can use XGetPixel() to get a pixel from an XImage. What do I use to get a pixel from a Pixmap?
It would be nice if there was at least one function to pull a pixel from the server's drawable. Sadly, there isn't one. However the following might be of some use.
Since Pixmap is a Drawable you can pass XGetImage() the Pixmap which will return a pointer to an XImage. Now that you have an XImage you can use XGetPixel().
XGetImage Parameters:
XImage *XGetImage(display, d, x, y, width, height, plane_mask, format)
Display *display;
Drawable d;
int x, y;
unsigned int width, height;
unsigned long plane_mask;
int format;
Better yet you could have a pre-created XImage and pass it, along with the Pixmap to XGetSubImage(). You can grab a single pixel by passing a width and height both set to 1, and then use XGetPixel() on your XImage.
XGetSubImage Parameters:
XImage *XGetSubImage(display, d, x, y, width, height, plane_mask, format, dest_image, dest_x,
dest_y)
Display *display;
Drawable d;
int x, y;
unsigned int width, height;
unsigned long plane_mask;
int format;
XImage *dest_image;
int dest_x, dest_y;
Note: XGetSubImage() returns a pointer to the same XImage structure specified by dest_image.
Generally it's a bad idea from performance point of view: never read from screen, only push to it. If you need to get pixel state, maintain local buffer of screen. If you can't modify program you are using, then +1 to Jonny Henly's answer: do a GetImage request first do download a region containing your pixel first, then read locally. If you want to access multiple pixels in a loop it's better to grab them all in one request
Related
I'm using stb_image to upload an image to the GPU. If I just upload the image with stbi_load I can confirm (nvidia Nsight) that the image is correctly stored in the GPU memory. However, some images I like to resize before I upload the to the GPU. In this case, I get a crash. This is the code:
int textureWidth;
int textureHeight;
int textureChannelCount;
stbi_uc* pixels = stbi_load(fullPath.string().c_str(), &textureWidth, &textureHeight, &textureChannelCount, STBI_rgb_alpha);
if (!pixels) {
char error[512];
sprintf_s(error, "Failed to load image %s!", pathToTexture);
throw std::runtime_error(error);
}
stbi_uc* resizedPixels = nullptr;
uint32_t imageSize = 0;
if (scale > 1.0001f || scale < 0.9999f) {
stbir_resize_uint8(pixels, textureWidth, textureHeight, 0, resizedPixels, textureWidth * scale, textureHeight * scale, 0, textureChannelCount);
stbi_image_free(pixels);
textureWidth *= scale;
textureHeight *= scale;
imageSize = textureWidth * textureHeight * textureChannelCount;
} else {
resizedPixels = pixels;
imageSize = textureWidth * textureHeight * textureChannelCount;
}
// Upload the image to the gpu
When this code is run with scale set to 1.0f, it works fine. However, when I set the scale to 0.25f, the program crashes in method stbir_resize_uint8. The image I'm providing in both cases is a 1920x1080 RGBA PNG. Alpha channel is set to 1.0f across the whole image.
Which function do I have to use to resize the image?
EDIT: If I allocate the memory myself, the function no longer crashes and works fine. But I though stb handles all memory allocation internally. Was I wrong?
I see you found and solved the problem in your edit but here's some useful advice anyway:
It seems like the comments in the source (which is also the documentation) don't explicitly mention that you have to allocate memory for the resized image, but it becomes clear when you take a closer look at the function's signature:
STBIRDEF int stbir_resize_uint8( const unsigned char *input_pixels , int input_w , int input_h , int input_stride_in_bytes,
unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
int num_channels);
Think about how you yourself would return the address of a memory chunk that you allocated in a function. The easiest would be to return the pointer directly like so:
unsigned char* allocate_memory( int size )
{ return (unsigned char*) malloc(size); }
However the return seems to be reserved for error codes, so your only option would be to manipulate the pointer as a side-effect. To do that, you'd need to pass a pointer to it (pointer to pointer):
int allocate_memory( unsigned char** pointer_to_array, int size )
{
*pointer_to_array = (unsigned char*) malloc(size);
/* Check if allocation was successful and do other stuff... */
return 0;
}
If you take a closer look at the resize function's signature, you'll notice that there's no such parameter passed, so there's no way for it to return the address of internally allocated memory. (unsigned char* output_pixels instead of unsigned char** output_pixels). As a result, you have to allocate the memory for the resized image yourself.
I hope this helps you in the future.
There is a mention of memory allocation in the docs but as far as I understand, it's about allocations required to perform the resizing, which is unrelated to the output.
I am trying to construct a right-view image from a left-view image and its disparity map. I use the middleburry dataset 2003 (http://vision.middlebury.edu/stereo/data/scenes2003/) with the full size images, which means the value v of each pixel in the disparity map corresponds to a shift of v pixels on the left-view image.
My algorithm is quite simple. For each pixel of coordinates (x, y) in the left-view image, I copy this pixel on the right-view image but at the coordinates (x - d, y) where d is the value of the disparity map at the coordinates (x, y). If the disparity value is 0, I just don't do anything. I use openCV to manipulate the images.
Here is my code:
void computeCorrespondingImage(const cv::Mat &img, const cv::Mat &disparity, cv::Mat &dest,
const bool leftInput, const int disparityScale)
{
const int shiftDirection = leftInput ? -1 : 1;
dest.create(img.rows, img.cols, img.type());
for (int i(0) ; i < img.rows ; ++i) {
for (int j(0) ; j < img.cols ; ++j) {
const uchar d(disparity.at<const uchar>(i, j));
const int computedColumn(j + shiftDirection * (d / disparityScale));
// No need to consider pixels who would be outside of the image's bounds
if (d > 0 && computedColumn >= 0 && computedColumn < img.cols) {
dest.at<cv::Vec3b>(i, computedColumn) = img.at<const cv::Vec3b>(i, j);
}
}
}
}
Since the disparity map is a ground-truth disparity map, I would expect to get an image quite like the right-view image provided in the dataset with some black areas (for which the disparity is unknown).
However, for some reasons it's like the computed right-view image is split at the center, making the image unusable.
Left-view image :
Ground-truth disparity map :
What I get :
Thank you in advance for your help.
Ok, I figured it out. I was loading the disparity image with imread without specifying that it was a gray scale image (with IMREAD_GRAYSCALE). Therefore, openCV loaded it as an RGB image and when I was accessing a pixel of the disparity with at(), I was specifying uchar as the wanted type. So I guess there was kind of a conversion from Vec3b to uchar that gave false values.
I timed a DDB drawing operation which uses multiple StretchBlt and StretchDIBits calls.
And I found that, time to complete is increase/decrease proportionally to the destination window size.
With 900x600 window it takes around 5ms, but with 1920x1080 it takes as large as 55ms (source image is 1280x640).
It seems Stretch.. APIs don't use any hardware acceleration features.
Source image (actually this is temporary drawing canvas) is created with CreateDIBSection because I need resulting (stretched and merged) bitmap's pixel data for every frame drawn.
Let's assume, Windows GDI is hopeless. Then what is the promising alternative?
I considered D3D, D2D with WIC method (write to WIC bitmap and draw it with D2D then read back pixel data from the WIC bitmap).
I planed to try D2D with WIC method because I will needed to use extensive text drawing feature sometime soon.
But it seems WIC is not that promising: What is the most effective pixel format for WIC bitmap processing?
I've implemented D2D + WIC routine today. Test results are really good.
With my previous GDI StretchDIBits version, it took 20 ~ 60ms time for drawing 1280x640 DDB into a 1920x1080 window. After switching to Direct2D + WIC, it usually takes under 5ms, also picture quality looks better.
I used ID2D1HwndRenderTarget with WicBitmapRenderTarget, because I need to read/write raw pixel data.
HWndRenderTarget is only used for screen painting (WM_PAINT).
The main advantage of HWndRenderTarget is that the destination window size doesn't affect drawing performance.
WicBitmapRenderTarget is used as a temporary drawing canvas (as Memory DC in GDI drawing). We can create WicBitmapRenderTarget with a WIC bitmap object (like GDI DIBSection). We can read/write raw pixel data from/to this WIC bitmap at any time. Also it's very fast. For side note, somewhat similar D3D GetFrontBufferData call is really slow.
Actual pixel I/O is done through IWICBitmap and IWICBitmapLock interface.
Writing:
IWICBitmapPtr m_wicRemote;
...
const uint8* image = ...;
...
WICRect rcLock = { 0, 0, width, height };
IWICBitmapLockPtr wicLock;
hr = m_wicRemote->Lock(&rcLock, WICBitmapLockWrite, &wicLock);
if (SUCCEEDED(hr))
{
UINT cbBufferSize = 0;
BYTE *pv = NULL;
hr = wicLock->GetDataPointer(&cbBufferSize, &pv);
if (SUCCEEDED(hr))
{
memcpy(pv, image, cbBufferSize);
}
}
m_wicRenderTarget->BeginDraw();
m_wicRenderTarget->SetTransform(D2D1::Matrix3x2F::Identity());
ID2D1BitmapPtr d2dBitmap;
hr = m_wicRenderTarget->CreateBitmapFromWicBitmap(m_wicRemote, &d2dBitmap.GetInterfacePtr());
if (SUCCEEDED(hr))
{
float cw = (renderTargetSize.width / 2);
float ch = renderTargetSize.height;
float x, y, w, h;
FitFrameToCenter(cw, ch, (float)width, (float)height, x, y, w, h);
m_wicRenderTarget->DrawBitmap(d2dBitmap, D2D1::RectF(x, y, x + w, y + h));
}
m_wicRenderTarget->EndDraw();
Reading:
IWICBitmapPtr m_wicCanvas;
IWICBitmapLockPtr m_wicLockedData;
...
UINT width, height;
HRESULT hr = m_wicCanvas->GetSize(&width, &height);
if (SUCCEEDED(hr))
{
WICRect rcLock = { 0, 0, width, height };
hr = m_wicCanvas->Lock(&rcLock, WICBitmapLockRead, &m_wicLockedData);
if (SUCCEEDED(hr))
{
UINT cbBufferSize = 0;
BYTE *pv = NULL;
hr = m_wicLockedData->GetDataPointer(&cbBufferSize, &pv);
if (SUCCEEDED(hr))
{
return pv; // return data pointer
// need to Release m_wicLockedData after reading is done
}
}
}
Drawing:
ID2D1HwndRenderTargetPtr m_renderTarget;
....
D2D1_SIZE_F renderTargetSize = m_renderTarget->GetSize();
m_renderTarget->BeginDraw();
m_renderTarget->SetTransform(D2D1::Matrix3x2F::Identity());
m_renderTarget->Clear(D2D1::ColorF(D2D1::ColorF::Black));
ID2D1BitmapPtr d2dBitmap;
hr = m_renderTarget->CreateBitmapFromWicBitmap(m_wicCanvas, &d2dBitmap.GetInterfacePtr());
if (SUCCEEDED(hr))
{
UINT width, height;
hr = m_wicCanvas->GetSize(&width, &height);
if (SUCCEEDED(hr))
{
float x, y, w, h;
FitFrameToCenter(renderTargetSize.width, renderTargetSize.height, (float)width, (float)height, x, y, w, h);
m_renderTarget->DrawBitmap(d2dBitmap, D2D1::RectF(x, y, x + w, y + h));
}
}
m_renderTarget->EndDraw();
In my opinion, GDI Stretch.. APIs are totally useless in Windows 7+ setup (for performance sensitive applications).
Also note that, unlike Direct3D, basic graphics operations such as text drawing, ling drawing are really simple in Direct2D.
I get the general idea that the frame.data[] is interpreted depending on which pixel format is the video (RGB or YUV). But is there any general way to get all the pixel data from the frame? I just want to compute the hash of the frame data, without interpret it to display the image.
According to AVFrame.h:
uint8_t* AVFrame::data[AV_NUM_DATA_POINTERS]
pointer to the picture/channel planes.
int AVFrame::linesize[AV_NUM_DATA_POINTERS]
For video, size in bytes of each picture line.
Does this mean that if I just extract from data[i] for linesize[i] bytes then I get the full pixel information about the frame?
linesize[i] contains stride for the i-th plane.
To obtain the whole buffer, use the function from avcodec.h
/**
* Copy pixel data from an AVPicture into a buffer, always assume a
* linesize alignment of 1. */
int avpicture_layout(const AVPicture* src, enum AVPixelFormat pix_fmt,
int width, int height,
unsigned char *dest, int dest_size);
Use
int avpicture_get_size(enum AVPixelFormat pix_fmt, int width, int height);
to calculate the required buffer size.
avpicture_* API is deprecated. Now you can use av_image_copy_to_buffer() and av_image_get_buffer_size() to get image buffer.
You can also avoid creating new buffer memory like above (av_image_copy_to_buffer()) by using AVFrame::data[] with the size of each array/plane can be get from av_image_fill_plane_sizes(). Only do this if you clearly understand the pixel format.
Find more here: https://www.ffmpeg.org/doxygen/trunk/group__lavu__picture.html
I am pretty new to Objective C and working with Cocoa Framework. I want to read an image and then extract the image data (just pixel data and not the header) and then write the data to a binary file. I am kind of stuck with this, I was going through the methods of NSImage but I couldn't find a suitable one. Can anyone suggest me some other ways of doing this?
Cocoa-wise, the easiest approach is to use the NSBitmapImageRep class. Once initialized with a NSData object, for example, you can access the color value at any coordinate as a NSColor object using the -setColor:atX:y: and -colorAtX:y: methods. Note that if you call these methods in tight loops, you may suffer a performance hit from objc_msg_send. You could consider accessing the raw bitmap data as C array via the -bitmapData method. When dealing with a RGB image, for example, the color values for each channel are stored at offsets of 3.
For example:
color values: [R,G,B][R,G,B][R,G,B]
indices: [0,1,2, 3,4,5, 6,7,8]
To loop through each pixel in the image and extract the RGB components:
unsigned char *bitmapData = [bitmapRep bitmapData];
if ([bitmapRep samplesPerPixel] == 3) {
for (i = 0; i < [image size].width * [image size].height; i++) {
int base = (i * 3);
// these range from 0-255
unsigned char red = bitmapData[base + 0];
unsigned char green = bitmapData[base + 1];
unsigned char blue = bitmapData[base + 2];
}
}