Use cuda to get screenshot of visible application - windows

I am owner of a Nvidia GTX 1060. I know that with Windows it is not possible to directly access the video memory of the graphics card. But I read somewhere on an email list that it is possible to fetch the pixels that are displayed right now on the monitor with use of the CUDA API. Unfortunately I found no code example for this.
What I want is something like an bitmap-like array containing the rgb values of the pixels
arr[0] = 255 // red
arr[1] = 255 // green
arr[2] = 0 // blue
arr[3] = 0 // alpha
arr[4] = 255
...
Can someone please show me an example how to do it with CUDA?
I know there is a way using DirectX or openGL. I already tried it out but it is either too complex or too slow and won't work with fullscreen and windowed applications at the same time.

What you are describing is not only impossible, it may be even seen as a security flaw. In general, one application should not be able to retrieve the shown window of another application without some kind of consent and control from the operating system.
I remember reading about exactly that kind of a flaw few years ago, but I couldn't find the article anymore. However, there are many other articles discussing similar issues. For example in:
http://ece.neu.edu/groups/nucar/NUCARTALKS/snp14-gpusteal.pdf
they discuss GPU vulnerabilities allowing a malicious program to partially reconstruct a webpage shown in Chromium.

Related

How to use 'D3d Shared Texture' method for screen shots?

I am trying to create a code that takes series of screenshot of a game, use Open-CV to edit it, and display it in a window, all in real-time. I am using windows operating system. For taking screenshots I have tried almost all the methods answered in this question. None of them gives FPS of more than 30, but do affect the performance of the game.
I found out that for games OBS Studio uses a different method called D3d shared texture to capture frames from the game directly on the GPU..
I searched for shared texture in OBS github code but I find the results confusing.
Searching on search engine does not give relevant results.
Can someone explain what is this shared texture method and how do I implement it in my code?

AccessibleObjectFromPoint and per-monitor DPI

I'm using accessibility with the AccessibleObjectFromPoint function, and I'd like it to work correctly on a per-monitor DPI environment. Unfortunately, I can't get it to work. I tried many things, and the situation for now is:
My app is marked as per-monitor-DPI-aware in the manifest. (True/PM)
I use GetCursorPos and then AccessibleObjectFromPoint.
How can the problem be reproduced:
Have two monitors, one with 100% DPI, the other with 125%.
Run Chrome on the 125% monitor.
Use AccessibleObjectFromPoint on one of the tab names, it won't work.
It works with some apps (DPI-aware, it seems, like explorer), but doesn't work with others. I tried several relevant functions, such as GetPhysicalCursorPos and PhysicalToLogicalPointForPerMonitorDPI, but nothing works.
It's worth noting that Microsoft's inspect.exe works as expected.
I’ve been struggling with this exact same problem for several weeks and can now tell you my findings. Unfortunately I can’t give you more than a hint of code, because the project I am working on, is proprietary.
The issue started at Windows 8.1. The problem did not exist on Windows 7 or Vista, because AccessibleObjectFromPoint always used raw physical coordinates, as documented here: https://msdn.microsoft.com/en-us/library/windows/desktop/dd317984(v=vs.85).aspx .
“Microsoft Active Accessibility does not use logical coordinates. The following methods and functions either return physical coordinates or take them as parameters.” This has not been true since Windows 8.1.
AccessibleObjectFromPoint now uses a flawed calculation that cannot always find the correct window for reasons similar to my question here: High DPI scaling, mouse hooks and WindowFromPoint .
My findings lead me to one conclusion: The API is broken. This does not mean it is not possible though.
Possible solutions I have partially tested that seem to work follow.
Prerequisites are that you
1/. Make your process per monitor DPI aware, NOT USING THE MANIFEST (more on that later).
2/. Determine the hWnd of the window you want to query (WindowFromPoint() variants)
3/. Determine the monitor DPI of the queried hWnd
4/. Determine the DPI of your process
5/. Determine the DPI of the queried hWnd
6/. Determine the monitor origin and offset for the queried hWnd (MonitorFromWindow() and GetMonitorInfo() )
Next, depends on your platform
Windows 10.0.14393+
Write a function that finds the IAccessible (AccessibleObjectFromWindow() ) from the top level window, and then recursively call IAccessible::accHitTest until you reach the bottom-most IAccessible and perhaps ChildID data. Return that as if you would call AccessibleObjectFromPoint.
To call it successfully, you will need to scale the (x,y) co-ordinates into the scale system of the queried hWnd, using the DPIs and co-ordinates fetched in the list above. Watch out for systems where monitors are not the same size or if monitors are partially offset, or above and below.
And now for the important part for 10.0.14393 – Set your thread to the same DPI_AWARENESS_CONTEXT of the hWnd you are querying. Now call your new function. Now revert your thread to monitor DPI aware, and voila, it works, even if the window is not maximised. This is why you must not use the manifest.
If you are on Windows 8.1 to 10.0.10586 you have a tougher task.
Instead of calling accHitTest, as above, you have to recursively call AccessibleChildren and iterate the call IAccessible::accLocation to determine if your test point is within each child. This is tricky and starts to get really messy when you get to e.g. combo boxes in products like Office, which is only system DPI aware.
That’s all I can give you for now.
To do it successfully on multi-platform (mine has to work from Vista to Windows-Current) the only really safe bet is to write a wrapper DLL in C++ that can determine at runtime which OS it is on and change code path accordingly. The reason you want to do it in C++ is to avoid passing IAccessible objects across the .Net/unmanaged marshalling boundary. You can call IUnknown::Release on objects you don’t need to return n the unmanaged side. You can do it all in .Net, but it will be slow.
P.S. also watch out for Chrome returning infinite trees where parents are children of their parents, some snity checks are required. Also, Chrome does not return accRole correctly, and will give you HTML tags instead of VT_I4.
Good luck
A fairly workable solution is as follows, in your IAccessible recursive function:
Use getwindowrect to capture the physical right on main window
Use accChild.accLocation in loop to capture left and Width on each Object
Add this simple test
If l > rct2r.Right And l > arrIACC.x2 Then
arrIACC.x2 = l + w
End If
if dpi = 100 then no Object is furter out than physical right
if dpi > 100 then closebutton is...x pix offset
Use the difference to rescale all values you are in use of Width
arrIACC.w1 = CInt(((-rct2r.Left + arrIACC.w1) / arrIACC.x2) * rct2r.Right)
This solution is from an Excel plugin I have developed, I was testing the Width of the quick access toolbar qat and my result was +- 5 pixels regardless of any DPI.

Qt, CEGUI or wxWidgets for a text game GUI?

I tried to sign up, but I was unable; perhaps a problem from my side. Hopefully I'll get an answer as anonymous.
I apologize for the grammar/syntax, but English isn't my native language.
Recently I lost my job, so I have enough spare time to try something fun. I decided to create a simple text RPG game for me and some friends. It will very close to the board games like Talisman, Dungeon Run, and HeroQuest, using dice and a simple attribute/skill system. So no 3d graphics. The only 2d element, if I decide to include it, will be a map
that will allow the hero to move between locations. Currently I'm using Windows XP SP3, for the game I use wxDev-C++, and although cross platform would be cool, I don't really care.
I have some experience in C++ (currently using wxDev-C++), but I'm far from being called an expert or even a great programmer. I was about to start writing parts of the code, but I decided to check if creating a GUI for the game is possible. In some forums, many suggested I use Qt, CEGUI or wxWidgets, but most examples I saw are grey boxes that are
indifferent at best, when I want something that fits better in a fantasy setting. I don't claim I would do better, but I want a GUI that is more fantasy related.
What I want from the GUI:
1. A "cool" Gui with decent graphics. I could even create an image to serve as a mask in Photoshop, but the GUI builder will have to support imported images.
2. A relatively large textbox in the middle (with a scrollbar) that will display die rolls, damage and options.
3. The ability to display dynamically values (like the change in the health after each action without requiring to refresh manually)
4. Display an icon or a small image of the character in the area where I display stats/abilities.
5. Open new windows created with tha same GUI builder to allocate points, buy/sell things and open a map.
About the map in the game: I decided to create a map in photoshop. When the hero decides to move to another location, a new window will open showing the map. I thought of 2 possible ways to move between locations: 1) Create hotspots on the image and select one by clicking on the name of the location.(I dare not think about the complexity of this so we
move to idea #2) and 2) Have the image as a backgroung to a grid with vertical and horizontal coordinates. When the hero selects a new area to visit, he clicks on the area, but what he really does is click on the grid, which returns the two values (x,y) of the location and informs the game about the area the hero wants to visit.
Yeah, yeah, I know it's too much, so what I'm most interested in are the 1-3. I know that even if they are possible, it will propably take forever, but as I said I have spare time, and I like learning new things. I apologize for the size of the post, but I decided to post as many info as possible so you know what I want.
If any of you has used Qt, CEGUI or wxWidgets could you tell which covers most of my criteria? I saw some great stuff build with CEGUI, but I don't know if it is too hard to learn?
Thank in advance.
I know my answer comes pretty late, I only recently started using stackoverflow fairly recently, but maybe this response will help anybody.
CEGUI fully supports skinning widgets using XML. Our CEED editor (WYSIWYG) fully supports layout editing, but the skinning editor (LNF editor) is not finished as of now (11.11.2014), the development version supports exchanging images however and changing sizes and proportions, but more advanced adjustments have to be done in XML.
CEGUI has an imageset editor, fully supported by the CEED editor. Creating imagesets (sets of named subimages, with position and dimension inside a big texture atlas) is supported there. Additionally there is a way to create imagesets from just a bunch of jpg/png/... files using a tool. You would have to ask for specifics in the forum though because it is not integrated into CEED yet.
So basically with CEGUI you are free to make whatever fantasy GUI you want. Skinning simple elements like buttons and progress bars isn't much work in XML anyways. Without the finished editor, some more advanced widgets are more work to skin, but many skins have already been created done this way and some of them are even publically available in the forum and in the CEGUI stock files.
StaticText widgets supports what you want, you can even use images in there or change fonts and colours in the text if you want. Scrollbars are supported too.
I am not sure what you mean by this. You have to specify this.
A simple "Generic/Image" widget is available in CEGUI for this purpose. You can use precreated images or even RTT textures.
You can create and destroy windows in CEGUI without issues.
Regarding the map: I m not sure what you mean, but getting the position of a click in respect to an image (representing the map) is possible in CEGUI.
CEGUI is not particularly hard to learn. There is always the forums and the chat if you got questions. For an Open Source project it is quite well documented so if you read all of the API docu, and look at the supplied samples in the sample browser, you should already get quite far. And for everything additional there is the forum (search), the IRC chat and a community wiki (mind the targeted versions of an article there though)
For a project like yours, CEGUI seems perfectly suited (this is what it was created for in the first place). Qt is not really optimal for games for numerous reasons. wxWidgets I have never used.

Setting up OpenGL/Cuda interop in Windows

I am writing a DLL that needs to do some work in Cuda 3.2 and some work in OpenGL. OpenGL will render some grayscale images that my Cuda code needs to read in and modify, and then give back to OpenGL as a texture. I believe I need to create PBOs to do that. I have done some basic OpenGL stuff before but never worked with extensions, and that's where my problem is - I've been searching for 2 days and so far haven't been able to find a working example, despite wading through pages and pages of code. None of the samples I've tried work (and I'm sure my vid card will support it, being a GTX470)
Some specific questions:
1. I installed the nvidia opengl sdk. Should I be using glew.h and wglew.h to access the extensions?
2. My DLL does not have any UI - do I need to create a hidden window or is there an easier way to create an off-screen rendering context?
3. Can I create a grayscale PBO by using GL_RED_8UI format? Will both cuda and gl be happy with that? I read the opengl interop section in the cuda programming manual and it said GL_RGBA_8UI was only usable by pixel shaders because it was an OpenGL 3.0 feature, but I didn't know if that applied to a 1-channel format. 1 channel float would also work for my purposes.
4. I thought this would be fairly easy to do - does it really require hundreds of lines of code?
Edit:
I have code to create an OpenGL context attached to a HBITMAP. Should I create a bitmap-rendering context and then try to attach a PBO to that? Or will that slow me down by also rendering to CPU memory? Is it better to create an invisible window and attach the PBO to that? Also, does the pixel format of my PBO have to match the window/bitmap? What about the dimensions?
Thanks,
Alex
There's actually an example of how to use OpenGL and CUDA together. Look at the SimpleGL example.
You may want to take a look at this example:
https://github.com/nvpro-samples/gl_cuda_interop_pingpong_st

how to take screenshots of virtual box

how to take screenshots of palm emulator
We use HyperSnap 6 for all our screen capture needs. It's quite capable of capturing screen information inside a VMWare guest from the actual host machine and I suspect it would have little trouble with a Palm emulator as well, since it basically operates at the host level.
You can easily capture dynamic ranges (where you use the mouse/keyboard to mark top-left and bottom-right corners) as well as pre-specified ranges like "same range as last time" or "the current window" with simple key combinations (this last one is ideal for our demo and documentation purposes).
One other nice feature is the fact that it's like a mini-paint program (actually, it's a lot better than MSPaint) which allows us to highlight important bits of the image with markers and such, before saving.
Sorry to sound like an advert, but you did ask :-)
you can always take screenshots natively. orange + sym + s i believe (not sure what keys they are in emulator, but those are ones on the phone). you can then send that photo to wherever.

Resources