I need to automate a windows only java graphical app in a cloud vm.
I'm currently trying with windwos server 2008 and robotJS in node, if I'm connected to the VM in RDP and showing the screen, all is good and it work, but it's no real automation.
If managed to launch the user session at start, and added the app and my script in the startup dire to have them launched when the user logon, but when I do that, the robot only see a black screen with the start button when I'm taking a screenshot, and it can't interact with the app.
As someone already try to do that ?
Any insight would be good, I can change the server version, use another automation tool, anything if it work.
I let that here because it might one day help someone.
The trick was to use keyboard shortcut instead of mouse, not sure why.
Related
We are using Guacamole for developing an application with RDP. We did POC using code from the following GitHub repositories:
https://github.com/wwt/guac
https://github.com/wwt/guac-vue
Configuration details are:
Windows Server 2016. RDP service is running here.
With this setup, we could successfully access the application remotely, however, the "minimize" action is not behaving as we expected. That is, the remote app window vanishes when we press the "minimize" button and a black screen is shown.
We could get back to the app by pressing 'Alt+Tab' combination but what we are expecting here is that the app getting placed at the bottom of the window showing three buttons: "Minimize, Restore and Close", so that we can take the further action. (As shown below.)
Has someone come across such a scenario and was able to address the need? Any help will be highly appreciated.
We came across this thread that talks about a similar problem but it doesn't have a solution.
We also explored official documentation of guacamole but had no luck.
Thanks in advance!
I believe you are starting the application using RemoteApp mechanism. The RemoteApp means that the remote application will be started integrated with the local computer desktop. The local computer desktop, or better window manager, will handle the minimise action.
In case of the Guacamole, the "local computer desktop" is the browser window, which does not have window manager. This means that there is no place for app to go when minimised.
You may try the Guacamole parameter "initial-program" instead of "remote-app". This parameter will launch the application immediately upon the connection is established, but the session will also have full desktop from the remote machine.
I understand that desktop/GUI apps are not supported in Windows containers. They do run but there's no built-in way to interact with them. I had the following idea - maybe I could use the Desktop Sharing API (https://learn.microsoft.com/en-us/windows/win32/api/_rdp/) for this purpose, the idea is to run a desktop app, then run a sharing program that uses the Desktop Sharing API, and connect to it using a Desktop Sharing API viewing program from the host.
I had to do some recap about window stations and desktops, and I noticed that when starting the container with cmd in interactive mode, I'm logged with ContainerAdministrator as a service (logon type 5). I tried running some WinAPI functions that deal with desktops and winstation and got some access denied results, so I switched to running cmd as system.
The window station of the cmd process (and other child processes) is not the interactive WinSta0, but instead some other service window station, which makes sense since I'm logged on as a service, and I figured that I can't use this window station, so I used a little program I wrote to run notepad in Winsta0 in the Default desktop. Afterwards I ran another program that enumerates the windows on WinSta0\Default, and the notepad window does get enumerated and I also get it's title, so it's running somewhere.
So now I tried running the desktop sharing API program (also on WinSta0\Default). It runs and I can connect from the host, but I only get a black screen without anything on it. I also tried running a program that takes a screenshot of the windows but I get an empty bitmap.
So I thought maybe the Default desktop is not the active desktop, and by using the OpenInputDesktop function I could confirm it - the current desktop was the Winlogon desktop, so I used the SwitchDesktop function to switch to the Default desktop (I used OpenInputDesktop again to verify that it actually worked).
Unfortunately, this didn't change anything, I still get an empty screen and empty bitmaps.
I know that containers are built for micro services and are not supposed to run GUI apps and so, but still - is there a way to make this work? Or any ideas of what else I can check? Alternatively, if you know that it can't work - I would also be happy to hear a good technical explanation of why it doesn't work.
I'm setting up Selenium Grid. We have a separate machine for each version of the browser.
Each node is started as System Service running on LocalSystem account with interactions with user desktop enabled.
This is required because Selenium Grid node starting Internet Explorer have problems making screenshots and transferring them when there was no interaction with user desktop.
As far as I was able to check, it looks like that interactions with user desktop for service are only allowed for LocalSystem account. Event changing manually flags in registry does not seem to works (as it was in windows 2008)
Everything is working fine except the test where I need to perform upload of the file. When there is an action to open dialog for file browsing, following popup appears
Is there any way to prevent this (creation of folder Desktop does not seem to work) ?
From the other hand, if there is a way to run service under different account with interactions with user desktop enabled, that would also be a case.
I would appreciate any help because I'm stuck with the problem
I have checked some additional solutions, like running selenium grid nodes via PowerShell Invoke-Command and this did not worked too.
I have managed to run selenium grid nodes as Windows Service with desktop interactions using 3rd party tool FireDaemon Pro Service Manager.
I didn't try this but PsExec -s should work
https://technet.microsoft.com/en-us/sysinternals/bb897553
I want to write a script that involves mouse click and stuff. I am little confused though. Is there a way mouse clicks (in the form of x and y co-ordinates) would work on a VPS which is logged on, but disconnected from the client?
Example:
MouseClick("left",588, 268)
To my knowledge, these VPSes don't have their own native screen resolution. I have already written the script for my local pc and it's working fine on my resolution. But when tried with the VPS, it doesn't work.
PS: I am running the script from the remote desktop itself and not from my local pc. Please correct me if there's anything I am doing wrong :)
Update: For anyone still looking for an answer, consider installing a vnc server, that way the screen resolution you specify is preserved even when not connected to the server.
Ex: https://www.realvnc.com/en/connect/download/vnc/
It's relevant to almost any automation tool, not only AutoIt.
Remote Desktop has its own GUI context only if it's connected to the remote PC. But you can lose focus for a Remote Desktop window (without disconnection & without minimizing) and continue other local work.
The same effect takes place for VNC server software.
One team in our organization uses it for massive test runs automation: a master server creates a remote session for each test machine and keeps them all connected until tests passes.
Use ControlClick to have the mouse click over RDP connections, even when you're not connected. For some reason it can still be iffy for me, but at least it worked more so that Click which was none at all.
I need to run a GUI script (AutoHotKey, which makes mouse clicks and press keys) on:
A different windows user (i.e. I run the script, then switch user
while keeping the session active)
A remote Amazon windows server (using remote desktop) where I also
run the script then close the Remote Desktop while keeping the
instance running
Unfortunately, in both cases, the script doesn't run as it seems that Windows enters a "GUI-less" mode where all GUI components are not active anymore.
Is there any solution to this?
Any hint would be greatly appreciated as I've now lost days trying to solve this!
Many thanks, Thomas
I do not believe this is possible due to the way Remote Desktop is implemented. When you close the RD connection, the GUI is no longer drawn. Therefore, AutoHotkey is unable to perform mouse clicks and key presses.
A possible workaround would be to make some registry tweaks (if possible on your server) which allow GUI interaction while minimized.
Locate HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client
Create RemoteDesktop SuppressWhenMinimized as DWORD with a value of 2
Registry Tweak Source