Execute Lua WINAPI code without displaying the shell - shell

Question

I created a simple Lua application using Alien for Lua. It works perfectly, except when you execute it, the Lua Shell shows as well. Is there a way to "hide" this shell, run it in background, turn it off, etc so that I simply see the message box?
Code:
require "luarocks.require"
require "alien"
local MessageBox = alien.User32.MessageBoxA
MessageBox:types{ret = "long", abi = "stdcall", "long", "string", "string", "long" }
MessageBox(0, "Hello World!", "My Window Title", 0x00000040)
Current Output:
Desired Output:

Answer 1

tl;dr
Rename your script to hello.wlua so that wlua.exe is used.
Details
While it is likely possible, if verbose, to locate and close the offending console window that Windows provided your process, it would be better if that console never appeared in the first place. If it does appear, then it is likely to flash on screen, and cause some users to be confused.
Subsystems
Windows has, since its earliest days, had the concept of a "subsystem" which each individual executable identifies with. Normal GUI applications are linked with /SUBSYSTEM:WINDOWS and get the full GUI treatment including the responsibility to create and display their own window(s) if and when needed.
Applications that expect to be run from a command line (or batch file) are linked with /SUBSYSTEM:CONSOLE, and as a result have standard file handles that are guaranteed to be open and are likely to be connected to some console window (or a pipe, or redirected to a file, but they do exist). That guarantee is strong enough that when a console program is started outside of a console (as when double-clicked from Exporer, or named in the Start|Run box) then the system automatically creates a console window for it, and binds the standard file handles to the new console.
There are other subsystems, but those two are the only important ones for normal users and developers.
lua.exe and wlua.exe
So why does this matter?
The stock lua.exe will be linked for the console, because that makes it possible to use interactively from a command prompt. However, it means that it will always be supplied with a console window even when you don't want one.
The Lua for Windows distribution (which from the pathname showing in your console's title bar it looks like you are using) includes a second copy named wlua.exe which only differs by being linked for the Windows subsystem. As a result, it only displays a window if the script explicitly creates one to display. Of course, it also means that it cannot be used interactively at the command prompt.
File types and associations
For convenience, you can associate the file type .wlua with wlua.exe, and name your GUI script with that file type. That will enable launching programs in the usual way without getting the extra consoles. Of course, when debugging them, you can always run them with lua.exe from a command prompt and take advantage of the existence of stdout and the utility of the print function.
On my PC (64-bit Win 7 Pro) I have the following associations, which look like they were created by the installation of Lua for Windows:
C:...>assoc .lua
.lua=Lua.Script
C:...>ftype lua.script
lua.script="C:\Program Files (x86)\Lua\5.1\lua.exe" "%1" %*
C:...>assoc .wlua
.wlua=wLua.Script
C:...>ftype Wlua.script
Wlua.script="C:\Program Files (x86)\Lua\5.1\wlua.exe" "%1" %*
Extra credit: PATHEXT
You could also add .lua to the PATHEXT environment variable to save typing the file type at the command prompt. I'm not configured that way presently, but have in the past done that. I found that the standard practice of naming both modules and scripts with the same file type made that less useful.
The PATHEXT environment variable lists the file types that will be searched for in the PATH when you name a program to run without specifying its file type. Documentation for this is rather hard to locate, as there does not appear to be a single MSDN page listing all the "official" environment variables and their usage. This chapter of a book about Windows NT has a nice description of the interaction of PATH and PATHEXT, and despite being subtly out of date in some respects, it is the clearest detailed explanation of how the command prompt operates that I've come across.
It clarifies that each folder in PATH is searched for each extension named in PATHEXT:
If the command name includes a file extension, the shell searches each directory for the exact file name specified by the command name. If the command name does not include a file extension, the shell adds the extensions listed in the PATHEXT environment variable, one by one, and searches the directory for that file name. Note that the shell tries all possible file extensions in a specific directory before moving on to search the next directory (if there is one).
It also documents how file types and associations interact with the command prompt. Despite its age, it is well worth the read.

Answer 2

Windows executables explicitly list the subsystem they run on. As the windows "lua.exe" is linked for the console subsystem, windows automagically creates a console window for it. Just relink "lua.exe" for gui subsystem, and you won't get to see the output any more unless you run it from a console window. BTW: Gui programs can programmatically create the console.
An alternative is closing the created console on start.
For that, you must first use SetStdHandle to redirect STDIN, STDOUT and STDERR (use a file open to device nul if you don't want it at all), and then call FreeConsole to finally dismiss your unloved console window. No sweat, you have "alien" set up already...

Answer 3

Programmatic solution (run the same script under wlua.exe if possible)
do
local i, j = 0, 0
repeat j = j + 1 until not arg[j]
repeat i = i - 1 until not arg[i-1]
local exe = arg[i]:lower()
-- check if the script is running under lua.exe
if exe:find('lua%.exe$') and not exe:find('wlua%.exe$') then
arg[i] = exe:gsub('lua%.exe$','w%0')
-- check if wlua.exe exists
if io.open(arg[i]) then
-- run the same script under wlua.exe
os.execute('"start "" "'..table.concat(arg,'" "',i,j-1)..'""')
-- exit right now to close console window
os.exit()
end
end
end
-- Your main program is here:
require "luarocks.require"
require "alien"
local MessageBox = alien.User32.MessageBoxA
MessageBox:types{ret = "long", abi = "stdcall", "long", "string", "string", "long" }
MessageBox(0, "Hello World!", "My Window Title", 0x00000040)

Answer 4

If you can use winapi module or have similar calls in Alien, you can find the handler of the console window and hide the window itself. The code would be similar to this:
require winapi
local pid = winapi.get_current_pid()
local wins = winapi.find_all_windows(function(w)
return w:get_process():get_pid() == pid
and w:get_class_name() == 'ConsoleWindowClass'
end)
for _,win in pairs(wins) do win:show_async(winapi.SW_HIDE) end
You'll need to check if this leave the MessageBox visible or not.