What could cause Perl system calls to start failing? - windows

A small request: I read Stack Overflow's Perl questions every day, and answer/contribute where I can; today I need the community's help!
Perl setup: I'm running Active Perl 5.8.8 on Windows. The installation is on our department server's local drive, which is also shared to the network. All department users run Perl on their own PCs by pointing to this network-installed Perl. This has worked for years, and isn't causing a problem, but it's a piece of info needed to understand the problem.
The server in question is also our "cron" (Scheduled Task) server, handling a variety of automation tasks. Suddenly last week, system calls in Perl scripts (on the server) started failing (details below). At first, I suspected a corrupt Perl installation, but all of the client PCs can still run the same Perl scripts without any issues, leading me to think it's a server issue. I've rebooted the server twice, and the problem is persistent, thus I need help!
Here are some examples of the various way that system calls are failing, boiled down to Perl one-liners:
% perl -e "system('dir')"
That should print a "dir" listing, but instead it opens a sub-shell. If I type "exit", I can exit the sub-shell, and I'm back in the original shell (confirmed by examining the shell history using the UP arrow key).
% perl -e "print `dir`"
This actually hangs. Nothing happens at all. If I Ctrl-C to kill the process, I get the message "Terminating on signal SIGINT(2)" and the DOS prompt comes back. But, any future commands in the DOS prompt (even just hitting ENTER) cause the error "The process tried to write to a nonexistent pipe.". You have to exit the DOS prompt as it's effectively useless.
Last example:
% perl -e "system('Z:/Scripts/rebuild.pl')"
'ebuild.pl' is not recognized as an internal or external command,
operable program or batch file.
In this case, Perl switches the forward-slashes (/) to DOS/Windows back-slashes (), which it has done just fine for years. But, Perl is interpreting the "\r" at the beginning of the "rebuild.pl" filename as a carriage-return (I think) and looking for the remaining "ebuild.pl". Calls to other scriptnames whose characters can't be misinterpreted like that result in the above hangs (if you use backticks) of sub-shells being opened (for system() calls).
I'm not just puzzled by this - I'm desperate! Our department server's "cron" jobs are useless right now since we use a lot of system calls.
Again, I don't think this is a corrupt Perl install, since the network users can run fine. So, what could happen on an individual machine (not tied to the Perl install itself) that could cause Perl's system calls to fail like this?
Environment settings, as requested:
ALLUSERSPROFILE=C:\Documents and Settings\All Users
APPDATA=C:\Documents and Settings\engmodem\Application Data
CDSROOT=Z:\Cadence\SPB_16.5
CDS_CONCEPT_NOSPLASH=TRUE
CDS_LIC_ONLY=1
CDS_SITE=Z:\Cadence\Sites\16.5
CHDL_LIB_INST_DIR=%CDSROOT%
CLIENTNAME=USENTUTTLJL3C
ClusterLog=C:\WINDOWS\Cluster\cluster.log
CommonProgramFiles=C:\Program Files\Common Files
COMPUTERNAME=CORPUSAPP5
ComSpec=C:\WINDOWS\system32\cmd.exe
CONCEPT_INST_DIR=%CDSROOT%
FP_NO_HOST_CHECK=NO
HOMEDRIVE=H:
HOMEPATH=\
HOMESHARE=\\PF1\HOME
ICMHOME=Z:\Software\PTC\INTERC~1
INSTDIR=%CDSROOT%
LOGONSERVER=\\ENGMAHO5
LSF_BINDIR=Z:\Software\LSF\bin
LSF_ENVDIR=\\hwc151\LSF_6.2\etc
MESSAGE=BROADCAST
NUMBER_OF_PROCESSORS=2
OA_PLUGIN_PATH=%CDSROOT%\Share\oaPlugIns
OS=Windows_NT
Path=C:\Program Files\Legato\nsr\bin;Z:\oracle\ora92\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Windows Resource Kits\Tools\;Z:\Software\Perl\5.8.8\bin;C:\Program Files\Oracle\jre\1.3.1\bin;C:\Program Files\Oracle\jre\1.1.8\bin;C:\Program Files\Support Tools\;Z:\Software\LSF\bin;C:\Program Files\PHP\;C:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files\EMC RepliStor;C:\GitStack\python;C:\GitStack\python\Scripts;C:\GitStack\git\cmd;Z:\Scripts;Z:\bin;Z:\Cadence\SPB_16.5\tools\bin;Z:\Cadence\SPB_16.5\tools\fet\bin;Z:\Cadence\SPB_16.5\tools\pcb\bin;Z:\Cadence\SPB_16.5\OpenAccess\bin\win32\opt
PATHEXT=.COM;.EXE;.BAT;.PL;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.VBS
PCB_LIBRARY=16
PERL5SHELL=cmd
PHPRC=C:\Program Files\PHP\
PROCESSOR_ARCHITECTURE=x86
PROCESSOR_IDENTIFIER=x86 Family 6 Model 29 Stepping 1, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=1d01
ProgramFiles=C:\Program Files
PROMPT=$P$G
PULLUP_DIFF_PAIRS=TRUE
SESSIONNAME=RDP-Tcp#1
SystemDrive=C:
SystemRoot=C:\WINDOWS
TZ=EST5EDT
VISUALSVN_SERVER=C:\Program Files\VisualSVN Server\
WF_RESOURCES=Z:\oracle\ora92\WF\RES\WFus.RES
windir=C:\WINDOWS

It turned out the reason of this weird behavior was incorrectly defined PERL5SHELL variable: cmd.exe (the shell interpreter in Windows) should be called with some parameters for proper processing - there parameters went missing after some updates. )
By the way, in The Doc it's said that Perl usually assumes the 'cmd.exe /x /c' line as a shell executable anyway if PERL5SHELL environment variable is not defined at all.
P.S. I really like this thread: it clearly shows the purpose of comments. )

Related

How can I execute Windows commands from Perl without leaving command windows open?

I have a Perl script on Win10 that uses the system() command to run a couple of different command line processes, including:
Start Windows Media Player with a specified .mp3 file.
my $cmd = "start call wmplayer.exe myRadoShow.mp3";
system($cmd);
Start another Perl program that does something else at the same time (specifically reads and broadcasts a set of timecodes and titles).
my $cmd2 = "secondScript.pl some_params";
system($cmd2);
All of this works correctly; the minor problem is that #1 above starts up a new command line window each time it executes the system($cmd) command. I have to later go back and close those windows.
If I don't use "start call" the Perl script doesn't continue to #2.
Is there a preferred way to execute #1 that doesn't leave these windows open?
I realize this question may be more about Windows commands than Perl.
Firstly, If youre using active perl versions, there is a wperl.exe you can call your script with instead of the default perl.exe. Not to sure if this will hide sub processes created by your script but you can give it a go.
If that doesnt work, then maybe you can use Win32::GUI:
use Win32::GUI;
my $hw = Win32::GUI::GetPerlWindow();
Win32::GUI::Hide($hw);
there are other modules that provide this functionality as well. Good luck!

How can I run a Bash shell script as a Build Event in Visual Studio?

I would like to run a Bash shell script (.sh) using the Windows Subsystem for Linux as part of a Build Event in Visual Studio, in a C++ project.
But there are lots of errors when I do this, and I can't seem to find the right combination of quotation marks and apostrophes and backslashes to either make Bash run in the first place, or to properly pass the path to the script.
How do I make Visual Studio run my Bash shell script as a build event?
(Feel free to skip to the bottom of this answer if you don't care about how to solve the problem and just want a command you can copy and paste!)
Overview
I run a number of Bash shell scripts as part of Build events in Visual Studio, and I used to use Cygwin to run them. Now that the Windows Subsystem for Linux is available, I spent some time switching my builds over to use WSL, and it wasn't as easy as I'd hoped, but it can work, with a little time and energy.
There are several issues you'll run into if you're going to do this:
The path to bash.exe may not be what you think it is, because under the hood, Visual Studio uses a 32-bit build process, so if you're on a 64-bit machine, you can't simply run the 64-bit bash.exe without getting the dreaded 'C:\Windows\System32\bash.exe' is not recognized error.
The path to your solution or project is a Windows path that uses backslashes (\), and those don't play nice in Unix, which prefers forward slashes (/) as a path delimiter.
The root drive of the solution, typically something like C:\, is meaningless gibberish in Unix; to reach the root drive in WSL, you'll need to use a mounted drive under /mnt.
The casing of the drive letter is different between Windows and WSL: In Windows, it's uppercase C:\, and in WSL, it's lowercase /mnt/c.
And to make it a little harder, we don't want to hard-code any of the paths: It should all Just Work, no matter where the solution is found.
The good news is that they're all solvable issues! Let's tackle them one at a time.
Fixing the Issues
1. The proper path to Bash
Per the answer given here, you'll need to use a magic path to reach Bash when running it from a Visual Studio build. The correct path is not C:\Windows\System32\bash.exe, but is actually
%windir%\sysnative\bash.exe
The magic sysnative folder avoids the invisible filesystem redirection performed by the WOW64 layer, and points to the native bash.exe file.
2. Fixing the backslashes
The next problem you're likely to run into is the backslashes. Ideally, you'd like to run a project script like $(ProjectDir)myscript.sh, but that expands to something like C:\Code\MySolution\MyProject\myscript.sh. At a minimum, you'd like that to be at least C:/Code/MySolution/MyProject/myscript.sh, which isn't exactly right, but which is a lot closer to correct.
So sed to the rescue! sed is a Unix tool that mutates text in files: It searches for text using regular expressions, and, among other things, can replace that text with a modified version. Here, we're going to pipe the path we have into sed, and then use some regex magic to swap the path separators, like this (with lines wrapped here for readability):
%windir%\sysnative\bash.exe -c "echo '$(ProjectDir)myscript.sh'
| sed -e 's/\\\\/\//g;'"
If you include this as your build event, you'll see that it now doesn't run the script, but it at least prints something like C:/Code/MySolution/MyProject/myscript.sh to the output console, which is a step in the right direction.
And yes, that's a lot of backslashes and quotes and apostrophes to get the escaping right, because Nmake.exe and bash and sed are all going to consume some of those special symbols while processing their respective command-lines.
3. Fixing the C:\ root path
We want to mutate the sed script so that it turns the C:\ into /mnt/C. A little more regex substitution magic can do that. (And we have to turn on the -r flag in sed so that we can easily use capture groups.)
%windir%\sysnative\bash.exe -c "echo '$(ProjectDir)myscript.sh'
| sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\1/i;'"
If you run this, you'll now see the output path as something like /mnt/C/Code/MySolution/MyProject/myscript.sh, which is almost but not quite correct.
4. Fixing the case-change in the root path
WSL mounts your disks in lowercase, and Windows mounts them in uppercase. Consistency! How do we fix this? Yet more sed magic!
The \L command can be used to tell sed to transform succeeding characters to lowercase (and there's an equivalent \U for uppercase). The \E command will switch output back to "normal" mode, where characters are left untouched.
Adding these in finally results in the correct path being output:
%windir%\sysnative\bash.exe -c "echo '$(ProjectDir)myscript.sh'
| sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\L\1\E/i;'"
5. Running it
This whole time, Bash has just been printing out the path to the script. How do we run it instead, now that it's the correct path?
The answer is to add `backticks`! Backticks cause Bash to execute the command contained within them, and to then use that command's output as the arguments to the next command. In this case, we're not going to output anything: We just want to run the output of sed as a command.
So including the backticks, here's the result:
%windir%\sysnative\bash.exe -c "`echo '$(ProjectDir)myscript.sh'
| sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\L\1\E/i;'`"
The Complete Solution
Here's what the whole solution looks like, for running a script named myscript.sh as a Build Event, in the current Project directory of the current Solution:
%windir%\sysnative\bash.exe -c "`echo '$(ProjectDir)myscript.sh' | sed -re 's/\\\\/\//g; s/([A-Z]):/\/mnt\/\L\1\E/i;'`"
Here's a screen-shot showing the same thing in Visual Studio 2017, for a real C++ project:
It's not easy, but it's not impossible.
If you have Git for Windows installed, try this. I found it simpler than installing WSL. The basic idea is to create an intermediate batch script to call your bash script, using Git bash's in-built bash or sh command from the batch script.
With Git for Windows, you'll have a Git\bin folder e.g. at:
C:\Program Files\Git\bin
Inside that directory you should see the bash.exe and sh.exe programs. So if you add that directory to your Windows Path environment variable then you'll be able to use sh and bash from the Windows command line. These commands will allow you to run your bash scripts "inline" within a CMD console window. That is, they won't spawn a new bash window; meaning the console output will be visible in your VS build.
From there, just create a .bat file which calls your .sh file using either the sh command or the bash command. Not sure the difference; we just use the sh command. So if your bash script is pre.sh, then your batch file would be just a single line calling the bash script:
sh %~dp0\pre.sh
if errorlevel 1 (
exit /b %errorlevel%
)
The %~dp0 assumes the batch and bash scripts are in the same directory. You then point your VS build event to the .bat file. The check for error level is necessary so that any failures from the bash script are forwarded up to the batch script. See: How do I get the application exit code from a Windows command line?.
To hook this in as a build event in VS2019 then, just follow the standard instructions for hooking in a .bat file: https://learn.microsoft.com/en-us/visualstudio/ide/specifying-custom-build-events-in-visual-studio?view=vs-2019.
Update: Beware Visual Studio's (VS's) Path Variable Behaviour
One thing we found quite frustrating with this solution was the tendency of VS to not load in the path variable correctly. It seems to prefer the user variable over the system variable. But even after we deleted the user variable, sometimes the path didn't seem to be getting picked up by VS, and we kept getting "sh is not recognised..." messages on our build console. Whenever that happened, restarting VS seemed to do the trick. Not very satisfying, but it gets us by.
Update: This is not a Full Unix Solution
Git for Windows does have a lot of Unix commands available, but not all of them. So in general, this won't work. For the general case, WSL is more robust. However, if it's just pretty lightweight Unix you need, this will suffice, and will likely be an easier approach for Windows users who would rather avoid the steeper setup cost of installing the full WSL.
Original idea to use Git bash came from here: https://superuser.com/questions/1218943/windows-command-prompt-capture-output-of-bash-script-in-one-step
Instead of backticks, you can wrap command with $( and )

problems in Perl script with localized user name on Windows

I have a perl script to start a script file with default program.
system("Start C:\\Temp\\test.jsx");
It works file with English user names but when I change user name to ai𥹖Ц中 it doesn't work.
Also no error message appears to I'm not able to debug.
perl on Windows uses so called ANSI functions to interface with the outside world. That means, if you use interesting characters (for example, certain Turkish letters on a US-English Windows install), perl cannot see them. As I wrote on my blog:
You can't pass characters that are outside of the Windows code page to perl on the command line. It doesn't matter whether you have set the code page to 65001 and use the -CA command line argument to perl: Because perl uses main instead of wmain as the entry point, it never sees anything other than characters in the ANSI code page.
For example:
$ chcp 65001
$ perl -CAS -E "say for #ARGV" şey
sey
That's because ş does not appear in CP 437 which is what my laptop is using. By the time it reaches the internals of perl, it has already become s.
So, there is not much you can do with a stock perl. I was working on a set of patches, but things intervened. I may still get around to it this summer.
Now, in your case, you are passing "interesting characters" to an external program via system. The same problem applies. Because perl is using the ANSI versions of functions used to spawn processes etc, the program spawned will not see a Unicode environment. So, if you are trying to use Korean or Japanese programs with a system code page that does not include them, I am not sure what will happen.
There is not much you can do once perl is running. The environment, command line arguments, everything lives in the ANSI world from that point on. There may be funky work-arounds, but for that one would need to know exactly how 'ai𥹖Ц中' gets from your perl program to the external program.

Cross-platform command line script (e.g. .bat and .sh)

I noticed that Windows 7 enables to execute .sh files as if they were .bat files. That got me wondering whether it is possible to write a .sh file such that it can be executed in Windows and Linux (let's say bash).
The first thing that comes to my mind is to fabricate an if-statement such that Windows and Ubuntu can deal with it and jump into the according block to execute plattform-specific commands. How could this be done?
Note: I know this is not good practice. I also know that scripting languages like Python are far better suited to solve this problem than a mixed-syntax command line script would be. I'm just curious...
You could use this:
rem(){ :;};rem '
#goto b
';echo sh;exit
:b
#echo batch
It's valid shell script and batch, and will execute different blocks depending on how it's run.
Modify the echo and #echo lines to do what you want.
AFAIK, you can't directly run .sh files from Windows' cmd.exe. For that you'll need a *nix emulation layer to run the shell. Check out Cygwin or Msys/MinGW

How do I send commands to the ADB shell directly from my app?

I want to send commands in the ADB shell itself as if i had done the following in cmd.
>adb shell
shell#:/ <command>
I am using python 3.4 on a windows 7 OS 64bit machine. I can send one-line shell commands simply using subprocess.getoutput such as:
subprocess.getoutput ('adb pull /storage/sdcard0/file.txt')
as long as the adb commands themselves are recognized by ADB specifically, such as pull and push, however there are other commands such as grep that need to be run IN the shell, like above, since they are not recognized by adb. for example, the following line will not work:
subprocess.getoutput ('adb shell ls -l | grep ...')
To enter the commands in the shell I thought I needed some kind of expect library as that is what 'everyone' suggests, however pexpect, wexpect, and winexpect all failed to work. they were written for python 2 and after being ported to python 3 and my going through the .py files by hand, even those tweaked for windows, nothing was working - each of them for different reasons.
how can i send the input i want to the adb shell directly?
If none of the already recommended shortcuts work for you you can still go the 'regular' way using 'subprocess.Popen' for entering commands in the adb shell with Popen:
cmd1 = 'adb shell'
cmd2 = 'ls -l | grep ...'
p = subprocess.Popen(cmd1.split(), stdin=PIPE)
time.sleep(1)
p.stdin.write(cmd2.encode('utf-8'))
p.stdin.write('\n'.encode('utf-8'))
p.stdin.flush()
time.sleep(3)
p.kill()
Some things to remember:
even though you import subprocess you still need to invoke subprocess.Popen
sending cmd1 as a string or as items in a list should work too but '.split()' does the trick and is easier on the eyes
since you only specidfied you want to enter input to the shell you only need stdin=PIPE. stdout would only be necessary if you wanted to receive output from the shell
time.sleep(1) isn't really necessary, however since many complained about input issues being faster or slower in python 2 vs 3 consider maybe using it. 'they' might have been using versions of 'expect' that need the shell's reply first. this code also worked when i tested it with simply swapping out and in the process with time.sleep(0)
stdin.write will return an error if the input is not encoded properly. python's default is unicode. entering by binary did not work for me in my tests like this "b\ls ..." but .encode() worked. dont forget the endline!
if you use .encode() there is a worry that the line might not get sent properly, so to be sure it might be good to include a flush().
time.sleep(3) is completely uneccesary, but if your command takes a long time to execute (eg a regressive search through the entire device piped out to a txt file on the memory card) maybe give it some extra time before killing anyhting.
remember to kill. if you didnt kill it, the pipe may remain open, and even after exiting the test app on the console the next commend still went to the shell even though the prompt appearsed to be my regular cmd prompt.
Amichai, I have to start with pointing out that your own "solution" is pretty awful. And your explanation makes it even worse. Doing all those unnecessary things just because you do not understand how shell (here I mean your PC's OS shell, not adb) command parsing works.
When all you needed was just this one command:
subprocess.check_output(['adb', 'shell', 'ls /storage/sdcard0 | grep ...']).decode('utf-8')

Resources