not sure if this is a perl problem, or a cywin problem, or a Windows problem:
I'm running perl inside cygwin under Windows8. We have a comprensive number of small scripts for individual tasks, and I've recently written a top level script, which repeatedly calls several of these scripts via 'system' calls. All scripts for themselves run flawlessly, however execution is only happening in chunks, i.e. the top level script starts to operate, and after about 10 seconds it stops and the computer is idle for another 10-15 seconds, then starts again for 10 seconds, and so on. Apart from this script the PC is only running the usual Windows background processes, i.e. the top level script is the only process causing significant CPU load.
The script is too long to show here, but essentially consists of stuctures where a few variables are defined in loops and then combined via sprintf strings to call the scripts , just like the following snippet:
(...)
foreach $period (#periods)
{
foreach $wt (#wtlist)
{
foreach $type ('WT', 'Ref')
{
$out=1;
$dir1=0*$sectorwidth;
$dir2=1*$sectorwidth;
$addfile0 = sprintf("%s/files/monthly_recal/%s%s_from%s.%s_%03d.da1", $workingdir_correl, $nameRoot, $wt, $type, $period, $dir1 ) ;
$addfile1 = sprintf("%s/files/monthly_recal/%s%s_from%s.%s_%03d.da1", $workingdir_correl, $nameRoot, $wt, $type, $period, $dir2 ) ;
if (-e $addfile0 && -e $addfile1)
{
$cmd = sprintf ( "perl ../00Bin/add_sort_ts.pl $addfile0 $addfile1 %s/files/monthly_recal/tmp/%s%s_from%s.%s.out%02d 0 $timeStep\n", $workingdir_correl, $nameRoot, $wt, $type, $period, $out ) ;
print ($cmd);
system ( $cmd ) ;
}
}
}
}
(...)
All variables are defined (simple strings or integers) and the individual calls are all working.
When this top level script is running, it's running several loop iterations after another, so I don't think it's a matter of startup delays of the called scripts. To me it looks more as if Windows denies too many system calls in a row. I have other perl scripts withut 'system' calls, which run for 10 minutes without showing this intermittent behaviour.
I have no real clue where to look for, so any suggestion would be appreciated. The whole execution time of the top level script can take several hours, therefore any improvement here would greatly improve efficieny!
--UPDATE: From the discussion to Hakon's answer below it turned out that the problem lies in the shell that is used to run the perl scripts - intermittent operation appears when the code is run from Windows cmd or a non-login shell, but not when run explicitly from a login shell (e.g. when using bash --login or staring mintty -). I will open another thread soon to clarify why this happens... Thanks to all contributors here!
Can you try to simplify you problem a little bit? It will help to locate the problem more easily. For example, the following code running on Windows 10, Strawberry Perl 5.30 (running from CMD not Cygwin) shows no problems:
use strict;
use warnings;
for my $i (1..10) {
system 'cmd.exe /c worker.bat';
}
with a simple worker.bat like:
#echo off
echo %time%
timeout 5 > NUL
echo %time%
The output from running the Perl script is:
10:35:00.71
10:35:05.18
10:35:05.22
10:35:10.17
10:35:10.22
10:35:15.15
10:35:15.19
10:35:20.19
10:35:20.23
10:35:25.15
10:35:25.20
10:35:30.16
10:35:30.21
10:35:35.14
10:35:35.18
10:35:40.16
10:35:40.20
10:35:45.16
10:35:45.21
10:35:50.15
Showing no delays between the system calls.
Related
I have inherited an application which is written in dozens (maybe hundreds, I haven't counted exactly) of PostgreSQL functions. In order to check the application code into git and be able to easily work on specific functions, I used pg_extractor to export the database into a separate file for each function.
In order to easily apply updates from git (both on developer machines and in production), I wrote a bash script that uses the psql command line client to run all of the function files, which causes the database server to be updated to match the files from git.
The gist of it looks like this (with some initialization code replaced by comments for brevity):
#!/bin/bash
# Check if a .env file is present and load it to set the PGHOST, PGPORT, PGUSER, PGPASSWORD, and PGDATABASE environment variables
# Check that `psql` is on the PATH
# Check the the database set in PGDATABASE exists on the server
GREEN='\033[0;32m'
RED='\033[0;31m'
NC='\033[0m' # No Color
makeFunction () {
echo -n "CREATE OR REPLACE FUNCTION $2"
psql -q -x < "$DIR/$1/functions/$2.sql" > /dev/null
if [ $? -eq 0 ]; then
echo -e " - ${GREEN}COMPLETE${NC}"
else
echo -e " - ${RED}FAILED${NC}"
exit 1
fi
}
for schema in admin admin_internal main main_internal
do
if [ -d "$DIR/$schema/functions" ]; then
for function in $DIR/$schema/functions/*.sql
do
makeFunction $schema $(basename "$function" .sql)
done
fi
done
On most of our Linux machines (development and production, Ubuntu 16.04 and 18.04) this script takes 15-20 seconds. Example:
real 0m14.324s
user 0m6.894s
sys 0m1.742s
However, on our Windows development machines (when run using git-bash) it usually takes around three minutes to run the same script. Example:
real 3m0.825s
user 0m3.525s
sys 0m11.943s
(Thinking the issue might be with Bash on Windows, I tried converting the script to PowerShell, only to see the same issue. Thankfully I saw that it wouldn't make a difference while doing partial testing before spending too much time on it.)
It turns out that the problem is in actually making the connection to the PostgreSQL server. For example, using time psql -lqt to time listing all databases on the server (these are example numbers, but dozens of test runs have shown that they are consistently the similar to these):
On Ubuntu:
real 0m0.055s
user 0m0.032s
sys 0m0.020s
On Windows:
real 0m0.852s
user 0m0.000s
sys 0m0.030s
As you can see, it takes 15 times longer on Windows. Extend that out to all the times we are calling psql in the update script, and it's no wonder that it takes 9 times longer to run the full script on Windows than on Linux.
It is well known that Postgres performance will never be as good on Windows as Linux because of the one-process-per-connection model and the lack of fork() on Windows, but that should be a bottleneck in the creation of the connection, not in the execution of the commands. (That the bottleneck is in the connection and not the query execution is evidenced by the fact that a single example command is consistently 15x slower, but the whole script with much larger queries being run is only 9-12x slower.)
Is there a way to make this faster for our Windows users? Is there some way to reuse an existing psql session and pipe additional files into it? Or is my only option to rewrite this in some other language and write my own database communication code that reads the files and pipes them to PostgreSQL myself?
Running the connection through PgBouncer instead of directly to Postgres makes a huge difference.
Using it, my script run on Windows is reduced to around 30 seconds.
It's still not quite as fast as on Linux, but I can live with "only" a 6x improvement.
real 0m33.232s
user 0m2.740s
sys 0m9.785s
I am trying to start a number of jobs in different screens from a shell script. Each job will read in a different value of a parameter from a premade input file and run a simulation based on that value, then tee or > the output to a differently named file. So in a do loop around all the jobs, job 40 on screen "session40" will read in line 40 of the input file, run the simulation, and output to output40.dat, for example. (I am basically trying to run a few jobs in parallel in a very elementary way; it appears my computer has plenty of RAM for this).
I am encountering the issue that the > and | tee commands do not seem to work when I use "exec" to run a command on the remote screen, despite having attempted to start a bash shell there; when I use these commands, it just prints to standard output. Although these commands do work with the command "stuff," I do not know how to pass the job number to stuff, as it appears to only work with string inputs.
The current attempted script is as follows. I have replaced the simulation script with echo and > for a simpler example of the problem. Neither of the last two screen lines work.
for i in 1:10; do
screen -Sdm session$i bash
screen -S session$i -X exec echo $i > runnumber$i.output (method 1)
screen -S session$i -X stuff $'echo $i > runnumber$i.output\r' (method 2)
done
Might there be an easy fix?
Is there a way to use the time reserved word in zsh to time multiple commands, without starting a subshell?
I know that this works:
{ time (
sleep 5
sleep 3
PROMPT='foobar> '
) }
However the parentheses mean that a subshell is created, and variables initialized don't get exported.
I know I can capture the variables before and after, like
start=$(time)
# do something
end=$(time)
echo start - end | bc
Though for ad hoc timing this is a little cumbersome.
No, time can only work on a different process. So, it won't work with { ... } or with a builtin, like:
time { ls }
time echo
Note that your method capturing the time output won't work if there are already children (as their times when running the commands will also be taken into account). Ditto if you have traps and corresponding signals occur.
In bash I am able to write a script that contains something like this:
{ time {
#series of commands
echo "something"
echo "another command"
echo "blah blah blah"
} } 2> $LOGFILE
In ZSH the equivalent code does not work and I can not figure out how to make it work for me. This code works but I don't exactly know how to get it to wrap multiple commands.
{ time echo "something" } 2>&1
I know I can create a new script and put the commands in there then time the execution properly, but is there a way to do it either using functions or a similar method to the bash above?
Try the following instead:
{ time ( echo hello ; sleep 10s; echo hola ; ) } 2>&1
If you want to profile your code you have a few alternatives:
Time subshell execution like:
time ( commands ... )
Use REPORTTIME to check for slow commands:
export REPORTTIME=3 # display commands with execution time >= 3 seconds
setop xtrace as explained here
The zprof module
Try replace { with ( ?
I think this should help
You can also use the times POSIX shell builtin in conjunction with functions.
It will report the user and system time used by the shell and its children. See
http://pubs.opengroup.org/onlinepubs/009695399/utilities/times.html
Example:
somefunc() {
code you want to time here
times
}
The reason for using a shell function is that it creates a new shell context, at the start of which times is all zeros (try it). Otherwise the result contains the contribution of the current shell as well. If that is what you want, forget about the function and put times last in your script.
I have a Perl script that launches another Perl script in a new console through Win32::Process as follows:
Win32::Process::Create($ProcessObj,
"C:\\Perl\\bin\\perl.exe",
"$path_to_other_perl_script",
0,
NEW_CONSOLE,
".");
$ProcessObj->Suspend();
$ProcessObj->Resume();
$ProcessObj->Wait(0);
The problem is, there is no stdout in the new console created. If I don't use the new console option, the script runs silently in the background.
If I use cmd.exe to launch the Perl script, I can see the output fine but now I cannot control the child Perl script through Win32::Process.
Does anyone have a solution that works?
Update: Based on your comments, I get the feeling that your programs are not examples of best practices on Linux or Windows. However, I am sure, when reading the documentation for Win32::Process, you noticed that you can call the Kill method on the process to terminate it. So, I changed the example below to do that.
Your chances of getting useful help increase exponentially if you provide real code. Here are the arguments to Win32::Process::Create:
$iflags: flag: inherit calling processes handles or not
Now, I am not sure if you are trying to capture the STDOUT of the second process or if you are trying to have its STDOUT output show up in the same console as the parent.
If the latter, then the following scripts illustrate one way of doing that:
parent.pl
#!/usr/bin/perl
use strict;
use warnings;
use Win32;
use Win32::Process;
$| = 1;
my $p;
print "Starting child process ... \n";
Win32::Process::Create(
$p,
'c:/opt/perl/bin/perl.exe',
'perl hello.pl',
1,
NORMAL_PRIORITY_CLASS,
'.',
) or die Win32::FormatMessage( Win32::GetLastError() );
print "Waiting three seconds before killing 'hello.pl'\n";
for (1 .. 3) {
print;
sleep 1;
}
$p->Kill(0)
or die "Cannot kill '$p'";
hello.pl
#!/usr/bin/perl
$| = 1;
print "Hello World\n";
print "Sleeping 1000 seconds\n";
for (1 .. 1000) {
sleep 1;
print '.';
}
Output:
Starting child process ...
Waiting three seconds before killing 'hello.pl'
1Hello World
Sleeping 1000 seconds
2.3.
Now, I am still not sure why you are using Win32::Process. Unless there is a specific reason to tie your script to Win32, I would recommend using standard Perl facilities. For example, read perldoc -f open and perldoc perlipc (see esp. Using open for IPC).
Explain your question better to get answers that address your particular situation rather than generalities.
Using -w before $filepath fixed it ! Still need an explanation though.