I have this oprint script:
#!/usr/bin/env ruby
amount = 100
index = 0
loop do
index += 1
if index % 5 == 0
amount += 10
end
sleep 0.1
$stdout.puts amount
end
If I run oprint | echo, then I don't see anything. If I comment out the sleep 0.1 inside oprint, then I see a lot of output. Does sleep break the pipe? Is there a fix?
oprint | echo really shouldn't work, because echo doesn't read from the input stream. It echos its arguments. If you want to test a simple pipe, oprint | cat would be more appropriate.
Even then, you should add $stdout.flush after the puts when you have an infinite loop like that. Since lots of small IO calls can be a performance bottleneck, Ruby buffers its output by default - meaning it stores up lots of little output in a buffer and then writes the whole buffer all at once. Flushing the buffer manually ensures that it won't end up waiting forever to do the actual write.
Making those two changes gives me the expected output.
Related
I'd like to see a run counter at the top of my watch output.
Ex: this command should print a count value which increments every 2 seconds, in addition to the output of my main command, which, in this case is just echo "hello" for the purposes of this demonstration:
export COUNT=0 && watch -n 2 'export COUNT=$((COUNT+1)); echo "Count = $COUNT" \
&& echo "hello"'
But, all it outputs is this, with the count always being 1 and never changing:
Count = 1
hello
How can I get this Count variable to increment every 2 seconds when watch runs the command?
Thanks #Inian for pointing this out in the comments. I've consulted the cross-site duplicate, slightly modified it, and come up with this:
count=0; while sleep 2 ; do clear; echo "$((count++))"; echo "hello" ; done
Replace echo "hello" with the real command you want to run once every sleep n second now, and it works perfectly.
There's no need to use watch at all, and, as a matter of fact, no way to use watch in this way either unless you write the variable to a file rather than to RAM, which I'd like to avoid since that's unnecessary wear and tear write/erase cycles on a solid state drive (SSD).
Now I can run this command to repeatedly build my asciidoctor document so I can view it in a webpage, hitting only F5 each time I make and save a change to view the updated HTML page.
count=0; while sleep 2 ; do clear; echo "$((count++))"; \
asciidoctor -D temp README.adoc ; done
Sample output:
96
asciidoctor: ERROR: README.adoc: line 6: level 0 sections can only be used when doctype is book
Final answer:
And, here's a slightly better version which prints count = 2 instead of just 2, and which also runs first and then sleeps, rather than sleeping first and then running:
count=1; while true ; do clear; echo "count = $((count++))"; \
asciidoctor -D temp README.adoc; sleep 2 ; done
...but, it looks like it's not just updating the file if it's changed, it's rewriting constantly, wearing down my disk anyway. So, I'd need to write a script and only run this if the source file has changed. Oh well...for the purposes of my original question, this answer is done.
Example in Julia: let's say I have a file "test.jl" with the following code:
for i in 1:3
sleep(2)
print("i = ", i, "\n")
end
Then if I run
nohup julia test.jl &
I won't get each print result every 2 seconds. Instead, I will get all three print results at the end, which is useless for monitoring the progress of a loop that takes forever to complete.
Any suggestions?
It's likely that output buffering is occurring. Try manually flushing standard output:
for i in 1:3
sleep(2)
print("i = ", i, "\n")
flush(stdout)
end
nohup redirects standard output to $HOME/nohup.out until the process is completed then the results are printed onto screen all at once.
https://linux.die.net/man/1/nohup
you can still redirect it live I think, something like tee might work or >&1. Not sure would have to check.
I have to run unix code on a remote system that does not have MATLAB. I call it like this
% shared key between computers so no password required to type in
ssh_cmd = 'ssh login#ipaddress ';
for x = 1:nfiles
cmd = sprintf('calcPosition filename%d',x);
% as an example, basically it runs a c++ command
% on that computer and I pass in a different filename as an input
full_cmd = [ssh_cmd, cmd];
[Status,Message] = system(full_cmd);
% the stdout is a mix of strings and numbers
% <code that parses Message and handles errors>
end
For a small test this takes about 30 seconds to run. If I set it so that
full_cmd = [ssh_cmd, cmd1; cmd2; cmdN]; % obviously not valid syntax here
% do the same thing as above, but no loop
it takes about 5 seconds, so most of the time is spent connecting to the other system. But Message is the combined stdout of all n files.
I know I can pipe the outputs to separate files, but I'd rather pass it back directly to MATLAB without addition I/O. So is there a way to get the stdout (Message) as a cell array (or Unix equivalent) for each separate command? Then I could just loop over the cells of Message and the remainder of the function would not need to be changed.
I thought about using an identifier text around each command call as a way to help parse Message, e.g.
full_cmd = [ssh_cmd; 'echo "begincommand " '; cmd1; 'echo "begincommand " '; cmd2]; % etc
But there must be something more elegant than that. Any tips?
i have a basic script.sh that runs some commands inside. The script is like:
(script.sh)
......
`gcc -o program program.c`
if [ $? -eq 0 ]; then
echo "Compiled successfully....\n" >> out.txt
#set a timeout for ./program execution and append results to file
(gtimeout 10s ./program) 2> out.txt # <-- NOT WORKING
......
I run this script through the terminal like:
#Go to this directory,pass all folders to compile&execute the program.c file
czar#MBP~$ for D in /Users/czar/Desktop/1/*; do sh script.sh $D; done
EDIT: The output i get in the terminal, not so important though:
# program.c from 1st folder inside the above path
Cycle l=1: 46 46
Cycle l=1: 48 48
Cycle l=2: 250 274 250
Cycle l=1: 896 896
.........
# program.c from 2nd folder inside the above path
Cycle l=1: 46 46
Cycle l=1: 48 48
Cycle l=2: 250 274 250
Cycle l=1: 896 896
.........
The GOAL is to have those into the out.txt
The output i get is almost what i want: it executes whatever possible in those 10seconds but doesn't redirect the result to out.txt, it just prints to the terminal.
I have tried every suggestion proposed here but no luck.
Any other ideas appreciated.
EDIT 2: SOLUTION given in the comments.
The basic approach is much simpler than the command you copied from the answer to a completely different question. What you need to do is simply redirect standard output to your file:
# Use gtimeout on systems which rename standard Gnu utilities
timeout 10s ./program >> out.txt
However, that will probably not produce all of the output generated by the program if the program is killed by gtimeout, because the output is still sitting in a buffer inside the standard library. (There is nothing special about this buffer; it's just a block of memory malloc'd by the library functions the first time data is written to the stream.) When the program is terminated, its memory is returned to the operating system; nothing will even try to ensure that standard library buffers are flushed to their respective streams.
There are three buffering modes:
Block buffered: no output is produced until the stream's buffer is full. (Usually, the stream's buffer will be around 8kb, but it varies from system to system.)
Line buffered: output is produced when a newline character is sent to the stream. It's also produced if the buffer fills up, but it's rare for a single line to be long enough to fill a buffer.
Unbuffered: No buffering is performed at all. Every character is immediately sent to the output.
Normally, standard output is block buffered unless it is directed to a terminal, in which case it will be line buffered. (That's not guaranteed; the various standards allow quite a lot of latitude.) Line buffering is probably what you want, unless you're in the habit of writing programs which write partial lines. (The oddly-common idiom of putting a newline at the beginning of each output line rather than at the end is a really bad idea, precisely because it defeats line-buffering.) Unbuffered output is another possibility, but it's really slow if the program produces a substantial amount of output.
You can change the buffering mode before you write any data to the stream by calling setvbuf:
/* Line buffer stdout */
setvbuf(stdout, NULL, _IOLBF, 0);
(See man setvbuf for more options.)
You can also tell the library to immediately send any buffered data by calling fflush:
fflush(stdout);
That's an effective technique if you don't want the (slight) overhead of line buffering, but you know when it is important to send data (typically, because the program is about to do some very long computation, or wait for some external event).
If you can't modify the source code, you can use the Gnu utility stdbuf to change the buffering mode before starting the program. stdbuf will not work with all programs -- for example, it won't have any effect if the program does call setvbuf -- but it is usually effective. For example, to line buffer stdout, you could do this:
timeout 10s stdbuf -oL ./program >> out.txt
# Or: gtimeout 10s gstdbuf -oL ./program >> out.txt
See man stdbuf for more information.
I would like to write a script to execute the steps outlined below. If someone can provide simple examples on how to modify files and search through folders using a script (not necessarily solving my problem below), I will greatly appreciate it.
submit job MyJob in currentDirectory using myJobShellFile.sh to a queue
upon completion of MyJob, goto to currentDirectory/myJobDataFolder.
In myJobDataFolder, there are folders
myJobData.0000 myJobData.0001 myJobData.0002 myJobData.0003
I want to find the maximum number maxIteration of all the listed folders. Here it would be maxIteration=0003.\
In file myJobShellFile.sh, at the last line says
mpiexec ./main input myJobDataFolder
I want to append this line to
'mpiexec ./main input myJobDataFolder 0003'
I want to submit MyJob to the que while maxIteration < 10
Upon completion of MyJob, find the new maxIteration and change this number in myJobShellFile.sh and goto step 4.
I think people write python scripts typically to do this stuff, but am having a hard time finding out how. I probably don't know the correct terminology for this procedure. I am also aware that the script will vary slightly depending on the queing system, but any help will be greatly appreciated.
Quite a few aspects of your question are unclear, such as the meaning of “submit job MyJob in currentDirectory using myJobShellFile.sh to a que”, “append this line to
'mpiexec ./main input myJobDataFolder 0003'”, how you detect when a job is done, relevant parts of myJobShellFile.sh, and some other details. If you can list the specific shell commands you use in each iteration of job submission, then you can post a better question, with a bash tag instead of python.
In the following script, I put a ### at the end of any line where I am guessing what you are talking about. Lines ending with ### may be irrelevant to whatever you actually do, or may be pseudocode. Anyway, the general idea is that the script is supposed to do the things you listed in your items 1 to 5. This script assumes that you have modified myJobShellFile.sh to say
mpiexec ./main input $1 $2
instead of
mpiexec ./main input
because it is simpler to use parameters to modify what you tell mpiexec than it is to keep modifying a shell script. Also, it seems to me you would want to increment maxIter before submitting next job, instead of after. If so, remove the # from the t=$((1$maxIter+1)); maxIter=${t#1} line. Note, see the “Parameter Expansion” section of man bash re expansion of the ${var#txt} form, and the “Arithmetic Expansion” section re $((expression)) form. The 1$maxIter and similar forms are used to change text like 0018 (which is not a valid bash number because 8 is not an octal digit) to 10018.
#!/bin/sh
./myJobShellFile.sh MyJob ###
maxIter=0
while true; do
waitforjobcompletion ###
cd ./myJobDataFolder
maxFile= $(ls myJobData* | tail -1)
maxIter= ${maxFile#myJobData.} #Get max extension
# If you want to increment maxIter, uncomment next line
# t=$((1$maxIter+1)); maxIter=${t#1}
cd ..
if [[ 1$maxIter -lt 11000 ]] ; then
./myJobShellFile.sh MyJobDataFolder $maxIter
else
break
fi
done
Notes: (1) To test with smaller runs than 1000 submissions, replace 11000 by 10000+n; for example, to do 123 runs, replace it with 10123. (2) In writing the above script, I assumed that not-previously-known numbers of output files appear in the output directory from time to time. If instead exactly one output file appears per run, and you just want to do one run per value for the values 0000, 0001, 0002, 0999, 1000, then use a script like the following. (For testing with a smaller number than 1000, replace 1000 with (eg) 0020. The leading zeroes in these numbers tell bash to fill the generated numbers with leading zeroes.)
#!/bin/sh
for iter in {0000..1000}; do
./myJobShellFile.sh MyJobDataFolder $iter
waitforjobcompletion ###
done
(3) If the system has a command that sleeps while it waits for a job to complete on the supercomputing resource, it is reasonable to use that command in place of waitforjobcompletion in the above scripts. Otherwise, if the system has a command jobisrunning that returns true if a job is still running, replace waitforjobcompletion with something like the following:
while jobisrunning ; do sleep 15; done
This will run the jobisrunning command; if it returns true, the shell will sleep for 15 seconds and then retest. Here is an example that illustrates waiting for a file to appear and then for it to go away:
while [ ! -f abc ]; do sleep 3; echo no abc; done
while ls abc >/dev/null 2>&1; do sleep 3; echo an abc; done
The second line's test could be [ -f abc ] instead; I showed a longer example to illustrate how to suppress output and error messages by routing them to /dev/null. (4) To reverse the sense of a while statement's test, replace the word while with until. For example, while [ ! -f abc ]; ... is equivalent to until [ -f abc ]; ....