i have a basic script.sh that runs some commands inside. The script is like:
(script.sh)
......
`gcc -o program program.c`
if [ $? -eq 0 ]; then
echo "Compiled successfully....\n" >> out.txt
#set a timeout for ./program execution and append results to file
(gtimeout 10s ./program) 2> out.txt # <-- NOT WORKING
......
I run this script through the terminal like:
#Go to this directory,pass all folders to compile&execute the program.c file
czar#MBP~$ for D in /Users/czar/Desktop/1/*; do sh script.sh $D; done
EDIT: The output i get in the terminal, not so important though:
# program.c from 1st folder inside the above path
Cycle l=1: 46 46
Cycle l=1: 48 48
Cycle l=2: 250 274 250
Cycle l=1: 896 896
.........
# program.c from 2nd folder inside the above path
Cycle l=1: 46 46
Cycle l=1: 48 48
Cycle l=2: 250 274 250
Cycle l=1: 896 896
.........
The GOAL is to have those into the out.txt
The output i get is almost what i want: it executes whatever possible in those 10seconds but doesn't redirect the result to out.txt, it just prints to the terminal.
I have tried every suggestion proposed here but no luck.
Any other ideas appreciated.
EDIT 2: SOLUTION given in the comments.
The basic approach is much simpler than the command you copied from the answer to a completely different question. What you need to do is simply redirect standard output to your file:
# Use gtimeout on systems which rename standard Gnu utilities
timeout 10s ./program >> out.txt
However, that will probably not produce all of the output generated by the program if the program is killed by gtimeout, because the output is still sitting in a buffer inside the standard library. (There is nothing special about this buffer; it's just a block of memory malloc'd by the library functions the first time data is written to the stream.) When the program is terminated, its memory is returned to the operating system; nothing will even try to ensure that standard library buffers are flushed to their respective streams.
There are three buffering modes:
Block buffered: no output is produced until the stream's buffer is full. (Usually, the stream's buffer will be around 8kb, but it varies from system to system.)
Line buffered: output is produced when a newline character is sent to the stream. It's also produced if the buffer fills up, but it's rare for a single line to be long enough to fill a buffer.
Unbuffered: No buffering is performed at all. Every character is immediately sent to the output.
Normally, standard output is block buffered unless it is directed to a terminal, in which case it will be line buffered. (That's not guaranteed; the various standards allow quite a lot of latitude.) Line buffering is probably what you want, unless you're in the habit of writing programs which write partial lines. (The oddly-common idiom of putting a newline at the beginning of each output line rather than at the end is a really bad idea, precisely because it defeats line-buffering.) Unbuffered output is another possibility, but it's really slow if the program produces a substantial amount of output.
You can change the buffering mode before you write any data to the stream by calling setvbuf:
/* Line buffer stdout */
setvbuf(stdout, NULL, _IOLBF, 0);
(See man setvbuf for more options.)
You can also tell the library to immediately send any buffered data by calling fflush:
fflush(stdout);
That's an effective technique if you don't want the (slight) overhead of line buffering, but you know when it is important to send data (typically, because the program is about to do some very long computation, or wait for some external event).
If you can't modify the source code, you can use the Gnu utility stdbuf to change the buffering mode before starting the program. stdbuf will not work with all programs -- for example, it won't have any effect if the program does call setvbuf -- but it is usually effective. For example, to line buffer stdout, you could do this:
timeout 10s stdbuf -oL ./program >> out.txt
# Or: gtimeout 10s gstdbuf -oL ./program >> out.txt
See man stdbuf for more information.
Related
I generate (dynamically) a script concatenating the following files:
testscript1
echo Writing File
cat > /tmp/test_file <<EOF
testcontent
line1
second line
testscript2
EOF
echo File is written
And I execute by calling
$ cat testscript1 testcontent testscript2 | ssh remote_host bash -s --
The effect is that the file /tmp/test_file is filled with the desired content.
Is there also a variant thinkable where binary files can be supplied in a similar fashion? Instead of cat of course dd could be used or other Tools, but the problem I see is 'telling' them that the STDIN now ended (can I send ^D through that stream?)
I am not able to get my head around that problem, but there is likely no comparable solution. However, I might be wrong, so I'd be happy to hear from you.
Regards,
Mazze
can I send ^D through that stream
Yes but you don't want to.
Control+D, commonly notated ^D, is just a character -- or to be pedantic (as I often am), a codepoint in the usual character code (ASCII or a superset like UTF-8) that we treat as a character. You can send that character/byte by a number of methods, most simply printf '\004', but the receiving system won't treat it as end-of-file; it will instead be stored in the destination file, just like any other data byte, followed by the subsequent data that you meant to be a new command and file etc.
^D only causes end-of-file when input from a terminal (more exactly, a 'tty' device) -- and then only in 'cooked' mode (which is why programs like vi and less can do things very different from ending a file when you type ^D). The form of ssh you used doesn't make the input a 'tty' device. ssh can make the input (and output) a 'tty' (more exactly a subclass of 'tty' called a pseudo-tty or 'pty', but that doesn't matter here) if you add the -t option (in some situations you may need to repeat it as -t -t or -tt). But then if your binary file contains any byte with the value \004 -- or several other special values -- which is quite possible, then your data will be corrupted and garbage commands executed (sometimes), which definitely won't do what you want and may damage your system.
The traditional approach to what you are trying to do, back in the 1980s and 1990s, was 'shar' (shell archive) and the usual solution to handling binary data was 'uuencode', which converts binary data into only printable characters that could go safely go through a link like this, matched by 'uudecode' which converts it back. See this surviving example from GNU. uuencode and uudecode themselves were part of a communication protocol 'uucp' used mostly for email and Usenet, which are (all) mostly obsolete and forgotten.
However, nearly all systems today contain a 'base64' program which provides equivalent (though not identical) functionality. Within a single system you can do:
base64 <infile | base64 -d >outfile
to get the same effect as cp infile outfile. In your case you can do something like:
{ echo "base64 -d <<END# >outfile"; base64 <infile; echo "END#"; otherstuff; } | ssh remote bash
You can also try:
cat testscript1 testcontent testscript2 | base64 | ssh <options> "base64 --decode | bash"
Don't worry about ^D, because when your input is exhausted, the next processes of the pipeline will notice that they have reached the end of the input file.
I am running a program (pianobar) piped to a text file, that outputs every second. The resulting file ("pianobarout.txt") needs to be cleared regularly, or it grows to massive proportions. However, I do not want to stop pianobar to clear the file.
I have tried running > pianobarout.txt as well as echo "" > pianobarout.txt, but both cause the system's resources to spike heavily for almost 30 seconds, causing the audio from pianobar to skip. I tried removing the file, but it appears that the file is not recreated after being deleted, and I just lose the pipe.
I'm working from python, so if any library there can help, those are available to me.
Any ideas?
If you are currently redirecting with truncation, like yourprogram > file.txt, try redirecting with appending: yourprogram >> file.txt.
There is a big difference between the two when the output file is truncated.
With appending redirection, data is written to the current end of the file. If you truncate it to 0 bytes, the next write will happen at position 0.
With truncating redirection, data is written wherever the last write left off in the file. If you truncate it to 0 bytes, writes will continue at byte 1073741824 where it last left off.
This results in a sparse file if the filesystem supports it (ext2-4 and most Unix fs do), or a long wait while the file is written out if it doesn't (like fat32). A long wait could also be caused by anything following the file, such as tail -f, which has to potentially catch up by reading a GB of zeroes.
Alternatives include yourprogram | split -b 1G - output-, which will write 1GB each to output-aa, output-ab, etc, letting you delete old files at your leasure.
I have a very long file with numbers. Something like output of this perl program:
perl -le 'print int(rand() * 1000000) for 1..10'
but way longer - around hundreds of gigabytes.
I need to split this file into many others. For test purposes, let's assume that 100 files, and output file number is taken by taking module of number with 100.
With normal files, I can do it simply with:
perl -le 'print int(rand() * 1000000) for 1..1000' | awk '{z=$1%100; print > z}'
But I have a problem when I need to compress splitted parts. Normally, I could:
... | awk '{z=$1%100; print | "gzip -c - > "z".txt.gz"}'
But, when ulimit is configured to allow less open files than number of "partitions", awk breaks with:
awk: (FILENAME=- FNR=30) fatal: can't open pipe `gzip -c - > 60.txt.gz' for output (Too many open files)
This doesn't break with normal file output, as GNU awk is apparently smart enough to recycle file handles.
Do you know any way (aside from writing my own stream-splitting-program, implementing buffering, and some sort of pool-of-filehandles management) to handle such case - that is: splitting to multiple files, where access to output files is random, and gzipping all output partitions on the fly?
I didn't write it in question itself, but since the additional information is together with solution, I'll write it all here.
So - the problem was on Solaris. Apparently there is a limitation, that no program using stdio on Solaris can have more than 256 open filehandles ?!
It is described in here in detail. The important point is that it's enough to set one env variable before running my problematic program, and the problem is gone:
export LD_PRELOAD_32=/usr/lib/extendedFILE.so.1
I have a problem when using Arduino to post data to Pachube. The Arduino is configured to return JSON data for the temperature when you send a 't' and return JSON data for the light level when you send an 'l'. This works perfectly through the Arduino Serial Monitor. I then created two bash scripts. One regularly sends the 't' and 'l' commands to Arduino and waits 10 seconds in between each request.
while true; do
echo -n t > /dev/ttyACM0
echo "$(date): Queried Arduino for temperature."
sleep 10
echo -n l > /dev/ttyACM0
echo "$(date): Queried Arduino for light."
sleep 10
done
This works fine. I get an echo message every 10 seconds. The other script reads the generated JSON from serial port (I basically copied it from some Web page).
ARDUINO_PORT=/dev/ttyACM0
ARDUINO_SPEED=9600
API_KEY='MY_PACHUBE_KEY'
FEED_ID='MY_FEED_ID'
# Set speed for usb
stty -F $ARDUINO_PORT ispeed $ARDUINO_SPEED ospeed $ARDUINO_SPEED raw
exec 6<$ARDUINO_PORT
# Read data from Arduino
while read -u 6 f ;do
# Remove trailing carriage return character added
# by println to satisfy stupid MS-DOS Computers
f=${f:0:${#f} - 1}
curl --request PUT --header "X-PachubeApiKey: $API_KEY" --data-binary "{ \"version\":\"1.0.0\", \"datastreams\":[ $f ] }" "http://api.pachube.com/v2/feeds/MY_FEED_ID"
echo "$(date) $f was read."
done
Unfortunately, this script goes crazy with echo messages telling me several times per 10 seconds that it posted data to Pachube although it should only do it every 10 seconds (whenever the first script told Arduino to create a JSON message). I thought it might be an issue with buffered messages on the Arduino but even when switching it off and on again the problem remains. Any thoughts? Thanks in advance.
I am completely unfamiliar with Arduino and a handful of other things you're doing here but here are a few general things I see:
Bash is almost entirely incapable of handling binary data reliably. There is no way to store a NUL byte in a Bash string. Looks like you're trying to pull some trickery to make arbitrary data readable - hopefully you're sending nothing but character data into read, otherwise this isn't likely going to work.
read reads newline-delimited input (or the given value of -d if your bash is new enough). I don't know the format the while loop is reading, but it has to be a newline delimited string of characters.
Use read -r unless you want escape sequences interpreted. (You almost always want -r with read.)
Unconditionally stripping a character off the end of each string isn't the greatest. I'd use: f=${f%+($'\r')}, which removes 1 or more adjacent \r's from the end of f. Remember to shopt -s extglob at the top of your script if this isn't the default.
This shouldn't be actually causing an issue, but I prefer not using exec unless it's really required - which it isn't here. Just put done <$ARDUINO_PORT to terminate the while loop and remove the -u 6 argument from read (unless something inside the loop is specifically reading from stdin and can't conflict, which doesn't appear to be the case). The open FD will automatically close when exiting the loop.
Don't create your own all-caps variable names in scripts because they are reserved and can conflict with variables from the environment. Use at least one lower-case letter. This of course doesn't apply if those variables are set by something in your system and you're only using or modifying them.
I am using the open4 gem and having problems reading from the spawned processes stdout. I have a ruby program, test1.rb:
print 'hi.' # 3 characters
$stdin.read(1) # block
And another ruby program in the same directory, test2.rb:
require 'open4'
pid, stdin, stdout, stderr = Open4.popen4 'ruby test1.rb'
p stdout.read(2) # 2 characters
When I run the second program:
$ ruby test2.rb
It just sits there forever without printing anything. Why does this happen, and what can I do to stop it?
I needed to change test1.rb to this. I don't know why.
print 'hi.' # 3 characters
$stdout.flush
$stdin.read(1) # block
By default, everything that you printto stdout or to another file is written into a buffer of Ruby (or the standard C library, which is underneath Ruby). The content of the buffer is forwarded to the OS if one of the following events occurs:
The buffer gets full.
You close stdout.
You have printed a newline sequence (`\n')
You call flush explicitly.
For other files, a flush is done on other occasions, too, like ftell.
If you put stdout in unbuffered mode ($stdout.sync = true), the buffer will not be used.
stderr is unbuffered by default.
The reason for doing buffering is efficiency: Aggregating output data in a buffer can save many system call (calls to operating system). System calls are very expensive: They take many hundreds or even thousands of CPU cycles. Avoiding them with a little bit of code and some buffers in user space results in a good speedup.
A good reading on buffering: Why does printf not flush after the call unless a newline is in the format string?
I'm not an expert in process.
From my first sight of API document, the sequence of using open4 is like this:
first send text to stdin, then close stdin and lastly read text from stdout.
So. You can the test2.rb like this
require 'open4'
pid, stdin, stdout, stderr = Open4.popen4 'ruby test1.rb'
stdin.puts "something" # This line is important
stdin.close # It might be optional, open4 might close itself.
p stdout.read(2) # 2 characters