Ok, obviously I am NOT a bash guru and am in need of one!
I have never used 'coproc' before, but it seems to be just what I need. But, I have to admit that I can not extrapolate from the various 'ping' examples out there! [I did try for a couple of hours...]
All I want to do is to start a 'coproc' shell script that can take input from standard in and writes it's results to standard out. I want the main script to do the sending and processing of those commands and results respectively.
Here is one of the simplest outlines of what I am trying to do:
EDITED WITH BETTER DETAIL
#! /bin/bash
coproc bkgndProc {
/some/path/to/usefulScript.sh maybeSomeArgsHere
}
// send command #1 to bkgndProc here
result=$(echo 'command' <&${bkgndProc[0]}) ### Doesn't work for me
echo "Did it work? $result" ### this just prints back the 'command' I used
// here execute conditional logic based on result:
// if result1; then
// send command #2 here, getting results
// else
// send command #3 here, again getting results
// fi
Sorry about using pseudo-code above, but I'm not sure what those sending commands should be! If anyone could supply the details that would be greatly appreciated!
result = $(echo 'command' <&${bkgndProc[0]}) ### Doesn't work for me
wouldn't work at least basically since you have spaces on it
result=$(echo 'command' <&${bkgndProc[0]})
---- Update ----
A simple concept could be shown in a script like this:
#!/bin/bash
# create the co-process
coproc myproc {
bash
}
# send a command to it (echo a)
echo 'echo a' >&"${myproc[1]}"
# read a line from its output
read line <&"${myproc[0]}"
# show the line
echo "$line"
Outputs:
a
Another which reads multiple lines using a timeout:
#!/bin/bash
coproc myproc {
bash
}
# send a command to message 4 random numbers in 4 lines
echo 'echo "$RANDOM"; echo "$RANDOM"; echo "$RANDOM"; echo "$RANDOM"' >&"${myproc[1]}"
# keep reading the line until it times out
while read -t 1 -u "${myproc[0]}" line; do
echo "$line"
done
Output:
17393
1423
8368
1782
If we use cat, it would no longer quit since the other end is still alive and connected, and EOF is not yet reached. It's the reason why we used timeouts.
cat <&"${myproc[0]}"
Related
What I'm attempting to do is receive values from the command line (instead of using the read method and asking the user to enter the values and/or file names in multiple steps).
./hello.sh 5 15 <file_name.txt
I have heard that simply using an array can help do the same, but I am not able to-
Avoid printing
5 15
on the next line
Since 5 and 15 are being printed, I'd expect the string 'abcdefgh' (contents of file_name.txt) to be printed; however, the output stops at
5 15
I would really appreciate it if someone could point out why my code isn't sufficient, and if possible, point me in the direction of some learning resources to broaden my knowledge of this concept.
Here is the code:
#! /usr/bin/bash
echo "$#"
I am simply testing things out (wanted to print out the variables before doing anything with and to them).
<file_name.txt is a redirection. It is not passed as a parameter. The parameters of the script are 5 and 15. The < redirects the file file_name.txt to standard input stdin of the script. You can read from stdin with for example cat.
#!/usr/bin/bash
echo "$#" # outputs parameters of the script joined with spaces
cat # redirects standard input to standard output, i.e. reads from the fiel
why my code isn't sufficient
Your script is not reading from the file, so the content of the file is ignored.
point me in the direction of some learning resources
File descriptors and redirections and standard streams are basic tools in shell - you should learn about them in any shell and linux introduction. My 5 min google search resulted in this link https://www.digitalocean.com/community/tutorials/an-introduction-to-linux-i-o-redirection , which looks like some introduction to the topic.
Will this work?
./hello.sh 5 15 `catfile_name.txt`
And update hello.sh to:
#! /usr/bin/bash
shift 2
echo $#
Here is a more generic solution. It looks at each input parameter in turn. If it is a valid file, it outputs the contents of the file. Otherwise if just prints the parameter.
#! /usr/bin/bash
for $parameter in "${#}"; do # Quotation marks avoid splitting parameters with spaces.
if [ -f $parameter ]; then # '-f {value}' tests if {value} is a file.
cat $parameter
else
echo $parameter # You could also use 'echo -n ...' to skip newlines.
fi
done
I apologize in advance - I don't fully understand the ideas behind what I'm asking well enough to understand why it's not working (I don't know what I need to learn). I searched stack exchange for answers first - I found some information that seemed possibly relevant, but didn't explain the concepts well enough that I understood how to build a working solution. I've been scouring google but haven't found any information that describes exactly what's going on in such a way that I understand. Any direction to background concepts that may help me understand what's going on would be greatly appreciated.
Is it possible to get user input in a bash script that was executed from a pipe?
For example:
wget -q -O - http://myscript.sh | bash
And in the script:
read -p "Do some action (y/n): " __response
if [[ "$__response" =~ ^[Yy]$ ]]; then
echo "Performing some action ..."
fi
As I understand it, this doesn't work because read attempts to read the input from stdin and the bash script is currently "executing through that pipe" (i'm sure there is a more technical accurate way to describe what is occurring, but i don't know how).
I found a solution that recommended using:
read -t 1 __response </dev/tty
However, this does not work either.
Any light shed on the concepts I need to understand to make this work, or explanations of why it is not working or solutions would be greatly appreciated.
The tty solution works. Test it with this code, for example:
$ date | { read -p "Echo date? " r </dev/tty ; [ "$r" = "y" ] && cat || echo OK ; }
Echo date? y
Sat Apr 12 10:51:16 PDT 2014
$ date | { read -p "Echo date? " r </dev/tty ; [ "$r" = "y" ] && cat || echo OK ; }
Echo date? n
OK
The prompt from read appears on the terminal and read waits for a response before deciding to echo the date or not.
What I wrote above differs from the line below in two key aspects:
read -t 1 __response </dev/tty
First, the option -t 1 gives read a timeout of one second. Secondly, this command does not provide a prompt. The combination of these two probably means that, even though read was briefly asking for input, you didn't know it.
The main reason why this is not working is, as the OP validly indicated,
The | <pipe> which is used, sends the standard output from the first command as standard input to the second command. In this case, the first command is
wget -q -O - http://myscript.sh
which passes a downloaded script via the pipe to its interpreter bash
The read statement in the script uses the same standard input to obtain its value.
So this is where it collapses because read is not awaiting input from you but takes it from its own script. Example:
$ cat - <<EOF | bash
> set -x
> read p
> somecommand
> echo \$p
> EOF
+ read p
+ echo somecommand
somecommand
In this example, I used a here-document which is piped to bash. The script enables debugging using set -x to show what is happening. As you see, somecommand is never executed but actually read by read and stored in the variable p which is then outputted by echo (note, the $ has been escaped to avoid the substitution in the here-document).
So how can we get this to work then?
First of, never pipe to an interpreter such as {ba,k,z,c,tc,}sh. It is ugly and should be avoided, even though it feels the natural thing to do. The better thing to do is to use any of its options:
bash -c string: If the -c option is present, then commands are read from string. If there are arguments after the string, they are assigned to the positional parameters, starting with $0.
$ bash -c "$(command you want to pipe)"
This also works for zsh, csh, tcsh, ksh, sh and probably a lot of others.
I have a simple Bash script:
#!/usr/bin/env bash
read X
echo "X=$X"
When I execute it with ./myscript.sh it works. But when I execute it with cat myscript.sh | bash it actually puts echo "X=$X" into $X.
So this script prints Hello World executed with cat myscript.sh | bash:
#!/usr/bin/env bash
read X
hello world
echo "$X"
What's the benefit of executing a script with cat myscript.sh | bash? Why doesn't do it the same things as if I execute it with ./myscript.sh?
How can I avoid Bash to execute line by line but execute all lines after the STDIN reached the end?
Instead of just running
read X
...instead replace it with...
read X </dev/tty || {
X="some default because we can't read from the TTY here"
}
...if you want to read from the console. Of course, this only works if you have a /dev/tty, but if you wanted to do something robust, you wouldn't be piping from curl into a shell. :)
Another alternative, of course, is to pass in your value of X on the command line.
curl https://some.place/with-untrusted-code-only-idiots-will-run-without-reading \
| bash -s "value of X here"
...and refer to "$1" in your script when you want X.
(By the way, I sure hope you're at least using SSL for this, rather than advising people to run code they download over plain HTTP with no out-of-band validation step. Lots of people do it, sure, but that's making sites they download from -- like rvm.io -- big targets. Big, easy-to-man-in-the-middle-or-DNS-hijack targets).
When you cat a script to bash the code to execute is coming from standard input.
Where does read read from? That's right also standard input. This is why you can cat input to programs that take standard input (like sed, awk, etc.).
So you are not running "a script" per-se when you do this. You are running a series of input lines.
Where would you like read to read data from in this setup?
You can manually do that (if you can define such a place). Alternatively you can stop running your script like this.
I apologize in advance - I don't fully understand the ideas behind what I'm asking well enough to understand why it's not working (I don't know what I need to learn). I searched stack exchange for answers first - I found some information that seemed possibly relevant, but didn't explain the concepts well enough that I understood how to build a working solution. I've been scouring google but haven't found any information that describes exactly what's going on in such a way that I understand. Any direction to background concepts that may help me understand what's going on would be greatly appreciated.
Is it possible to get user input in a bash script that was executed from a pipe?
For example:
wget -q -O - http://myscript.sh | bash
And in the script:
read -p "Do some action (y/n): " __response
if [[ "$__response" =~ ^[Yy]$ ]]; then
echo "Performing some action ..."
fi
As I understand it, this doesn't work because read attempts to read the input from stdin and the bash script is currently "executing through that pipe" (i'm sure there is a more technical accurate way to describe what is occurring, but i don't know how).
I found a solution that recommended using:
read -t 1 __response </dev/tty
However, this does not work either.
Any light shed on the concepts I need to understand to make this work, or explanations of why it is not working or solutions would be greatly appreciated.
The tty solution works. Test it with this code, for example:
$ date | { read -p "Echo date? " r </dev/tty ; [ "$r" = "y" ] && cat || echo OK ; }
Echo date? y
Sat Apr 12 10:51:16 PDT 2014
$ date | { read -p "Echo date? " r </dev/tty ; [ "$r" = "y" ] && cat || echo OK ; }
Echo date? n
OK
The prompt from read appears on the terminal and read waits for a response before deciding to echo the date or not.
What I wrote above differs from the line below in two key aspects:
read -t 1 __response </dev/tty
First, the option -t 1 gives read a timeout of one second. Secondly, this command does not provide a prompt. The combination of these two probably means that, even though read was briefly asking for input, you didn't know it.
The main reason why this is not working is, as the OP validly indicated,
The | <pipe> which is used, sends the standard output from the first command as standard input to the second command. In this case, the first command is
wget -q -O - http://myscript.sh
which passes a downloaded script via the pipe to its interpreter bash
The read statement in the script uses the same standard input to obtain its value.
So this is where it collapses because read is not awaiting input from you but takes it from its own script. Example:
$ cat - <<EOF | bash
> set -x
> read p
> somecommand
> echo \$p
> EOF
+ read p
+ echo somecommand
somecommand
In this example, I used a here-document which is piped to bash. The script enables debugging using set -x to show what is happening. As you see, somecommand is never executed but actually read by read and stored in the variable p which is then outputted by echo (note, the $ has been escaped to avoid the substitution in the here-document).
So how can we get this to work then?
First of, never pipe to an interpreter such as {ba,k,z,c,tc,}sh. It is ugly and should be avoided, even though it feels the natural thing to do. The better thing to do is to use any of its options:
bash -c string: If the -c option is present, then commands are read from string. If there are arguments after the string, they are assigned to the positional parameters, starting with $0.
$ bash -c "$(command you want to pipe)"
This also works for zsh, csh, tcsh, ksh, sh and probably a lot of others.
I'm currently using the following to capture everything that goes to the terminal and throw it into a log file
exec 4<&1 5<&2 1>&2>&>(tee -a $LOG_FILE)
however, I don't want color escape codes/clutter going into the log file. so i have something like this that sorta works
exec 4<&1 5<&2 1>&2>&>(
while read -u 0; do
#to terminal
echo "$REPLY"
#to log file (color removed)
echo "$REPLY" | sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' >> $LOG_FILE
done
unset REPLY #tidy
)
except read waits for carriage return which isn't ideal for some portions of the script (e.g. echo -n "..." or printf without \n).
Follow-up to Jonathan Leffler's answer:
Given the example script test.sh:
#!/bin/bash
LOG_FILE="./test.log"
echo -n >$LOG_FILE
exec 4<&1 5<&2 1>&2>&>(tee -a >(sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' > $LOG_FILE))
##### ##### #####
# Main
echo "starting execution"
printf "\n\n"
echo "color test:"
echo -e "\033[0;31mhello \033[0;32mworld\033[0m!"
printf "\n\n"
echo -e "\033[0;36mEnvironment:\033[0m\n foo: cat\n bar: dog\n your wife: hot\n fix: A/C"
echo -n "Before we get started. Is the above information correct? "
read YES
echo -e "\n[READ] $YES" >> $LOG_FILE
YES=$(echo "$YES" | sed 's/^\s*//;s/\s*$//')
test ! "$(echo "$YES" | grep -iE '^y(es)?$')" && echo -e "\nExiting... :(" && exit
printf "\n\n"
#...some hundreds of lines of code later...
echo "Done!"
##### ##### #####
# End
exec 1<&4 4>&- 2<&5 5>&-
echo "Log File: $LOG_FILE"
The output to the terminal is as expected and there is no color escape codes/clutter in the log file as desired. However upon examining test.log, I do not see the [READ] ... (see line 21 of test.sh).
The log file [of my actual bash script] contains the line Log File: ... at the end of it even after closing the 4 and 5 fds. I was able to resolve the issue by putting a sleep 1 before the second exec - I assume there's a race condition or fd shenanigans to blame for it. Unfortunately for you guys, I am not able to reproduce this issue with test.sh but I'd be interested in any speculation anyone may have.
Consider using the pee program discussed in Is it possible to distribute stdin over parallel processes. It would allow you to send the log data through your sed script, while continuing to send the colours to the actual output.
One major advantage of this is that it would remove the 'execute sed once per line of log output'; that is really diabolical for performance (in terms of number of processes executed, if nothing else).
I know it's not a perfect solution, but cat -v will make non visible chars like \x1B to be converted into visible form like ^[[1;34m. The output will be messy, but it will be ascii text at least.
I use to do stuff like this by setting TERM=dumb before running my command. That pretty much removed any control characters except for tab, CR, and LF. I have no idea if this works for your situation, but it's worth a try. The problem is that you won't see color encodings on your terminal either since it's a dumb terminal.
You can also try either vis or cat (especially the -v parameter) and see if these do something for you. You'd simply put them in your pipeline like this:
exec 4<&1 5<&2 1>&2>&>(tee -a | cat -v | $LOG_FILE)
By the way, almost all terminal programs have an option to capture the input, and most clean it up for you. What platform are you on, and what type of terminal program are you using?
You could attempt to use the -n option for read. It reads in n characters instead of waiting for a new line. You could set it to one. This would increase the number of iteration the code runs, but it would not wait for newlines.
From the man:
-n NCHARS read returns after reading NCHARS characters rather than waiting for a complete line of input.
Note: I have not tested this
You can use ANSIFilter to strip or transform console output with ANSI escape sequences.
See http://www.andre-simon.de/zip/download.html#ansifilter
Might not screen -L or the script commands be viable options instead of this exec loop?