Okay, so I've recently discovered the magic of here documents for feeding stdin style lines into interactive commands. However, I'm trying to use this with SSH to execute a bunch of commands on a remote server, but I also need to pipe in some actual input, before executing the extra commands, to confound matters further I also need to get some results back ;)
Here's what I'm trying to use:
#!/bin/sh
RESULT=$(find -type f "$PATH" | gzip | ssh "$HOST" <<- 'REMOTE_SYNC'
cat > "/tmp/.temp_file"
# Do something with /tmp/.temp_file
REMOTE_SYNC
Is this actually correct? Part of the problem I'm having as well is that I need to pipe the data to that file in /tmp, but I should really be generating a randomly named temp file, but I'm not sure how I could do that, assign the name to a variable (so I can get back to it) and still send stdin into it.
I may also extract the find | gzip part to a separate command run locally first, as the gzipped file will likely be small enough that sending it when ready will result in a much shorter SSH connection then sending it as it's generated, but it still doesn't get around the fact that I need to be able to provide both stdin and my extra commands to SSH.
No, you can't do it like this. Both heredoc and the piped input compete for stdin, and only one wins. Look at this example:
echo test | cat << EOF
TEST
EOF
What will this print? test, TEST or both? It prints TEST, so the heredoc wins (at least in bash).
You don't really need this anyway. Luckily ssh takes a command argument, which will be passed on to the shell on the remote host, so you can just use your command as a string here. So something like this:
echo TEST | ssh user#host 'cat > tempfile; cat tempfile; rm tempfile'
would work (althoug it doesn't make much sense), the output of the left side commands is piped through ssh to the remote host and supplied as stdin there.
If you want the data to be compressed when sending it through ssh, you can just enable compression using the -C option.
edit:
Using linebreaks inside a string is perfectly fine, so this works fine too:
echo TEST | ssh user#host '
cat > tempfile
cat tempfile
rm tempfile
'
The only difference to a heredoc would be that you have to escape quotes.
If you use something like echo TEST | ssh user#host "$(<script.sh)" you can write everything into a file...
Related
I have a simple Bash script:
#!/usr/bin/env bash
read X
echo "X=$X"
When I execute it with ./myscript.sh it works. But when I execute it with cat myscript.sh | bash it actually puts echo "X=$X" into $X.
So this script prints Hello World executed with cat myscript.sh | bash:
#!/usr/bin/env bash
read X
hello world
echo "$X"
What's the benefit of executing a script with cat myscript.sh | bash? Why doesn't do it the same things as if I execute it with ./myscript.sh?
How can I avoid Bash to execute line by line but execute all lines after the STDIN reached the end?
Instead of just running
read X
...instead replace it with...
read X </dev/tty || {
X="some default because we can't read from the TTY here"
}
...if you want to read from the console. Of course, this only works if you have a /dev/tty, but if you wanted to do something robust, you wouldn't be piping from curl into a shell. :)
Another alternative, of course, is to pass in your value of X on the command line.
curl https://some.place/with-untrusted-code-only-idiots-will-run-without-reading \
| bash -s "value of X here"
...and refer to "$1" in your script when you want X.
(By the way, I sure hope you're at least using SSL for this, rather than advising people to run code they download over plain HTTP with no out-of-band validation step. Lots of people do it, sure, but that's making sites they download from -- like rvm.io -- big targets. Big, easy-to-man-in-the-middle-or-DNS-hijack targets).
When you cat a script to bash the code to execute is coming from standard input.
Where does read read from? That's right also standard input. This is why you can cat input to programs that take standard input (like sed, awk, etc.).
So you are not running "a script" per-se when you do this. You are running a series of input lines.
Where would you like read to read data from in this setup?
You can manually do that (if you can define such a place). Alternatively you can stop running your script like this.
The Bash command I used:
$ ssh user#myserver.com ps -aux|grep -v \"grep\"|grep "/srv/adih/server/app.js"|awk '{print $2}'
6373
$ ssh user#myserver.com echo $(ps -aux|grep -v \"grep\"|grep "/srv/adih/server/app.js"|awk '{print $2}')
8630
The first result is the correct one and the second one will change echo time I execute it. But I don't know why they are different.
What am I doing?
My workstation has very limited resources, so I use a remote machine to run my Node.js application. I run it using ssh user#remotebox.com "cd /application && grunt serve" in debug mode. When I command Ctrl + C, the grunt task is stopped, but the application is is still running in debug mode. I just want to kill it, and I need to get the PID first.
The command substitution is executed by your local shell before ssh runs.
If your local system's name is here and the remote is there,
ssh there uname -n
will print there whereas
ssh there echo $(uname -n) # should have proper quoting, too
will run uname -n locally and then send the expanded command line echo here to there to be executed.
As an additional aside, echo $(command) is a useless use of echo unless you specifically require the shell to perform wildcard expansion and whitespace tokenization on the output of command before printing it.
Also, grep x | awk { y } is a useless use of grep; it can and probably should be refactored to awk '/x/ { y }' -- but of course, here you are reinventing pidof so better just use that.
ssh user#myserver.com pidof /srv/adih/server/app.js
If you want to capture the printed PID locally, the syntax for that is
pid=$(ssh user#myserver.com pidof /srv/adih/server/app.js)
Of course, if you only need to kill it, that's
ssh user#myserver.com pkill /srv/adih/server/app.js
Short answer: the $(ps ... ) command substitution is being run on the local computer, and then its output is sent (along with the echo command) to the remote computer. Essentially, it's running ssh user#myserver.com echo 8630.
Your first command is also probably not doing what you expect; the pipes are interpreted on the local computer, so it's running ssh user#myserver.com ps -aux, piping the output to grep on the local computer, piping that to another grep on the local computer, etc. I'm guessing that you wanted that whole thing to run on the remote computer so that the result could be used on the remote computer to kill a process.
Long answer: the order things are parsed and executed in shell is a bit confusing; with an ssh command in the mix, things get even more complicated. Basically, what happens is that the local shell parses the command line, including splitting it into separate commands (separated by pipes, ;, etc), and expanding $(command) and $variable substitutions (unless they're in single-quotes). It then removes the quotes and escapes (they've done their jobs) and passes the results as arguments to the various commands (such as ssh). ssh takes its arguments, sticks all the ones that look like parts of the remote command together with spaces between them, and sends them to a shell on the remote computer which does this process over again.
This means that quoting and/or escaping things like $ and | is necessary if you want them to be parsed/acted on by the remote shell rather than the local shell. And quotes don't nest, so putting quotes around the whole thing may not work the way you expect (e.g. if you're not careful, the $2 in that awk command might get expanded on the local computer, even though it looks like it's in single-quotes).
When things get messy like this, the easiest way is sometimes to pass the remote command as a here-document rather than as arguments to the ssh command. But you want quotes around the here-document delimiter to keep the various $ expansions from being done by the local shell. Something like this:
ssh user#myserver.com <<'EOF'
echo $(ps -aux|grep -v "grep"|grep "/srv/adih/server/app.js"|awk '{print $2}')
EOF
Note: be careful with indenting the remote command, since the text will be sent literally to the remote computer. If you indent it with tab characters, you can use <<- as the here-document delimiter (e.g. <<-'EOF') and it'll remove the leading tabs.
EDIT: As #tripleee pointed out, there's no need for the multiple greps, since awk can do the whole job itself. It's also unnecessary to exclude the search commands from the results (grep -v grep) because the "/" characters in the pattern need to be escaped, meaning that it won't match itself.. So you can simplify the pipeline to:
ps -aux | awk '/\/srv\/adih\/server\/app.js/ {print $2}'
Now, I've been assuming that the actual goal is to kill the relevant pid, and echo is just there for testing. If that's the case, the actual command winds up being:
ssh user#myserver.com <<'EOF'
kill $(ps -aux | awk '/\/srv\/adih\/server\/app.js/ {print $2}')
EOF
If that's not right, then the whole echo $( ) thing is best skipped entirely. There's no reason to capture the pipeline's output and then echo it, just run it and let it output directly.
And if pkill (or something similar) is available, it's much simpler to use that instead.
I have a simple Bash script:
#!/usr/bin/env bash
read X
echo "X=$X"
When I execute it with ./myscript.sh it works. But when I execute it with cat myscript.sh | bash it actually puts echo "X=$X" into $X.
So this script prints Hello World executed with cat myscript.sh | bash:
#!/usr/bin/env bash
read X
hello world
echo "$X"
What's the benefit of executing a script with cat myscript.sh | bash? Why doesn't do it the same things as if I execute it with ./myscript.sh?
How can I avoid Bash to execute line by line but execute all lines after the STDIN reached the end?
Instead of just running
read X
...instead replace it with...
read X </dev/tty || {
X="some default because we can't read from the TTY here"
}
...if you want to read from the console. Of course, this only works if you have a /dev/tty, but if you wanted to do something robust, you wouldn't be piping from curl into a shell. :)
Another alternative, of course, is to pass in your value of X on the command line.
curl https://some.place/with-untrusted-code-only-idiots-will-run-without-reading \
| bash -s "value of X here"
...and refer to "$1" in your script when you want X.
(By the way, I sure hope you're at least using SSL for this, rather than advising people to run code they download over plain HTTP with no out-of-band validation step. Lots of people do it, sure, but that's making sites they download from -- like rvm.io -- big targets. Big, easy-to-man-in-the-middle-or-DNS-hijack targets).
When you cat a script to bash the code to execute is coming from standard input.
Where does read read from? That's right also standard input. This is why you can cat input to programs that take standard input (like sed, awk, etc.).
So you are not running "a script" per-se when you do this. You are running a series of input lines.
Where would you like read to read data from in this setup?
You can manually do that (if you can define such a place). Alternatively you can stop running your script like this.
I need to "cat" a text inside shell script to remote machine via ssh.
This is required for simplicity, so i don't need to keep extra file.
For instance
#!/bin/sh
VAR="some"
VAR1="something"
cat << EOF
apple
green
tree
EOF ---> cat to file.txt on remote machine
do some command
do some command1
exit 0
Try something like this if you're generating file content in runtime:
cat <<EOF | ssh remote 'cat - > /tmp/my_remote_file.txt'
apple
green
tree
EOF
Or simply use scp if file is static.
Just want to note for everyone. Above example works fine for plain text, but if heredoc includes variables then substitution will take place. So important to protect heredoc phrase with single quotes to keep everything intact
For example
cat <<'EOF' | ssh remote 'cat - > /tmp/my_remote_file.txt'
host=$(uname -a)
echo $host
EOF
host variable won't be substituted with actual hostname.
As reference (paragraph 19-7) http://tldp.org/LDP/abs/html/here-docs.html
I have a file with a list of servers:
SERVERS.TXT:
192.168.0.100
192.168.0.101
192.168.0.102
From a gnome terminal script, I want open a new terminal, with a tab for each server.
Here is what I tried:
gnome-terminal --profile=TabProfile `while read SERVER ; do echo "--tab -e 'ssh usr#$SERVER'"; done < SERVERS.TXT`
Here is the error:
Failed to parse arguments: Argument to "--command/-e" is not a valid command: Text ended before matching quote was found for '. (The text was ''ssh')
Tried removing the space after the -e
gnome-terminal --profile=TabProfile `while read SERVER ; do echo "--tab -e'ssh usr#$SERVER'"; done < SERVERS.TXT`
And I get a similar error:
Failed to parse arguments: Argument to "--command/-e" is not a valid command: Text ended before matching quote was found for '. (The text was 'usr#192.168.0.100'')
Obviously there is a parsing error since the the shell is trying to be helpful by using the spaces to predict and place delimiters. The server file is changed without notice and many different sets of servers need to be looked at.
I found this question while searching for an answer to the issue the OP had, but my issue was a little different. I knew the list of servers, they where not in a file.
Anyway, the other solutions posted did not work for me, but the following script does work, and is what I use to get around the "--command/-e" is not a valid command" error.
The script should be very easy change to suit any need:
#!/bin/sh
# Open a terminal to each of the servers
#
# The list of servers
LIST="server1.info server2.info server3.info server4.info"
cmdssh=`which ssh`
for s in $LIST
do
title=`echo -n "${s}" | sed 's/^\(.\)/\U\1/'`
args="${args} --tab --title=\"$title\" --command=\"${cmdssh} ${s}.com\""
done
tmpfile=`mktemp`
echo "gnome-terminal${args}" > $tmpfile
chmod 744 $tmpfile
. $tmpfile
rm $tmpfile
Now the big question is why does this work when run from a file, but not from within a script. Sure, the issue is about the escaping of the --command part, but everything I tried failed unless exported to a temp file.
I would try something like:
$ while read SERVER;do echo -n "--tab -e 'ssh usr#$SERVER' "; \
done < SERVERS.txt | xargs gnome-terminal --profile=TabProfile
This is to avoid any interpretation that the shell could do of the parameters (anything starting with a dash).
Because it is concatenating strings (using -n), it is necessary to add an space between them.
Is this a problem of parsing command-line options? Sometimes if you have one command sending arguments to another command, the first can get confused. The convention is to use a -- like so:
echo -- "--tab -e 'ssh usr#$SERVER'";
Try to type
eval
before gnome terminal command.
it should be something like this:
eval /usr/bin/gnome-terminal $xargs
worked for me!