Using sed command on remote system - shell

I am using the below command on the local machine and it gives me the expected result:
sed -n 's/^fname\(.*\)".*/\1/p' file.txt
When I use the same command(only changed ' to ") to a same file present in the remote system, I do not get any output.
ssh remote-system "sed -n "s/^fname\(.*\)".*/\1/p" file.txt"
Please help me to get this corrected. Thanks for your help.

" and ' are different things in bash, and they are not interchangeable (they're not interchangeable in many languages, however the differences are more subtle) The single quote means 'pretend everything inside here is a string'. The only thing that will be interpreted is the next single quote.
The double quote allows bash to interpret stuff inside
For example,
echo "$TERM"
and
echo '$TERM'
return different things.
(Untested) you should be able to use single quotes and escape the internal single quotes :
ssh remote-system 'sed -n \'s/^fname(.)"./\1/p\' file.txt'
Looks like you can send a single quote with the sequence '"'"' (from this question)
so :
ssh remote-machine 'sed -n '"'"'s/^fname\(.*\)".*/\1/p'"'"' file.txt'
This runs on my machine if I ssh into localhost, there's no output because file.txt is empty, but it's a proof-of-concept.
Or - can you do the ssh session interactively/with a heredoc?
ssh remote-system
[sed command]
exit
or (again untested, look up heredocs for more info)
ssh remote-system <<-EOF
[sed command]
EOF

Related

How can I pass the filename from a variable locally into ssh? [duplicate]

When I stumble across an evil web site that I want blocked from corporate access, I edit my named.conf file on my bind server and then update my proxy server blacklist file. I'd like to automate this somewhat with a bash script. Say my script is called "evil-site-block.sh" and contains the following:
ssh root#192.168.0.1 'echo "#date added $(date +%m/%d/%Y)" >> /var/named/chroot/etc/named.conf; echo "zone \"$1\" { type master; file \"/etc/zone/dummy-block\"; };" >> /var/named/chroot/etc/named.conf'
It is then run as
$ evil-site-block.sh google.com
When I look at the contents of named.conf on the remote machine I see:
#date added 09/16/2014
zone "" { type master; file "/etc/zone/dummy-block"; };
What I can't figure out is how to pass "google.com" as $1.
First off, you don't want this to be two separately redirected echo statements -- doing that is both inefficient and means that the lines could end up not next to each other if something else is appending at the same time.
Second, and much more importantly, you don't want the remote command that's run to be something that could escape its quotes and run arbitrary commands on your server (think of if $1 is '$(rm -rf /)'.spammer.com).
Instead, consider:
#!/bin/bash
# ^ above is mandatory, since we use features not found in #!/bin/sh
printf -v new_contents \
'# date added %s\nzone "%s" { type master; file "/etc/zone/dummy-block"; };\n' \
"$(date +%m/%d/%Y)" \
"$1"
printf -v remote_command \
'echo %q >>/var/named/chroot/etc/named.conf' \
"$new_contents"
ssh root#192.168.0.1 bash <<<"$remote_command"
printf %q escapes data such that an evaluation pass in another bash shell will evaluate that content back to itself. Thus, the remote shell will be guaranteed (so long as it's bash) to interpret the content correctly, even if the content attempts to escape its surrounding quotes.
Your problem: Your entire command is put into single quotes – obviously so that bash expressions are expanded on the server and not locally.
But this also applies to your $1.
Simple solution: “Interupt” the quotation by wrapping your local variable into single quotes.
ssh root#192.168.0.1 'echo "#date added $(date +%m/%d/%Y)" >> /var/named/chroot/etc/named.conf; echo "zone \"'$1'\" { type master; file \"/etc/zone/dummy-block\"; };" >> /var/named/chroot/etc/named.conf'
NB: \"$1\" → \"'$1'\".
NOTE: This solution is a simple fix for the one-liner as posted in the question above. If there's the slightest chance that this script is executed by other people, or it could process external output of any kind, please have a look at Charles Duffy's solution.

echo $(command) gets a different result with the output of the command

The Bash command I used:
$ ssh user#myserver.com ps -aux|grep -v \"grep\"|grep "/srv/adih/server/app.js"|awk '{print $2}'
6373
$ ssh user#myserver.com echo $(ps -aux|grep -v \"grep\"|grep "/srv/adih/server/app.js"|awk '{print $2}')
8630
The first result is the correct one and the second one will change echo time I execute it. But I don't know why they are different.
What am I doing?
My workstation has very limited resources, so I use a remote machine to run my Node.js application. I run it using ssh user#remotebox.com "cd /application && grunt serve" in debug mode. When I command Ctrl + C, the grunt task is stopped, but the application is is still running in debug mode. I just want to kill it, and I need to get the PID first.
The command substitution is executed by your local shell before ssh runs.
If your local system's name is here and the remote is there,
ssh there uname -n
will print there whereas
ssh there echo $(uname -n) # should have proper quoting, too
will run uname -n locally and then send the expanded command line echo here to there to be executed.
As an additional aside, echo $(command) is a useless use of echo unless you specifically require the shell to perform wildcard expansion and whitespace tokenization on the output of command before printing it.
Also, grep x | awk { y } is a useless use of grep; it can and probably should be refactored to awk '/x/ { y }' -- but of course, here you are reinventing pidof so better just use that.
ssh user#myserver.com pidof /srv/adih/server/app.js
If you want to capture the printed PID locally, the syntax for that is
pid=$(ssh user#myserver.com pidof /srv/adih/server/app.js)
Of course, if you only need to kill it, that's
ssh user#myserver.com pkill /srv/adih/server/app.js
Short answer: the $(ps ... ) command substitution is being run on the local computer, and then its output is sent (along with the echo command) to the remote computer. Essentially, it's running ssh user#myserver.com echo 8630.
Your first command is also probably not doing what you expect; the pipes are interpreted on the local computer, so it's running ssh user#myserver.com ps -aux, piping the output to grep on the local computer, piping that to another grep on the local computer, etc. I'm guessing that you wanted that whole thing to run on the remote computer so that the result could be used on the remote computer to kill a process.
Long answer: the order things are parsed and executed in shell is a bit confusing; with an ssh command in the mix, things get even more complicated. Basically, what happens is that the local shell parses the command line, including splitting it into separate commands (separated by pipes, ;, etc), and expanding $(command) and $variable substitutions (unless they're in single-quotes). It then removes the quotes and escapes (they've done their jobs) and passes the results as arguments to the various commands (such as ssh). ssh takes its arguments, sticks all the ones that look like parts of the remote command together with spaces between them, and sends them to a shell on the remote computer which does this process over again.
This means that quoting and/or escaping things like $ and | is necessary if you want them to be parsed/acted on by the remote shell rather than the local shell. And quotes don't nest, so putting quotes around the whole thing may not work the way you expect (e.g. if you're not careful, the $2 in that awk command might get expanded on the local computer, even though it looks like it's in single-quotes).
When things get messy like this, the easiest way is sometimes to pass the remote command as a here-document rather than as arguments to the ssh command. But you want quotes around the here-document delimiter to keep the various $ expansions from being done by the local shell. Something like this:
ssh user#myserver.com <<'EOF'
echo $(ps -aux|grep -v "grep"|grep "/srv/adih/server/app.js"|awk '{print $2}')
EOF
Note: be careful with indenting the remote command, since the text will be sent literally to the remote computer. If you indent it with tab characters, you can use <<- as the here-document delimiter (e.g. <<-'EOF') and it'll remove the leading tabs.
EDIT: As #tripleee pointed out, there's no need for the multiple greps, since awk can do the whole job itself. It's also unnecessary to exclude the search commands from the results (grep -v grep) because the "/" characters in the pattern need to be escaped, meaning that it won't match itself.. So you can simplify the pipeline to:
ps -aux | awk '/\/srv\/adih\/server\/app.js/ {print $2}'
Now, I've been assuming that the actual goal is to kill the relevant pid, and echo is just there for testing. If that's the case, the actual command winds up being:
ssh user#myserver.com <<'EOF'
kill $(ps -aux | awk '/\/srv\/adih\/server\/app.js/ {print $2}')
EOF
If that's not right, then the whole echo $( ) thing is best skipped entirely. There's no reason to capture the pipeline's output and then echo it, just run it and let it output directly.
And if pkill (or something similar) is available, it's much simpler to use that instead.

Remote ssh command, LF missing on STDOUT

This is my first question here :)
I'm running into an issue while running a remote SSH command. The issue is that the output of the command comes back in a string without LF to delimit the lines. I checked the output of Hive by itself and it outputs the LFs normally, they are missing when I run the command from a remote host. Is there any way to maintain the output of the remote host?
Here's a snippet of the code:
#!/bin/bash
partition="2013-04-02"
query=$(cat <<EOF
set mapred.job.name="Unique IP - $partition";
select
distinct
src
,dst
,service
,proto
,action
from
logs
where
src rlike "^10\."
and src not rlike "^10\.90\."
and src not rlike "^10[0-9]\."
and src not rlike "\.5[2-8]$"
and dst <> ""
and day="$partition";
EOF)
ssh=`ssh user#hadoop "hive -S -e '$query' 2>&1"
echo $ssh > output.tsv
It's got the newline(s) in there, but you're not seeing them because you don't have double-quotes around the parameter expansion in your echo command.
echo "$ssh" >output.tsv
ETA: The $(...) syntax for command substitution is generally much better than the `...` syntax, as it nests and works sanely with nested quotation marks. And while it doesn't matter in this case, it's good to get in the habit of quoting command substitutions as well:
ssh="$(ssh user#hadoop "hive -S -e '$query' 2>&1")"
Note that the double-quotes inside the $(..) don't cause any problems with the outer quotation marks.
Finally, if all you're doing with this value is echoing it into a file, you can skip the variable and just redirect it there in the first place:
ssh user#hadoop "hive -S -e '$query' 2>&1" >output.tsv
or just
ssh user#hadoop "hive -S -e '$query' " >&output.tsv
The backticks around ssh … are converting your newlines to spaces because they are supposed to.
Try
ssh user#hadoop "hive -S -e '$query' 2>&1" > output.tsv

gnome terminal tabs open multiple ssh connections

I have a file with a list of servers:
SERVERS.TXT:
192.168.0.100
192.168.0.101
192.168.0.102
From a gnome terminal script, I want open a new terminal, with a tab for each server.
Here is what I tried:
gnome-terminal --profile=TabProfile `while read SERVER ; do echo "--tab -e 'ssh usr#$SERVER'"; done < SERVERS.TXT`
Here is the error:
Failed to parse arguments: Argument to "--command/-e" is not a valid command: Text ended before matching quote was found for '. (The text was ''ssh')
Tried removing the space after the -e
gnome-terminal --profile=TabProfile `while read SERVER ; do echo "--tab -e'ssh usr#$SERVER'"; done < SERVERS.TXT`
And I get a similar error:
Failed to parse arguments: Argument to "--command/-e" is not a valid command: Text ended before matching quote was found for '. (The text was 'usr#192.168.0.100'')
Obviously there is a parsing error since the the shell is trying to be helpful by using the spaces to predict and place delimiters. The server file is changed without notice and many different sets of servers need to be looked at.
I found this question while searching for an answer to the issue the OP had, but my issue was a little different. I knew the list of servers, they where not in a file.
Anyway, the other solutions posted did not work for me, but the following script does work, and is what I use to get around the "--command/-e" is not a valid command" error.
The script should be very easy change to suit any need:
#!/bin/sh
# Open a terminal to each of the servers
#
# The list of servers
LIST="server1.info server2.info server3.info server4.info"
cmdssh=`which ssh`
for s in $LIST
do
title=`echo -n "${s}" | sed 's/^\(.\)/\U\1/'`
args="${args} --tab --title=\"$title\" --command=\"${cmdssh} ${s}.com\""
done
tmpfile=`mktemp`
echo "gnome-terminal${args}" > $tmpfile
chmod 744 $tmpfile
. $tmpfile
rm $tmpfile
Now the big question is why does this work when run from a file, but not from within a script. Sure, the issue is about the escaping of the --command part, but everything I tried failed unless exported to a temp file.
I would try something like:
$ while read SERVER;do echo -n "--tab -e 'ssh usr#$SERVER' "; \
done < SERVERS.txt | xargs gnome-terminal --profile=TabProfile
This is to avoid any interpretation that the shell could do of the parameters (anything starting with a dash).
Because it is concatenating strings (using -n), it is necessary to add an space between them.
Is this a problem of parsing command-line options? Sometimes if you have one command sending arguments to another command, the first can get confused. The convention is to use a -- like so:
echo -- "--tab -e 'ssh usr#$SERVER'";
Try to type
eval
before gnome terminal command.
it should be something like this:
eval /usr/bin/gnome-terminal $xargs
worked for me!

How to remotely edit a line containing double quotes and variables using sed across ssh?

I have a sed command that works perfectly if I run it locally on Ubuntu or our embedded client Arago:
sed -i 's/export PART="$1"/export PART="A"/' flash.sh
This results in exactly what I need on both versions of sed, a line that changes:
export PART="$1"
to
export PART="A"
My problem is, I need to run that same command across a network to the embedded client so I tried this on an Ubuntu server using bash, to an Arago client using sh:
ssh -n -o stricthostkeychecking=no root#10.14.150.113 sed -i 's/PART="$1"/PART="A"/' flash.sh
Which results in a line that contains:
export PART=A"$1"
The substitution command needs to stay inside the single quotes so the double quotes are passed as literals, or maybe there's a better way to keep the double quotes in the two strings? This looks to me like the $1 is simply being ignored like it's empty and the "A" is being passed as A and replacing the end of PART=. I've tried encapsulating the command in single and double quotes, which both result in the same thing. I've also tried escaping the quotes with backslashes, same result. I think this is something with quote expansion with sh that I simply don't understand. Or possibly something I don't understand with ssh.
I've read through a number of other threads that are similar, but none of them dealt with using ssh to run the command remotely. I'm no longer a novice at sed, but this one has become quite the puzzle for me.
ssh -n -o stricthostkeychecking=no root#10.14.150.113 'sed -i "s/export PART=\"\\\$1\"/export PART=\"A\"/" flash.sh'
You can switch back and forth to get the quoting you need. try this
ssh -n -o stricthostkeychecking=no root#10.14.150.113 sed -i 's/PART='"$1"'/PART="A"/' flash.sh
# -------------------------------------------------------------------^^^^^^
OR if you really need the value of $1 surrounded by dbl-quotes,
ssh -n -o stricthostkeychecking=no root#10.14.150.113 sed -i 's/PART='\"$1\"'/PART="A"/' flash.sh
# -------------------------------------------------------------------^^^^^^^^
You do want $1 to come from your local machine (not the ssh'd to machine), right?
I hope this helps.
P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, and/or give it a + (or -) as a useful answer.

Resources