i have a problem when attempt run multi command in remote linux, with Perl and Module Net::OpenSSH. .
use Net::OpenSSH;
my $new_connect = Net::OpenSSH->new($D_CONNECT{'HOST'}, %OPT);
my $file = "file.log.gz"
my ($output2, $error2) = $new_connect->capture({ timeout => 10 }, "gunzip -c /path/to/file/$file | tail -n1 | awk '/successfully completed/ {print \$NF}'");
the output that i get is:
bash: -c: line 1: syntax error near unexpected token |'
bash: -c: line 1: |tail -n1 |awk '/successfully completed/ {print $NF}''
;;;
any idea or suggestion, thanks.
Fcs
Probably a quoting error. Just let Net::OpenSSH take care of the quoting for you:
my ($output2, $error2) = $new_connect->capture({ timeout => 10 },
'gunzip', '-c', "/path/to/file/$file", \\'|',
'tail', '-n1', \\'|',
'awk', '/successfully completed/ {print $NF}');
Note how the pipes (|) are passed as a double reference so that they are passed unquoted to the remote shell. The module documentation has a section on quoting.
That looks like the error message you'd get if you had a newline at the end of your $file string, causing the pipe character to be at the start of a second line (interpreted as the start of a second command).
This test deomstrates the same error:
bash -c 'echo foo
| cat'
So I guess your bug doesn't really occur with $file = "file.log.gz" and whatever your real $file is, you need to chomp it.
A bigger mystery is why bash says the error is on line 1 of the -c. ash, ksh, and zsh all correctly report it on line 2.
Related
I am trying to create a script to find a JSON element and update it with the arg values.
#!/bin/bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
sed -i '' '/location/c\ \"location\" : \"$placeLocation\",' $file
sed -i '' '/heading/c\ \"heading\" : \"$vehicleHeading\",' $file
sed -i '' '/message/c\ \"message\" : \"$message\",' $file
One.txt
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
But getting following error
sed: 1: "/location/c\ \"locati ...": extra characters after \ at the end of c command
sed: 1: "/heading/c\ \"heading ...": extra characters after \ at the end of c command
sed: 1: "/message/c\ \"message ...": extra characters after \ at the end of c command
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
I have just started learning about sed editor and tried out multiple things but couldn't able to figure it out. Any help is much appreciated!
Note that you probably should consider using a tool like jq for editing JSON files. But I assume you have a good reason for using sed, so you have a couple of problems there.
The first is that you're trying to use GNU sed features on your Mac OS X version of sed that doesn't have those features. If you want GNU sed on Mac OS X, then install it:
▶ brew install gnu-sed
Fixing up your code for GNU sed (and also for other Bash style guide recommendations about quoting strings):
cat > FILE <<EOF
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
EOF
placeLocation=myPlaceLocation
vehicleHeading=myVehicleHeading
message=myMessage
file=FILE
gsed -i -e '/location/c\' -e '"location": "'"$placeLocation"'",' "$file"
gsed -i -e '/heading/c\' -e '"heading": "'"$vehicleHeading"'",' "$file"
gsed -i -e '/message/c\' -e '"message": "'"$message"'",' "$file"
As noted in the GNU sed manual, use of multiple -e commands on the same line with the c\ command is a GNU extension.
If you want to use Mac OS X's sed, just may be able to write it this way:
sed -i '' '
s/"location".*/"location": "'"$placeLocation"'",/
s/"heading".*/"heading": "'"$vehicleHeading"'",/
s/"message".*/"message": "'"$message"'",/
' "$file"
But note you would have to sanitise the input if you need the code to be robust to all inputs.
To do this robustly you need to use a tool that understands literal strings (which sed doesn't - see Is it possible to escape regex metacharacters reliably with sed) e.g. awk:
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1/2?' one.txt
"message":"what is 1/2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1&2?' one.txt
"message":"what is 1&2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is \1?' one.txt
"message":"what is \1?"
the above will work robustly using any awk in any shell on every UNIX box. Try using those as replacement strings in a sed command.
The full script to do what you want would be:
#!/bin/env bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
tmp=$(mktemp)
awk '
BEGIN {
split("location heading message", tags)
for (i in tags) {
vals["\"" tags[i] "\""] = "\"" ARGV[i] "\""
ARGV[i] = ""
}
FS=OFS=":"
}
$1 in vals {
tag = $1
sub(FS".*","")
$0 = tag OFS vals[tag]
}
1' "$placeLocation" "$vehicleHeading" "$message" "$file" > "$tmp" && mv "$tmp" "$file"
For every time the pattern shows up (In this example the case of a 2 digit number) I want to pass that pattern to a script and replace that pattern with the output of a script.
I'm using sed an example of what it should look like would be
echo 'siedi87sik65owk55dkd' | sed 's/[0-9][0-9]/.\/script.sh/g'
Right now this returns
siedi./script.shsik./script.showk./script.shdkd
But I would like it to return
siedi!!!87!!!sik!!!65!!!owk!!!55!!!dkd
This is what is in ./script.sh
#!/bin/bash
echo "!!!$1!!!"
It has to be replaced with the output. In this example I know I could just use a normal sed substitution but I don't want that as an answer.
sed is for simple substitutions on individual lines, that is all. Anything else, even if it can be done, requires arcane language constructs that became obsolete in the mid-1970s when awk was invented and are used today purely for the mental exercise. Your problem is not a simple substitution so you shouldn't try to use sed to solve it.
You're going to want something like:
awk '{
head = ""
tail = $0
while ( match(tail,/[0-9]{2}/) ) {
tgt = substr(tail,RSTART,RLENGTH)
cmd = "./script.sh " tgt
if ( (cmd | getline line) > 0) {
tgt = line
}
close(cmd)
head = head substr(tail,1,RSTART-1) tgt
tail = substr(tail,RSTART+RLENGTH)
}
print head tail
}'
e.g. using an echo in place of your script.sh command:
$ echo 'siedi87sik65owk55dkd' |
awk '{
head = ""
tail = $0
while ( match(tail,/[0-9]{2}/) ) {
tgt = substr(tail,RSTART,RLENGTH)
cmd = "echo !!!" tgt "!!!"
if ( (cmd | getline line) > 0) {
tgt = line
}
close(cmd)
head = head substr(tail,1,RSTART-1) tgt
tail = substr(tail,RSTART+RLENGTH)
}
print head tail
}'
siedi!!!87!!!sik!!!65!!!owk!!!55!!!dkd
Ed's awk solution is obviously the way to go here.
For fun, I tried to come up with a sed solution, and here is (a convoluted GNU sed) one that takes the pattern and the script to be run as parameters; the input is either read from standard input (i.e., you can pipe to it) or from a file supplied as the third argument.
For your example, we'd have infile with contents
siedi87sik65owk55dkd
siedi11sik22owk33dkd
(two lines to demonstrate how this works for multiple lines), then script with contents
#!/bin/bash
echo "!!!${1}!!!"
and finally the solution script itself, so. Usage is
./so pattern script [input]
where pattern is an extended regular expression as understood by GNU sed (with the -r option), script is the name of the command you want to run for each match, and the optional input is the name of the input file if input is not standard input.
For your example, this would be
./so '[[:digit:]]{2}' script infile
or, as a filter,
cat infile | ./so '[[:digit:]]{2}' script
with output
siedi!!!87!!!sik!!!65!!!owk!!!55!!!dkd
siedi!!!11!!!sik!!!22!!!owk!!!33!!!dkd
This is what so looks like:
#!/bin/bash
pat=$1 # The pattern to match
script=$2 # The command to run for each pattern
infile=${3:-/dev/stdin} # Read from standard input if not supplied
# Use sed and have $pattern and $script expand to the supplied parameters
sed -r "
:build_loop # Label to loop back to
h # Copy pattern space to hold space
s/.*($pat).*/.\/\"$script\" \1/ # (1) Extract last match and prepare command
# Replace pattern space with output of command
e
G # (2) Append hold space to pattern space
s/(.*)$pat(.*)/\1~~~\2/ # (3) Replace last match of pattern with ~~~
/\n[^\n]*$pat[^\n]*$/b build_loop # Loop if string contains match
:fill_loop # Label for second loop
s/(.*\n)(.*)\n([^\n]*)~~~([^\n]*)$/\1\3\2\4/ # (4) Replace last ~~~
t fill_loop # Loop if there was a replacement
s/(.*)\n(.*)~~~(.*)$/\2\1\3/ # (5) Final ~~~ replacement
" < "$infile"
The sed command works with two loops. The first one copies the pattern space to the hold space, then removes everything but the last match from the pattern space and prepares the command to be run. After the substitution with (1) in its comment, the pattern space looks like this:
./script 55
The e command (a GNU extension) then replaces the pattern space with the output of this command. After this, G appends the hold space to the pattern space (2). The pattern space now looks like this:
!!!55!!!
siedi87sik65owk55dkd
The substitution at (3) replaces the last match with a string hopefully not equal to the pattern and we get
!!!55!!!
siedi87sik65owk~~~dkd
The loop repeats if the last line of the pattern space still has a match for the pattern. After three loops, the pattern space looks like this:
!!!87!!!
!!!65!!!
!!!55!!!
siedi~~~sik~~~owk~~~dkd
The second loop now replaces the last ~~~ with the second to last line of the pattern space with substitution (4). The command uses lots of "not a newline" ([^\n]) to make sure we're not pulling the wrong replacement for ~~~.
Because of the way command (4) is written, the loop ends with one last substitution to go, so before command (5), we have this pattern space:
!!!87!!!
siedi~~~sik!!!65!!!owk!!!55!!!dkd
Command (5) is a simpler version of command (4), and after it, the output is as desired.
This seems to be fairly robust and can deal with spaces in the name of the script to be run as long as it's properly quoted when calling:
./so '[[:digit:]]{2}' 'my script' infile
This would fail if
The input file contains ~~~ (solvable by replacing all occurrences at the start, putting them back at the end)
The output of script contains ~~~
The pattern contains ~~~
i.e., the solution very much depends on ~~~ being unique.
Because nobody asked: so as a one-liner.
#!/bin/bash
sed -re ":b;h;s/.*($1).*/.\/\"$2\" \1/;e" -e "G;s/(.*)$1(.*)/\1~~~\2/;/\n[^\n]*$1[^\n]*$/bb;:f;s/(.*\n)(.*)\n([^\n]*)~~~([^\n]*)$/\1\3\2\4/;tf;s/(.*)\n(.*)~~~(.*)$/\2\1\3/" < "${3:-/dev/stdin}"
Still works!
A conceptually simpler multi-utility solution:
Using GNU utilities:
echo 'siedi87sik65owk55dkd' |
sed 's|[0-9]\{2\}|$(./script.sh &)|g' |
xargs -d'\n' -I% sh -c 'echo '\"%\"
Using BSD utilities (also works with GNU utilities):
echo 'siedi87sik65owk55dkd' |
sed 's|[0-9]\{2\}|$(./script.sh &)|g' | tr '\n' '\0' |
xargs -0 -I% sh -c 'echo '\"%\"
The idea is to use sed to translate the tokens of interest lexically into a string containing shell command substitutions that invoke the target script with the token, and then pass the result to the shell for evaluation.
Note:
Any embedded " and $ characters in the input must be \-escaped.
xargs -d'\n' (GNU) and tr '\n' '\0' / xargs -0 (BSD) are only needed to correctly preserve whitespace in the input - if that is not needed, the following POSIX-compliant solution will do:
echo 'siedi87sik65owk55dkd' |
sed 's|[0-9]\{2\}|$(./script.sh &)|g' | tr '\n' '\0' |
xargs -I% sh -c 'printf "%s\n" '\"%\"
I have a shell script to automate builds of my programm. I need to transform versioning numbers like V4_5_1-RC1 to 4.5.1-RC1. The V should be removed and the _ should be replaced with .. I tried several things, for example with sed:
$NAMEEXT = "V4_5_1-RC1"
$lffNameRSC = ${sed -e "s/V//g" <<< $NAMEEXT}
$lffNameRSC = ${sed -e "s/_/./g" <<< $lffNameRSC}
echo $lffNameRSC
but I'm getting errors.
./makerelease.sh: line 113: ${sed -e "s/V//g" <<< $NAMEEXT}: bad substitution
./makerelease.sh: line 114: ${sed -e "s/_/./g" <<< $lffNameRSC}: bad substitution
there should be no spaces around =
there should be $(..) instead of ${..} to evaluate the command
there should not be $ in variable assignment statement
With
#!/bin/bash
NAMEEXT="V4_5_1-RC1"
lffNameRSC=$(sed -e "s/V//g" <<< $NAMEEXT)
lffNameRSC=$(sed -e "s/_/./g" <<< $lffNameRSC)
echo $lffNameRSC
you will get
4.5.1-RC1
And, by the way, it could be done easier, like
$> echo "V4_5_1-RC1" | sed "s/V//g; s/_/./g"
4.5.1-RC1
Bash parameter expansion can do what you want without any external tools:
NAMEEXT="V4_5_1-RC1"
version=${NAMEEXT#V} # remove the leading V
version=${version//_/.} # replace all _ with .
echo $version # ==> 4.5.1-RC1
How to I concatenate stdin to a string, like this?
echo "input" | COMMAND "string"
and get
inputstring
A bit hacky, but this might be the shortest way to do what you asked in the question (use a pipe to accept stdout from echo "input" as stdin to another process / command:
echo "input" | awk '{print $1"string"}'
Output:
inputstring
What task are you exactly trying to accomplish? More context can get you more direction on a better solution.
Update - responding to comment:
#NoamRoss
The more idiomatic way of doing what you want is then:
echo 'http://dx.doi.org/'"$(pbpaste)"
The $(...) syntax is called command substitution. In short, it executes the commands enclosed in a new subshell, and substitutes the its stdout output to where the $(...) was invoked in the parent shell. So you would get, in effect:
echo 'http://dx.doi.org/'"rsif.2012.0125"
use cat - to read from stdin, and put it in $() to throw away the trailing newline
echo input | COMMAND "$(cat -)string"
However why don't you drop the pipe and grab the output of the left side in a command substitution:
COMMAND "$(echo input)string"
I'm often using pipes, so this tends to be an easy way to prefix and suffix stdin:
echo -n "my standard in" | cat <(echo -n "prefix... ") - <(echo " ...suffix")
prefix... my standard in ...suffix
There are some ways of accomplish this, i personally think the best is:
echo input | while read line; do echo $line string; done
Another can be by substituting "$" (end of line character) with "string" in a sed command:
echo input | sed "s/$/ string/g"
Why i prefer the former? Because it concatenates a string to stdin instantly, for example with the following command:
(echo input_one ;sleep 5; echo input_two ) | while read line; do echo $line string; done
you get immediatly the first output:
input_one string
and then after 5 seconds you get the other echo:
input_two string
On the other hand using "sed" first it performs all the content of the parenthesis and then it gives it to "sed", so the command
(echo input_one ;sleep 5; echo input_two ) | sed "s/$/ string/g"
will output both the lines
input_one string
input_two string
after 5 seconds.
This can be very useful in cases you are performing calls to functions which takes a long time to complete and want to be continuously updated about the output of the function.
You can do it with sed:
seq 5 | sed '$a\6'
seq 5 | sed '$ s/.*/\0 6/'
In your example:
echo input | sed 's/.*/\0string/'
I know this is a few years late, but you can accomplish this with the xargs -J option:
echo "input" | xargs -J "%" echo "%" "string"
And since it is xargs, you can do this on multiple lines of a file at once. If the file 'names' has three lines, like:
Adam
Bob
Charlie
You could do:
cat names | xargs -n 1 -J "%" echo "I like" "%" "because he is nice"
Also works:
seq -w 0 100 | xargs -I {} echo "string "{}
Will generate strings like:
string 000
string 001
string 002
string 003
string 004
...
The command you posted would take the string "input" use it as COMMAND's stdin stream, which would not produce the results you are looking for unless COMMAND first printed out the contents of its stdin and then printed out its command line arguments.
It seems like what you want to do is more close to command substitution.
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html#Command-Substitution
With command substitution you can have a commandline like this:
echo input `COMMAND "string"`
This will first evaluate COMMAND with "string" as input, and then expand the results of that commands execution onto a line, replacing what's between the ‘`’ characters.
cat will be my choice: ls | cat - <(echo new line)
With perl
echo "input" | perl -ne 'print "prefix $_"'
Output:
prefix input
A solution using sd (basically a modern sed; much easier to use IMO):
# replace '$' (end of string marker) with 'Ipsum'
# the `e` flag disables multi-line matching (treats all lines as one)
$ echo "Lorem" | sd --flags e '$' 'Ipsum'
Lorem
Ipsum#no new line here
You might observe that Ipsum appears on a new line, and the output is missing a \n. The reason is echo's output ends in a \n, and you didn't tell sd to add a new \n. sd is technically correct because it's doing exactly what you are asking it to do and nothing else.
However this may not be what you want, so instead you can do this:
# replace '\n$' (new line, immediately followed by end of string) by 'Ipsum\n'
# don't forget to re-add the `\n` that you removed (if you want it)
$ echo "Lorem" | sd --flags e '\n$' 'Ipsum\n'
LoremIpsum
If you have a multi-line string, but you want to append to the end of each individual line:
$ ls
foo bar baz
$ ls | sd '\n' '/file\n'
bar/file
baz/file
foo/file
I want to prepend my sql script with "set" statement before running it.
So I echo the "set" instruction, then pipe it to cat. Command cat takes two parameters : STDIN marked as "-" and my sql file, cat joins both of them to one output. Next I pass the result to mysql command to run it as a script.
echo "set #ZERO_PRODUCTS_DISPLAY='$ZERO_PRODUCTS_DISPLAY';" | cat - sql/test_parameter.sql | mysql
p.s. mysql login and password stored in .my.cnf file
For example look at the following line of bash-code
eval `echo "ls *.jpg"`
It lists all jpgs in the current directory. Now I want it to just print the line to the prompt so I can edit it before executing. (Like key-up does for example)
How do I do that?
The reason for this question comes from a much more usefull alias:
alias ac="history 2 | sed -n '1 s/[ 0-9]*//p' >> ~/.commands; sort -fu ~/.commands > ~/.commandsTmp; mv ~/.commandsTmp ~/.commands"
alias sc='oldIFS=$IFS; IFS=$'\n'; text=(); while read line ; do text=( ${text[#]-} "${line}") ; done < ~/.commands; PS3="Choose command by number: " ; eval `select selection in ${text[#]}; do echo "$selection"; break; done`; IFS=$oldIFS'
alias rc='awk '"'"'{print NR,$0}'"'"' ~/.commands; read -p "Remove number: " number; sed "${number} d" ~/.commands > ~/.commandsTmp; mv ~/.commandsTmp ~/.commands'
Where ac adds or remembers the last typed command, sc shows the available commands and executes them and rc deletes or forgets a command. (You need to touch ~/.commands before it works)
It would be even more usefull if I could edit the output of sc before executing it.
history -s whatever you want
will append "whatever you want" to your bash history. Then a simple up arrow (or !! followed by enter if you have shopt histreedit enabled --- I think that's the option I'm thinking of, not 100% sure), will give you "whatever you want" on the command line, ready to be edited.
Some comments on your aliases:
Simplified quoting:
alias rc='awk "{print NR,\$0}" ~/.commands ...'
No need for tail and you can combine calls to sed:
alias ac="history 2 | sed -n '1 s/[ 0-9]*//p'..."
Simplified eval and no need for $IFS:
alias sc='text=(); while read line ; do text+=("${line}") ; done < ~/.commands; PS3="Choose command by number: " ; select selection in "${text[#]}"; do eval "$selection"; break; done'
#OP, you should really put those commands into subroutines, and when you want to use them, source it. (taken from dennis's answers)
rc(){
awk "{print NR,\$0}" ~/.commands ...
}
ac(){
history 2 | sed -n '1 s/[ 0-9]*//p'...
}
sc(){
text=()
while read line
do
text+=("${line}")
done < ~/.commands
PS3="Choose command by number: "
select selection in "${text[#]}"
do
eval "$selection"
break
done
}
then save it as "library.sh" or something and when you want to use it
$ source /path/to/library.sh
Or
$ . /path/to/library.sh
Maybe you could use preexec.bash?
http://www.twistedmatrix.com/users/glyph/preexec.bash.txt
(On a related note, you can edit the current command line by using ctrl-x-e as well!)
cheers,
tavod