I'm trying to use variables in a sed command in Freebsd. sed in Freebsd expects \ after a. Basically I want to append a line if a particular line in the file matches a pattern. I'm using sed's append for that.
#!/usr/bin/bash
SYSLOG_SERVER="192.168.1.36"
SYSLOG_PORT="514"
syslog_conf_file="/etc/syslog.conf"
send_logs() {
logs=(messages auth.log )
send_logs[0]=`awk '(index($2, "messages") != 0) {print $1}' $syslog_conf_file`
send_logs[1]=`awk '(index($2, "auth.log") != 0) {print $1}' $syslog_conf_file`
for (( i = 0 ; i < ${#send_logs[#]} ; i++ ))
do
if [ ! -z "${send_logs[$i]}" ]; then
send_logs[i]=${send_logs[i]}" \t"#$SYSLOG_SERVER:$SYSLOG_PORT
sed "/${logs[$i]}$/a\
${send_logs[$i]} \
" $syslog_conf_file
fi
done
}
I'm facing this error. The variables are printed properly but the way in which I'm running the script is wrong. How can I fix this ?
root#Great# bash temp.sh
send_logs *.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err \t#192.168.1.36:514
logs messages
sed: 1: "/messages$/a ...": command a expects \ followed by text
send_logs auth.info;authpriv.info \t#192.168.1.36:514
logs auth.log
sed: 1: "/auth.log$/a ...": command a expects \ followed by text
Sample expected input for sed:
root#Great# sed '/messages$/a\
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err #192.168.1.36:514 \
' /etc/syslog.conf
Expected output:
*.err;kern.warning;auth.notice;mail.crit /dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err /var/log/messages
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err #192.168.1.36:514
security.* /var/log/security
auth.info;authpriv.info /var/log/auth.log
It's because \ followed by a newline character means following line to be joined, to avoid escape : \\ :
sed "/${logs[$i]}$/a\\
${send_logs[$i]} \\
" $syslog_conf_file
Related
I have a text file:
org.jitsi.videobridge.xmpp.user.shard-1.HOSTNAME=localhost
org.jitsi.videobridge.xmpp.user.shard-1.DOMAIN=auth.jc.name.com
org.jitsi.videobridge.xmpp.user.shard-1.USERNAME=name
org.jitsi.videobridge.xmpp.user.shard-1.PASSWORD=Hfr*7462
org.jitsi.videobridge.xmpp.user.shard-1.MUC_JIDS=JvbBredjoy#internal.auth.jc.name.com
org.jitsi.videobridge.xmpp.user.shard-1.MUC_NICKNAME=7896aee5-fgre-4b02-4569-0bcc75ed1d0d
I created a bash script:
#!/bin/bash
DPATH="/etc/jitsi/videobridge/sip-communicator.properties"
k=$(grep -o 'shard-1' $DPATH) # shard ends by a number#
i=$(grep -o 'shard-1' $DPATH | cut -c7)
m=$((i+1))
n="shard-$m"
sed -i "s|${k}|${n}|g" $DPATH
But I get error:
/home/scripts# ./shard_number
./shard_number: line 5: 1
1
1
1
1
1: syntax error in expression (error token is "1
1
1
1
1")
sed: -e expression #1, char 9: unterminated `s' command
Could you please help to solve this issue? Thank you.
If you call your script with bash -x /path/to/your/script or add set -x somewhere at the start of your script (after the #!shebang, but before the commands you want to debug), you will see that your grep commands return not a single 'shard-1' but rather one 'shard-1' per line :
++ grep -o shard-1 /etc/jitsi/videobridge/sip-communicator.properties
+ k='shard-1
shard-1
shard-1
shard-1
shard-1
shard-1'
Once cut, that gives the 1\n1\n1\n1\n1\n string that is mentionned in your error output as an invalid token for the $(( ... )) expression, which also breaks the syntax of your sed substitution :
++ cut -c7
++ grep -o shard-1 /etc/jitsi/videobridge/sip-communicator.properties
+ i='1
1
1
1
1
1'
Make that string a single number (for instance piping your grep into sort -u to unicize all the shards found) and your script will work just fine :
#!/bin/bash
DPATH="/etc/jitsi/videobridge/sip-communicator.properties"
k=$(grep -o 'shard-1' $DPATH | sort -u) # shard ends by a number#
i=$(grep -o 'shard-1' $DPATH | sort -u | cut -c7)
m=$((i+1))
n="shard-$m"
sed -i "s|${k}|${n}|g" $DPATH
You can try it here. Also check this test if you want to see your initial script debugged.
I am trying to create a script to find a JSON element and update it with the arg values.
#!/bin/bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
sed -i '' '/location/c\ \"location\" : \"$placeLocation\",' $file
sed -i '' '/heading/c\ \"heading\" : \"$vehicleHeading\",' $file
sed -i '' '/message/c\ \"message\" : \"$message\",' $file
One.txt
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
But getting following error
sed: 1: "/location/c\ \"locati ...": extra characters after \ at the end of c command
sed: 1: "/heading/c\ \"heading ...": extra characters after \ at the end of c command
sed: 1: "/message/c\ \"message ...": extra characters after \ at the end of c command
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
I have just started learning about sed editor and tried out multiple things but couldn't able to figure it out. Any help is much appreciated!
Note that you probably should consider using a tool like jq for editing JSON files. But I assume you have a good reason for using sed, so you have a couple of problems there.
The first is that you're trying to use GNU sed features on your Mac OS X version of sed that doesn't have those features. If you want GNU sed on Mac OS X, then install it:
▶ brew install gnu-sed
Fixing up your code for GNU sed (and also for other Bash style guide recommendations about quoting strings):
cat > FILE <<EOF
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
EOF
placeLocation=myPlaceLocation
vehicleHeading=myVehicleHeading
message=myMessage
file=FILE
gsed -i -e '/location/c\' -e '"location": "'"$placeLocation"'",' "$file"
gsed -i -e '/heading/c\' -e '"heading": "'"$vehicleHeading"'",' "$file"
gsed -i -e '/message/c\' -e '"message": "'"$message"'",' "$file"
As noted in the GNU sed manual, use of multiple -e commands on the same line with the c\ command is a GNU extension.
If you want to use Mac OS X's sed, just may be able to write it this way:
sed -i '' '
s/"location".*/"location": "'"$placeLocation"'",/
s/"heading".*/"heading": "'"$vehicleHeading"'",/
s/"message".*/"message": "'"$message"'",/
' "$file"
But note you would have to sanitise the input if you need the code to be robust to all inputs.
To do this robustly you need to use a tool that understands literal strings (which sed doesn't - see Is it possible to escape regex metacharacters reliably with sed) e.g. awk:
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1/2?' one.txt
"message":"what is 1/2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1&2?' one.txt
"message":"what is 1&2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is \1?' one.txt
"message":"what is \1?"
the above will work robustly using any awk in any shell on every UNIX box. Try using those as replacement strings in a sed command.
The full script to do what you want would be:
#!/bin/env bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
tmp=$(mktemp)
awk '
BEGIN {
split("location heading message", tags)
for (i in tags) {
vals["\"" tags[i] "\""] = "\"" ARGV[i] "\""
ARGV[i] = ""
}
FS=OFS=":"
}
$1 in vals {
tag = $1
sub(FS".*","")
$0 = tag OFS vals[tag]
}
1' "$placeLocation" "$vehicleHeading" "$message" "$file" > "$tmp" && mv "$tmp" "$file"
I'm trying to use a variable in a sed append and hitting an issue.
The following command works as expected:
sed -i "\:#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stderr:a file = /path/to/other/file" /etc/conf/service.conf
However if I replace the pattern with a variable I'm hitting an error:
$ echo $item
#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout
$ sed -i "\:$item:a file = /path/to/other/file" /etc/conf/service.conf
sed: -e expression #1, char 122: unterminated address regex
EDIT for more info: So the 'item' variable is being populated from an array. That array is created from a readarray and grep:
$readarray LINES < <(grep "#file = /mnt/var/" /etc/conf/service.conf)
$item=${LINES[1]}
$echo $item
#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout
However I've found if i populate 'item' manually it then works e.g:
$item="#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout"
$sed -i "\:$item:a file = /path/to/other/file" /etc/conf/service.conf
$
So something strange seems to be happening with the readarray/grep
So the problem here turned out to be newline characters that were being pulled in as part of the grep.
This is why populating $item manually worked - no '\n'
Thanks to Ed Morton for pointing me in the right direction. While
echo "$item" | cat -v
did not show anything I added '-t' to the readarray command to trim newline characters:
$readarray -t LINES < <(grep "#file = /mnt/var/" /etc/conf/service.conf)
After that things worked as expected.
I am trying to get a substring between &DEST= and the next & or a line break.
For example :
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
In this I need to extract "SFO"
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
In this I need to extract "SANFRANSISCO"
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
In this I need to extract "SANJOSE"
I am reading a file line by line, and I need to update the text after &DEST= and put it back in the file. The modification of the text is to mask the dest value with X character.
So, SFO should be replaced with XXX.
SANJOSE should be replaced with XXXXXXX.
Output :
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Please let me know how to achieve this in script (Preferably shell or bash script).
Thanks.
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=PORTORICA
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
$ sed -E 's/^.*&DEST=([^&]*)[&]*.*$/\1/' file
SFO
PORTORICA
SANFRANSISCO
SANJOSE
should do it
Replacing airports with an equal number of Xs
Let's consider this test file:
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
To replace the strings after &DEST= with an equal length of X and using GNU sed:
$ sed -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
To replace the file in-place:
sed -i -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
The above was tested with GNU sed. For BSD (OSX) sed, try:
sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
Or, to change in-place with BSD(OSX) sed, try:
sed -i '' -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
If there is some reason why it is important to use the shell to read the file line-by-line:
while IFS= read -r line
do
echo "$line" | sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta
done <file
How it works
Let's consider this code:
search_str="&DEST="
newfile=chart.txt
sed -E ':a; s/('"$search_str"'X*)[^X&]/\1X/; ta' "$newfile"
-E
This tells sed to use Extended Regular Expressions (ERE). This has the advantage of requiring fewer backslashes to escape things.
:a
This creates a label a.
s/('"$search_str"'X*)[^X&]/\1X/
This looks for $search_str followed by any number of X followed by any character that is not X or &. Because of the parens, everything except that last character is saved into group 1. This string is replaced by group 1, denoted \1 and an X.
ta
In sed, t is a test command. If the substitution was made (meaning that some character needed to be replaced by X), then the test evaluates to true and, in that case, ta tells sed to jump to label a.
This test-and-jump causes the substitution to be repeated as many times as necessary.
Replacing multiple tags with one sed command
$ name='DEST|ORIG'; sed -E ':a; s/(&('"$name"')=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=XXXX
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=XXXX
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Answer for original question
Using shell
$ s='MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546'
$ s=${s#*&DEST=}
$ echo ${s%%&*}
SFO
How it works:
${s#*&DEST=} is prefix removal. This removes all text up to and including the first occurrence of &DEST=.
${s%%&*} is suffix removal_. It removes all text from the first & to the end of the string.
Using awk
$ echo 'MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546' | awk -F'[=\n]' '$1=="DEST"{print $2}' RS='&'
SFO
How it works:
-F'[=\n]'
This tells awk to treat either an equal sign or a newline as the field separator
$1=="DEST"{print $2}
If the first field is DEST, then print the second field.
RS='&'
This sets the record separator to &.
With GNU bash:
while IFS= read -r line; do
[[ $line =~ (.*&DEST=)(.*)((&.*|$)) ]] && echo "${BASH_REMATCH[1]}fooooo${BASH_REMATCH[3]}"
done < file
Output:
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=fooooo
Replace the characters between &DEST and & (or EOL) with x's:
awk -F'&DEST=' '{
printf("%s&DEST=", $1);
xlen=index($2,"&");
if ( xlen == 0) xlen=length($2)+1;
for (i=0;i<xlen;i++) printf("%s", "X");
endstr=substr($2,xlen);
printf("%s\n", endstr);
}' file
I have a file in Unix, with data sample like the following:
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
The desired output is
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexico
456|Americas|Canada
567|APAC|Japan
I tried with a few sed commands. I could remove the following: '{', '}', ' " ', ':'
There are 2 issues with the output file
All rows from input appear in single line in the output.
Adding the pipe ('|') as delimiter.
Any pointers are highly appreciated.
I recommend the tool jq (http://stedolan.github.io/jq/); jq is a lightweight and flexible command-line JSON processor.
jq -r '"\(.ID)|\(.Region)|\(.Location)"' < infile
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Explanation
-r is --raw-output
Through awk,
awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
Example:
$ cat file
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
$ awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
EXplanation:
-F'"' Sets " as Field Separator value.
OFS="|" Sets | as Output Field Separator value.
Atfirst, awk would execute the function inside the BEGIN block. It helps to print the header section.
This sed one-liner does what you want. It's capturing the field values using parenthesized expressions, and then putting them into the output using \1, \2, and \3.
s/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/
Invoke it like:
$ sed -f one-liner.sed input.txt
Or you can invoke it within a Bash script, producing the header:
echo 'ID|Region|Location'
sed -e 's/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/' $input
It is a JSON file so it is best to use a JSON parser. Here is a perl implementation of it.
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
open my $fh, '<', 'path/to/your/file';
#keys of your structure
my #key = qw(ID Region Location);
print join ("|", #key), "\n";
#iterate over your file, decode it and print in order of your key structure
while (my $json = <$fh>) {
my $text = decode_json($json);
print join ("|", map { $$text{$_} } #key ),"\n";
}
Output:
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Using sed as follows
Command line
echo "my_string" |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
or
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g' my_file
I tried this in a terminal as follows:
echo '{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}' |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
Output
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Many thanks for your response and the pointers/ solutions did help a lot.
For some mysterious reasons, I couldn't get any sed commands work. So, I devised my own solution. Although it's not elegant, it's still worked.
Here is the script I prepared which resolved the issue.
#!/bin/bash
# ource file path.
infile=/home/exfile.txt
# remove if these temp file exist already.
rm ./efile.txt ./xfile.txt ./yfile.txt ./zfile.txt
# removing the curly braces from input file.
cat exfile.txt | cut -d "{" -f2 | cut -d "}" -f1 >> ./efile.txt
# setting input file name to different value.
infile=./efile.txt
# remove double quotes from the file.
while IFS= read -r line
do
echo $line | sed 's/\"//g' >> ./xfile.txt
done < "$infile"
# creating another temp file.
infile2=./xfile.txt
# remove colon from file.
while IFS= read -r line
do
echo $line | sed 's/\:/,/g' >> ./yfile.txt
done < "$infile2"
# set input file path to new temp file.
infile3=yfile.txt
# initialize variables to hold header column values.
t1=0
t3=0
t5=0
# read each of the line to extract header row. Exit loop after reading 1st row.
once=1
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
"$f1 $f2 $f3 $f4 $f5 $f6"
t1=$f1
t3=$f3
t5=$f5
if [ "$once" -eq 1 ]; then
break
fi
done < "$infile3"
# Read each of the line from input file. Write only the value to another output file.
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
echo "$f2|$f4|$f6" >> ./zfile.txt
done < "$infile3"
# insert the header column row into the file generated in the step above.
frstline="$t1|$t3|$t5"
sed -i '1i ID|Region|Location' ./zfile.txt