I'm trying to use a variable in a sed append and hitting an issue.
The following command works as expected:
sed -i "\:#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stderr:a file = /path/to/other/file" /etc/conf/service.conf
However if I replace the pattern with a variable I'm hitting an error:
$ echo $item
#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout
$ sed -i "\:$item:a file = /path/to/other/file" /etc/conf/service.conf
sed: -e expression #1, char 122: unterminated address regex
EDIT for more info: So the 'item' variable is being populated from an array. That array is created from a readarray and grep:
$readarray LINES < <(grep "#file = /mnt/var/" /etc/conf/service.conf)
$item=${LINES[1]}
$echo $item
#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout
However I've found if i populate 'item' manually it then works e.g:
$item="#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout"
$sed -i "\:$item:a file = /path/to/other/file" /etc/conf/service.conf
$
So something strange seems to be happening with the readarray/grep
So the problem here turned out to be newline characters that were being pulled in as part of the grep.
This is why populating $item manually worked - no '\n'
Thanks to Ed Morton for pointing me in the right direction. While
echo "$item" | cat -v
did not show anything I added '-t' to the readarray command to trim newline characters:
$readarray -t LINES < <(grep "#file = /mnt/var/" /etc/conf/service.conf)
After that things worked as expected.
Related
I'm expecting this to be an easy one for someone (alas not me!).
Using a bash script, I want to replace a value in a config file '/etc/app/app.cfg' (ini style), using variables for both search and replace.
The Value Name and Value I wish to update (note the space either side of the equals:
LOGDIR = /etc/app/logs
I have defined the following in the bash script:
# Get existing LogDir value
CURRENT_LOGDIR=$(grep 'LogDir =' /pathtofile | sed 's/LogDir *= *//g')
# Set New LogDir
LOGDIR=/mnt/eft/fs1/logs
# Update LogDir if different
if [[ -d $(echo $CURRENT_LOGDIR) != $LOGDIR ]] ; then
# Update LogDir value:
**bash command - I need help with !**
fi
I have tried many combinations with sed, to no avail, hence asking this question.
Things I've tried:
echo "LogDir = $LOGDIR" | sed '#s/$CURRENT_DIR/$LOGDIR/#g' /etc/app/app.cfg
sed -i '/#/!s/\(LogDir[[:space:]]*=[[:space:]]*\)\(.*\)/\1$LOGDIR#/g' /etc/app/app.cfg
sed -i 's/LogDir[[:space:]]=.*/LogDir = {LOGDIR}/' /etc/app/app.cfg
sed -i "s/^LogDir[[:space:]]*=.*/LogDir=$LOGDIR/}" /etc/app/app.cfg
sed -i '/#/!s/\(LogDir[[:space:]]*=[[:space:]]*\)\(.*\)/\1"$LOGDIR"/' /etc/app/app.cfg
Desired output:
Update LogDir value in /etc/app/app.cfg
For example:
LogDir = /mnt/eft/fs1/logs
What is that } doing on the end? Looks like a typo.
sed "s/^LogDir[[:space:]]*=.*/LogDir=$LOGDIR/" /etc/app/app.cfg
And sed edit file in place
Using sed
$ LOGDIR=/mnt/eft/fs1/logs
$ sed -i.bak "s|\([^=]*=[[:space:]]\).*|\1$LOGDIR|" /etc/app/app.cfg
$ cat /etc/app/app.cfg
LOGDIR = /mnt/eft/fs1/logs
How about this:
awk '$1 == "LogDir" {print "LogDir = /mnt/eft/fs1/logs"; next} {print}' old_configuration_file >new_configuration_file
The first awk clause replaces the old LogDir entry by the new one, and the second clause passes the other lines through unchanged.
as i said in the title im trying to replace a string in a file, that contains special characters , now the idea is to loop on every line of a "infofile" contains many lines of: whatiwantotreplace,replacer.
once I have this i want to do sed to a certain file to replace all the occurrences of string-> "whatiwantotreplace" with ->"replacer".
my code:
infofile="inforfilepath"
replacefile="replacefilepath"
while IFS= read -r line
do
what2replace="a" #$(echo "$line" | cut -d"," -f1);
replacer="b\\" #$(echo "$line" | cut -d"," -f2 );
sed -i -e "s/$what2replace/$replacer/g" "$replacefile"
#sed -i -e "s/'$what2replace'/'$replacer'/g" "$replacefile"
#sed -i -e "s#$what2replace#$replacer#g" "$replacefile"
#sed -i -e s/$what2replace/$replacer/g' "$replacefile"
#sed -i -e "s/${what2replace}/${replacer}/g" "$replacefile"
#${replacefile//what2replace/replacer}
done < "$infofile"
As you can see, the string that want to replace and the string that i want to replace with,may contain special characters , all the commented lines are the things I tried (things I saw online) but still clueless.
for some i got this error:
"sed: -e expression #1, char 8: unterminated `s' command"
and for some just nothing happend.
really need your help
Edit: inputs and outputs:
It's hard to give inputs and output, because all of the variations I tried had the same thing , didn't changed anything, the only one gave the above error is the variation with #.
thanks for your effort.
You're barking up the wrong tree - you're trying to do literal string replacements using a tool, sed, that doesn't have functionality to handle literal strings. See Is it possible to escape regex metacharacters reliably with sed for the convoluted mess required to try to force sed to do what you want and also https://unix.stackexchange.com/q/169716/133219 for why to avoid shell loops for manipulating text.
Just use awk instead since it has literal string functions and loops implicitly itself:
awk '
NR==FNR{map[$1]=$2; next}
{
for (old in map) {
new = map[old]
head = ""
tail = $0
while ( s = index(tail,old) ) {
head = head substr(tail,1,s-1) new
tail = substr(tail,s+length(old))
}
$0 = head tail
}
}
' "$infofile" "$replacefile"
The above is untested of course since you didn't provide any sample input/output.
You can try this way
what='a';to='b\\\\';echo 'sdev adfc xdae' | sed "s/${what}/${to}/g"
output
sdev b\\dfc xdb\\e
I am trying to get a substring between &DEST= and the next & or a line break.
For example :
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
In this I need to extract "SFO"
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
In this I need to extract "SANFRANSISCO"
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
In this I need to extract "SANJOSE"
I am reading a file line by line, and I need to update the text after &DEST= and put it back in the file. The modification of the text is to mask the dest value with X character.
So, SFO should be replaced with XXX.
SANJOSE should be replaced with XXXXXXX.
Output :
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Please let me know how to achieve this in script (Preferably shell or bash script).
Thanks.
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=PORTORICA
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
$ sed -E 's/^.*&DEST=([^&]*)[&]*.*$/\1/' file
SFO
PORTORICA
SANFRANSISCO
SANJOSE
should do it
Replacing airports with an equal number of Xs
Let's consider this test file:
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
To replace the strings after &DEST= with an equal length of X and using GNU sed:
$ sed -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
To replace the file in-place:
sed -i -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
The above was tested with GNU sed. For BSD (OSX) sed, try:
sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
Or, to change in-place with BSD(OSX) sed, try:
sed -i '' -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
If there is some reason why it is important to use the shell to read the file line-by-line:
while IFS= read -r line
do
echo "$line" | sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta
done <file
How it works
Let's consider this code:
search_str="&DEST="
newfile=chart.txt
sed -E ':a; s/('"$search_str"'X*)[^X&]/\1X/; ta' "$newfile"
-E
This tells sed to use Extended Regular Expressions (ERE). This has the advantage of requiring fewer backslashes to escape things.
:a
This creates a label a.
s/('"$search_str"'X*)[^X&]/\1X/
This looks for $search_str followed by any number of X followed by any character that is not X or &. Because of the parens, everything except that last character is saved into group 1. This string is replaced by group 1, denoted \1 and an X.
ta
In sed, t is a test command. If the substitution was made (meaning that some character needed to be replaced by X), then the test evaluates to true and, in that case, ta tells sed to jump to label a.
This test-and-jump causes the substitution to be repeated as many times as necessary.
Replacing multiple tags with one sed command
$ name='DEST|ORIG'; sed -E ':a; s/(&('"$name"')=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=XXXX
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=XXXX
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Answer for original question
Using shell
$ s='MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546'
$ s=${s#*&DEST=}
$ echo ${s%%&*}
SFO
How it works:
${s#*&DEST=} is prefix removal. This removes all text up to and including the first occurrence of &DEST=.
${s%%&*} is suffix removal_. It removes all text from the first & to the end of the string.
Using awk
$ echo 'MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546' | awk -F'[=\n]' '$1=="DEST"{print $2}' RS='&'
SFO
How it works:
-F'[=\n]'
This tells awk to treat either an equal sign or a newline as the field separator
$1=="DEST"{print $2}
If the first field is DEST, then print the second field.
RS='&'
This sets the record separator to &.
With GNU bash:
while IFS= read -r line; do
[[ $line =~ (.*&DEST=)(.*)((&.*|$)) ]] && echo "${BASH_REMATCH[1]}fooooo${BASH_REMATCH[3]}"
done < file
Output:
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=fooooo
Replace the characters between &DEST and & (or EOL) with x's:
awk -F'&DEST=' '{
printf("%s&DEST=", $1);
xlen=index($2,"&");
if ( xlen == 0) xlen=length($2)+1;
for (i=0;i<xlen;i++) printf("%s", "X");
endstr=substr($2,xlen);
printf("%s\n", endstr);
}' file
I have a file in Unix, with data sample like the following:
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
The desired output is
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexico
456|Americas|Canada
567|APAC|Japan
I tried with a few sed commands. I could remove the following: '{', '}', ' " ', ':'
There are 2 issues with the output file
All rows from input appear in single line in the output.
Adding the pipe ('|') as delimiter.
Any pointers are highly appreciated.
I recommend the tool jq (http://stedolan.github.io/jq/); jq is a lightweight and flexible command-line JSON processor.
jq -r '"\(.ID)|\(.Region)|\(.Location)"' < infile
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Explanation
-r is --raw-output
Through awk,
awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
Example:
$ cat file
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
$ awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
EXplanation:
-F'"' Sets " as Field Separator value.
OFS="|" Sets | as Output Field Separator value.
Atfirst, awk would execute the function inside the BEGIN block. It helps to print the header section.
This sed one-liner does what you want. It's capturing the field values using parenthesized expressions, and then putting them into the output using \1, \2, and \3.
s/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/
Invoke it like:
$ sed -f one-liner.sed input.txt
Or you can invoke it within a Bash script, producing the header:
echo 'ID|Region|Location'
sed -e 's/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/' $input
It is a JSON file so it is best to use a JSON parser. Here is a perl implementation of it.
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
open my $fh, '<', 'path/to/your/file';
#keys of your structure
my #key = qw(ID Region Location);
print join ("|", #key), "\n";
#iterate over your file, decode it and print in order of your key structure
while (my $json = <$fh>) {
my $text = decode_json($json);
print join ("|", map { $$text{$_} } #key ),"\n";
}
Output:
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Using sed as follows
Command line
echo "my_string" |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
or
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g' my_file
I tried this in a terminal as follows:
echo '{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}' |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
Output
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Many thanks for your response and the pointers/ solutions did help a lot.
For some mysterious reasons, I couldn't get any sed commands work. So, I devised my own solution. Although it's not elegant, it's still worked.
Here is the script I prepared which resolved the issue.
#!/bin/bash
# ource file path.
infile=/home/exfile.txt
# remove if these temp file exist already.
rm ./efile.txt ./xfile.txt ./yfile.txt ./zfile.txt
# removing the curly braces from input file.
cat exfile.txt | cut -d "{" -f2 | cut -d "}" -f1 >> ./efile.txt
# setting input file name to different value.
infile=./efile.txt
# remove double quotes from the file.
while IFS= read -r line
do
echo $line | sed 's/\"//g' >> ./xfile.txt
done < "$infile"
# creating another temp file.
infile2=./xfile.txt
# remove colon from file.
while IFS= read -r line
do
echo $line | sed 's/\:/,/g' >> ./yfile.txt
done < "$infile2"
# set input file path to new temp file.
infile3=yfile.txt
# initialize variables to hold header column values.
t1=0
t3=0
t5=0
# read each of the line to extract header row. Exit loop after reading 1st row.
once=1
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
"$f1 $f2 $f3 $f4 $f5 $f6"
t1=$f1
t3=$f3
t5=$f5
if [ "$once" -eq 1 ]; then
break
fi
done < "$infile3"
# Read each of the line from input file. Write only the value to another output file.
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
echo "$f2|$f4|$f6" >> ./zfile.txt
done < "$infile3"
# insert the header column row into the file generated in the step above.
frstline="$t1|$t3|$t5"
sed -i '1i ID|Region|Location' ./zfile.txt
I'm trying to use sed to change a variable in the site.js file on my server.
Here is the line: var url = "page.php"; I'm looking to just substitute page.php for whatever.php.
I thought this would be pretty simple and I figured this would work with no issues:
sed -i "s/\url = \".*\"/\url = \"page2.php\"/" /home/site.js
It works okay except instead of getting: var url = "page2.php"; I get: var R1 = "page2.php";
Why is the url value being changed to R1 when I use sed here?
You don't need \ before url.
sed -i -r 's#url\s*=\s*"[^"]+"#url = "page2.php"#' /home/site.js
Extra escaping of " can be eliminated by enclosing sed expression with ' instead of "
It's better to use different separator than / (here #) when the strings themselves may contain /
Try doing this :
sed -i -r 's#(var\s+url\s*=\s*")[^"]+"#\1whatever.php"#' file.js
/ is not mandatory as delimiter, I've picked up # there.
Here's another example: Took me while to figure that you change the / for delimiter and not the / in the directory path.
Use # instead of / for sed delimiter if you have dir path names.
First I tried this:
[root#ip-172-35-24-37 ec2-user]# egrep -q "^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$" /etc/fstab && sed -ri "s/^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$/\1/dev/shm\2nodev\3\4/" /etc/fstab
And got this error:
sed: -e expression #1, char 20: unknown option to `s'
So then I used # for the sed delimiter instead of /:
[root#ip-172-35-24-37 ec2-user]# egrep -q "^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$" /etc/fstab && sed -ri "s#^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$#\1/dev/shm\2nodev\3\4##" /etc/fstab
[root#ip-172-35-24-37 ec2-user]#
And it worked.
You can use something else besides # for a delimiter like ! or ? or %. Just don't use / if you have dir paths.