This should be easy but i just can't get out of it:
I need to replace a piece of text in a .php file using the unix command-line.
using: sudo sed -i '' 's/STRING/REPLACEMENT/g' /file.php (The quotes after -i are needed because it runs on Mac Os X)
The string: ['password'] = ""; needs to be replaced by: ['password'] = "$PASS";
$PASS is a variable, so it gets filled in.
I got up to something like:
sudo sed -i '' 's/[\'password\'] = ""\;/[\'password\'] = "$PASS"\;/g' /file.php
But as i'm new with UNIX i don't know what to escape...
What should be changed? Thanks!
Unfortunately sed cannot robustly handle variables that might contain various characters that are "special" to sed and shell. You need to use awk for this, e.g. with GNU awk for gensub():
gawk -v pass="$PASS" '{$0=gensub(/(\[\047password\047] = \")/,"\\1"pass,"g")}1' file
See how sed fails below when PASS contains a forward slash but awk doesn't care:
$ cat file
The string: ['password'] = ""; needs to be replaced
$ PASS='foo'
$ awk -v pass="$PASS" '{$0=gensub(/(\[\047password\047] = \")/,"\\1"pass,"g")}1' file
The string: ['password'] = "foo"; needs to be replaced
$ sed "s/\(\['password'\] = \"\)\(\";\)/\1$PASS\2/g" file
The string: ['password'] = "foo"; needs to be replaced
$ PASS='foo/bar'
$ awk -v pass="$PASS" '{$0=gensub(/(\[\047password\047] = \")/,"\\1"pass,"g")}1' file
The string: ['password'] = "foo/bar"; needs to be replaced
$ sed "s/\(\['password'\] = \"\)\(\";\)/\1$PASS\2/g" file
sed: -e expression #1, char 38: unknown option to `s'
You need to use \047 or some other method (e.g. '"'"' if you prefer) to represent a single quote within an awk script that's single-quote-delimitted.
In awks without gensub() you just use gsub() instead:
awk -v pass="$PASS" '{pre="\\[\047password\047] = \""; gsub(pre,pre pass)}1' file
if you want to expand variable in sed, you have to use double quote, so something like
sed -i... "s/.../.../g" file
that is, you don't have to escape those single quotes, also you could use group reference to save some typing. you could try:
sudo sed -i '' "s/\(\['password'\] = \"\)\(\";\)/\1$PASS\2/g" /file.php
Related
I want to replace a string in a file with the value of a variable. I a string lvl in the template file prm.prm which needs to be replaced by the value of SLURM_ARRAY.
I tried using
sed -i 's/lvl/${SLURM_ARRAY}/' prm.prm
This replaces the string lvl with ${SLURM_ARRAY} and not its value. How can I rectify this?
Every character between single quotes is used literally.
You could use double quotes instead as follows:
sed -i "s/lvl/${SLURM_ARRAY}/" prm.prm
However, your code now suffers from a code injection bug. There are characters (e.g. /) that will cause problems if found in the value of SLURM_ARRAY. To avoid this, these characters needs to be escaped.
quotemeta() { printf %s "$1" | sed 's/\([^a-zA-Z0-9]\)/\\\1/g'; }
sed -i "s/lvl/$( quotemeta "$SLURM_ARRAY" )/" prm.prm
However, it would be best to avoid generating programs from the shell. But that would require avoiding sed since it doesn't provide the necessary tools. For example, a couple of Perl solutions:
perl -i -spe's/lvl/$s/' -- -s="$SLURM_ARRAY" prm.prm
S="$SLURM_ARRAY" perl -i -pe's/lvl/$ENV{S}/' prm.prm
Replace pattern with the value of an environment variable, with no extra interpolation of the contents:
Using perl:
export SLURM_ARRAY
perl -pe's/lvl/$ENV{SLURM_ARRAY}/g' prm.prm
Using awk:
export SLURM_ARRAY
awk '
{
if (match($0, /lvl/)) {
printf "%s", substr($0, 1, RSTART - 1)
printf "%s", ENVIRON["SLURM_ARRAY"]
print substr($0, RSTART + RLENGTH)
}
else {
print
}
}
' prm.prm
There's also SLURM_ARRAY=$SLURM_ARRAY perl ...etc or similar, to set the environment of a single process.
It can also be done with the variable as an argument. With both perl and awk you can access and modify the ARGV array. For awk you have to reset it so it's not processed as a file. The perl version looks like perl -e 'my $r = $ARGV[0]; while (<STDIN>) {s/lvl/$r/g; print}' "$SLURM_ARRAY" < prm.prm. It looks even better as perl -spe's/lvl/$r/g' -- -r="$SLURM_ARRAY". Thanks to ikegami.
For awk, I should say that the reason for not using awk -v r=foo is the expansion of C escapes. You could also read the value from a file (or bash process substitution).
I am trying to create a script to find a JSON element and update it with the arg values.
#!/bin/bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
sed -i '' '/location/c\ \"location\" : \"$placeLocation\",' $file
sed -i '' '/heading/c\ \"heading\" : \"$vehicleHeading\",' $file
sed -i '' '/message/c\ \"message\" : \"$message\",' $file
One.txt
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
But getting following error
sed: 1: "/location/c\ \"locati ...": extra characters after \ at the end of c command
sed: 1: "/heading/c\ \"heading ...": extra characters after \ at the end of c command
sed: 1: "/message/c\ \"message ...": extra characters after \ at the end of c command
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
I have just started learning about sed editor and tried out multiple things but couldn't able to figure it out. Any help is much appreciated!
Note that you probably should consider using a tool like jq for editing JSON files. But I assume you have a good reason for using sed, so you have a couple of problems there.
The first is that you're trying to use GNU sed features on your Mac OS X version of sed that doesn't have those features. If you want GNU sed on Mac OS X, then install it:
▶ brew install gnu-sed
Fixing up your code for GNU sed (and also for other Bash style guide recommendations about quoting strings):
cat > FILE <<EOF
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
EOF
placeLocation=myPlaceLocation
vehicleHeading=myVehicleHeading
message=myMessage
file=FILE
gsed -i -e '/location/c\' -e '"location": "'"$placeLocation"'",' "$file"
gsed -i -e '/heading/c\' -e '"heading": "'"$vehicleHeading"'",' "$file"
gsed -i -e '/message/c\' -e '"message": "'"$message"'",' "$file"
As noted in the GNU sed manual, use of multiple -e commands on the same line with the c\ command is a GNU extension.
If you want to use Mac OS X's sed, just may be able to write it this way:
sed -i '' '
s/"location".*/"location": "'"$placeLocation"'",/
s/"heading".*/"heading": "'"$vehicleHeading"'",/
s/"message".*/"message": "'"$message"'",/
' "$file"
But note you would have to sanitise the input if you need the code to be robust to all inputs.
To do this robustly you need to use a tool that understands literal strings (which sed doesn't - see Is it possible to escape regex metacharacters reliably with sed) e.g. awk:
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1/2?' one.txt
"message":"what is 1/2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1&2?' one.txt
"message":"what is 1&2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is \1?' one.txt
"message":"what is \1?"
the above will work robustly using any awk in any shell on every UNIX box. Try using those as replacement strings in a sed command.
The full script to do what you want would be:
#!/bin/env bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
tmp=$(mktemp)
awk '
BEGIN {
split("location heading message", tags)
for (i in tags) {
vals["\"" tags[i] "\""] = "\"" ARGV[i] "\""
ARGV[i] = ""
}
FS=OFS=":"
}
$1 in vals {
tag = $1
sub(FS".*","")
$0 = tag OFS vals[tag]
}
1' "$placeLocation" "$vehicleHeading" "$message" "$file" > "$tmp" && mv "$tmp" "$file"
I'm trying to use a variable in a sed append and hitting an issue.
The following command works as expected:
sed -i "\:#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stderr:a file = /path/to/other/file" /etc/conf/service.conf
However if I replace the pattern with a variable I'm hitting an error:
$ echo $item
#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout
$ sed -i "\:$item:a file = /path/to/other/file" /etc/conf/service.conf
sed: -e expression #1, char 122: unterminated address regex
EDIT for more info: So the 'item' variable is being populated from an array. That array is created from a readarray and grep:
$readarray LINES < <(grep "#file = /mnt/var/" /etc/conf/service.conf)
$item=${LINES[1]}
$echo $item
#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout
However I've found if i populate 'item' manually it then works e.g:
$item="#file = /mnt/var/log/hadoop-yarn/containers/application_1495965866386_0001/container_1495965866386_0001_01_000002/stdout"
$sed -i "\:$item:a file = /path/to/other/file" /etc/conf/service.conf
$
So something strange seems to be happening with the readarray/grep
So the problem here turned out to be newline characters that were being pulled in as part of the grep.
This is why populating $item manually worked - no '\n'
Thanks to Ed Morton for pointing me in the right direction. While
echo "$item" | cat -v
did not show anything I added '-t' to the readarray command to trim newline characters:
$readarray -t LINES < <(grep "#file = /mnt/var/" /etc/conf/service.conf)
After that things worked as expected.
I'm trying to get a multiline output from a CSV into one line in Bash.
My CSV file looks like this:
hi,bye
hello,goodbye
The end goal is for it to look like this:
"hi/bye", "hello/goodbye"
This is currently where I'm at:
INPUT=mycsvfile.csv
while IFS=, read col1 col2 || [ -n "$col1" ]
do
source=$(awk '{print;}' | sed -e 's/,/\//g' )
echo "$source";
done < $INPUT
The output is on every line and I'm able to change the , to a / but I'm not sure how to put the output on one line with quotes around it.
I've tried BEGIN:
source=$(awk 'BEGIN { ORS=", " }; {print;}'| sed -e 's/,/\//g' )
But this only outputs the last line, and omits the first hi/bye:
hello/goodbye
Would anyone be able to help me?
Just do the whole thing (mostly) in awk. The final sed is just here to trim some trailing cruft and inject a newline at the end:
< mycsvfile.csv awk '{print "\""$1, $2"\""}' FS=, OFS=/ ORS=", " | sed 's/, $//'
If you're willing to install trl, a utility of mine, the command can be simplified as follows:
input=mycsvfile.csv
trl -R '| ' < "$input" | tr ',|' '/,'
trl transforms multiline input into double-quoted single-line output separated by ,<space> by default.
-R '| ' (temporarily) uses |<space> as the separator instead; this assumes that your data doesn't contain | instances, but you can choose any char. that you know not be part of your data.
tr ',|' '/,' then translates all , instances (field-internal to the input lines) into / instances, and all | instances (the temporary separator) into , instances, yielding the overall result as desired.
Installation of trl from the npm registry (Linux and macOS)
Note: Even if you don't use Node.js, npm, its package manager, works across platforms and is easy to install; try
curl -L https://git.io/n-install | bash
With Node.js installed, install as follows:
[sudo] npm install trl -g
Note:
Whether you need sudo depends on how you installed Node.js and whether you've changed permissions later; if you get an EACCES error, try again with sudo.
The -g ensures global installation and is needed to put trl in your system's $PATH.
Manual installation (any Unix platform with bash)
Download this bash script as trl.
Make it executable with chmod +x trl.
Move it or symlink it to a folder in your $PATH, such as /usr/local/bin (macOS) or /usr/bin (Linux).
$ awk -F, -v OFS='/' -v ORS='"' '{$1=s ORS $1; s=", "; print} END{printf RS}' file
"hi/bye", "hello/goodbye"
There is no need for a bash loop, which is invariably slow.
sed and tr can do this more efficiently:
input=mycsvfile.csv
sed 's/,/\//g; s/.*/"&", /; $s/, $//' "$input" | tr -d '\n'
s/,/\//g uses replaces all (g) , instances with / instances (escaped as \/ here).
s/.*/"&", / encloses the resulting line in "...", followed by ,<space>:
regex .* matches the entire pattern space (the potentially modified input line)
& in the replacement string represent that match.
$s/, $// removes the undesired trailing ,<space> from the final line ($)
tr -d '\n' then simply removes the newlines (\n) from the result, because sed invariably outputs each line with a trailing newline.
Note that the above command's single-line output will not have a trailing newline; simply append ; printf '\n' if it is needed.
In awk:
$ awk '{sub(/,/,"/");gsub(/^|$/,"\"");b=b (NR==1?"":", ")$0}END{print b}' file
"hi/bye", "hello/goodbye"
Explained:
$ awk '
{
sub(/,/,"/") # replace comma
gsub(/^|$/,"\"") # add quotes
b=b (NR==1?"":", ") $0 # buffer to add delimiters
}
END { print b } # output
' file
I'm assuming you just have 2 lines in your file? If you have alternating 2 line pairs, let me know in comments and I will expand for that general case. Here is a one-line awk conversion for you:
# NOTE: I am using the octal ascii code for the
# double quote char (\42=") in my printf statement
$ awk '{gsub(/,/,"/")}NR==1{printf("\42%s\42, ",$0)}NR==2{printf("\42%s\42\n",$0)}' file
output:
"hi/bye", "hello/goodbye"
Here is my attempt in awk:
awk 'BEGIN{ ORS = " " }{ a++; gsub(/,/, "/"); gsub(/[a-z]+\/[a-z]+/, "\"&\""); print $0; if (a == 1){ print "," }}{ if (a==2){ printf "\n"; a = 0 } }'
Works also if your Input has more than two lines.If you need some explanation feel free to ask :)
I am trying to get a substring between &DEST= and the next & or a line break.
For example :
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
In this I need to extract "SFO"
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
In this I need to extract "SANFRANSISCO"
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
In this I need to extract "SANJOSE"
I am reading a file line by line, and I need to update the text after &DEST= and put it back in the file. The modification of the text is to mask the dest value with X character.
So, SFO should be replaced with XXX.
SANJOSE should be replaced with XXXXXXX.
Output :
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Please let me know how to achieve this in script (Preferably shell or bash script).
Thanks.
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=PORTORICA
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
$ sed -E 's/^.*&DEST=([^&]*)[&]*.*$/\1/' file
SFO
PORTORICA
SANFRANSISCO
SANJOSE
should do it
Replacing airports with an equal number of Xs
Let's consider this test file:
$ cat file
MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=SANFRANSISCO&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=SANJOSE
To replace the strings after &DEST= with an equal length of X and using GNU sed:
$ sed -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
To replace the file in-place:
sed -i -E ':a; s/(&DEST=X*)[^X&]/\1X/; ta' file
The above was tested with GNU sed. For BSD (OSX) sed, try:
sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
Or, to change in-place with BSD(OSX) sed, try:
sed -i '' -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta file
If there is some reason why it is important to use the shell to read the file line-by-line:
while IFS= read -r line
do
echo "$line" | sed -Ee :a -e 's/(&DEST=X*)[^X&]/\1X/' -e ta
done <file
How it works
Let's consider this code:
search_str="&DEST="
newfile=chart.txt
sed -E ':a; s/('"$search_str"'X*)[^X&]/\1X/; ta' "$newfile"
-E
This tells sed to use Extended Regular Expressions (ERE). This has the advantage of requiring fewer backslashes to escape things.
:a
This creates a label a.
s/('"$search_str"'X*)[^X&]/\1X/
This looks for $search_str followed by any number of X followed by any character that is not X or &. Because of the parens, everything except that last character is saved into group 1. This string is replaced by group 1, denoted \1 and an X.
ta
In sed, t is a test command. If the substitution was made (meaning that some character needed to be replaced by X), then the test evaluates to true and, in that case, ta tells sed to jump to label a.
This test-and-jump causes the substitution to be repeated as many times as necessary.
Replacing multiple tags with one sed command
$ name='DEST|ORIG'; sed -E ':a; s/(&('"$name"')=X*)[^X&]/\1X/; ta' file
MYREQUESTISTO8764GETTHIS&DEST=XXX&ORIG=XXXX
MYREQUESTISTO8764GETTHIS&DEST=XXXXXXXXXXXX&ORIG=XXXX
MYREQUESTISTO8764GETTHISWITH&DEST=XXXXXXX
Answer for original question
Using shell
$ s='MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546'
$ s=${s#*&DEST=}
$ echo ${s%%&*}
SFO
How it works:
${s#*&DEST=} is prefix removal. This removes all text up to and including the first occurrence of &DEST=.
${s%%&*} is suffix removal_. It removes all text from the first & to the end of the string.
Using awk
$ echo 'MYREQUESTISTO8764GETTHIS&DEST=SFO&ORIG=6546' | awk -F'[=\n]' '$1=="DEST"{print $2}' RS='&'
SFO
How it works:
-F'[=\n]'
This tells awk to treat either an equal sign or a newline as the field separator
$1=="DEST"{print $2}
If the first field is DEST, then print the second field.
RS='&'
This sets the record separator to &.
With GNU bash:
while IFS= read -r line; do
[[ $line =~ (.*&DEST=)(.*)((&.*|$)) ]] && echo "${BASH_REMATCH[1]}fooooo${BASH_REMATCH[3]}"
done < file
Output:
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHIS&DEST=fooooo&ORIG=6546
MYREQUESTISTO8764GETTHISWITH&DEST=fooooo
Replace the characters between &DEST and & (or EOL) with x's:
awk -F'&DEST=' '{
printf("%s&DEST=", $1);
xlen=index($2,"&");
if ( xlen == 0) xlen=length($2)+1;
for (i=0;i<xlen;i++) printf("%s", "X");
endstr=substr($2,xlen);
printf("%s\n", endstr);
}' file