For a CI / CD pipeline I would like to replace a placeholder in my nginx.conf file with a generated map which looks like the following:
map $cookie_language $lang {
default en
en en
de de
es es
}
In the nginx.conf file there is the placeholder # REPLACE_ME_WITH_LANGUAGE_MAP which I am replacing with the following command:
sed -i -e "s/# REPLACE_ME_WITH_LANGUAGE_MAP/$languagemap/g" ./ci/nginx.conf
Full script looks like the following:
languagemap='map $cookie_language $lang {'
firstlanguage=$(jq -r '.locales[0]' src/assets/locales.json | jq -r '.build')
languagemap="${languagemap}| default $firstlanguage"
for locale in $(jq -r '.locales[] | #base64' src/assets/locales.json); do
lang=$(echo "$locale" | base64 --decode | jq -r '.build')
languagemap="${languagemap}| $lang $lang"
npm run ci-build -- --output-path ${OUTPUT_PATH}/$lang --configuration=${ANGULAR_CONFIGURATION} --i18n-format=xlf --i18n-file=src/locale/messages.$lang.xlf --i18n-locale=$lang
done
languagemap="${languagemap}|}"
sed -i "t" "s/# REPLACE_ME_WITH_LANGUAGE_MAP/$languagemap/g" ./ci/nginx.conf
Running this always brings up:
+ sed -i tmp s/# REPLACE_ME_WITH_LANGUAGE_MAP/map $cookie_language $lang {| default en| en en| de de|}/g ./ci/nginx.conf
sed: can't find label for jump to `mp'
What's the point?
The 10th line of your full script is
sed -i "t" <replace-command-script> <input-file>
and sed correctly interpters the "t" as a "branch to" command, which throws an error.
Replace this line with the variant you have provided above, namely
sed -i -e '<replace command script>' <input-file>
This should fix the problem.
Related
I am trying to create a script to find a JSON element and update it with the arg values.
#!/bin/bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
sed -i '' '/location/c\ \"location\" : \"$placeLocation\",' $file
sed -i '' '/heading/c\ \"heading\" : \"$vehicleHeading\",' $file
sed -i '' '/message/c\ \"message\" : \"$message\",' $file
One.txt
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
But getting following error
sed: 1: "/location/c\ \"locati ...": extra characters after \ at the end of c command
sed: 1: "/heading/c\ \"heading ...": extra characters after \ at the end of c command
sed: 1: "/message/c\ \"message ...": extra characters after \ at the end of c command
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
sed: 1: "file.txt": invalid command code f
I have just started learning about sed editor and tried out multiple things but couldn't able to figure it out. Any help is much appreciated!
Note that you probably should consider using a tool like jq for editing JSON files. But I assume you have a good reason for using sed, so you have a couple of problems there.
The first is that you're trying to use GNU sed features on your Mac OS X version of sed that doesn't have those features. If you want GNU sed on Mac OS X, then install it:
▶ brew install gnu-sed
Fixing up your code for GNU sed (and also for other Bash style guide recommendations about quoting strings):
cat > FILE <<EOF
"location":"<48.777098,9.181301> - 150.0m",
"message":"Hello there!",
"heading": "34",
EOF
placeLocation=myPlaceLocation
vehicleHeading=myVehicleHeading
message=myMessage
file=FILE
gsed -i -e '/location/c\' -e '"location": "'"$placeLocation"'",' "$file"
gsed -i -e '/heading/c\' -e '"heading": "'"$vehicleHeading"'",' "$file"
gsed -i -e '/message/c\' -e '"message": "'"$message"'",' "$file"
As noted in the GNU sed manual, use of multiple -e commands on the same line with the c\ command is a GNU extension.
If you want to use Mac OS X's sed, just may be able to write it this way:
sed -i '' '
s/"location".*/"location": "'"$placeLocation"'",/
s/"heading".*/"heading": "'"$vehicleHeading"'",/
s/"message".*/"message": "'"$message"'",/
' "$file"
But note you would have to sanitise the input if you need the code to be robust to all inputs.
To do this robustly you need to use a tool that understands literal strings (which sed doesn't - see Is it possible to escape regex metacharacters reliably with sed) e.g. awk:
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1/2?' one.txt
"message":"what is 1/2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is 1&2?' one.txt
"message":"what is 1&2?"
$ awk 'BEGIN{FS=OFS=":"; val=ARGV[1]; ARGV[1]=""} $1=="\"message\""{sub(FS".*",FS); print $1, "\""val"\""}' 'what is \1?' one.txt
"message":"what is \1?"
the above will work robustly using any awk in any shell on every UNIX box. Try using those as replacement strings in a sed command.
The full script to do what you want would be:
#!/bin/env bash
# Shell script to verify the end to end D1 request flow
placeLocation=$1
vehicleHeading=$2
message=$3
file=one.txt
tmp=$(mktemp)
awk '
BEGIN {
split("location heading message", tags)
for (i in tags) {
vals["\"" tags[i] "\""] = "\"" ARGV[i] "\""
ARGV[i] = ""
}
FS=OFS=":"
}
$1 in vals {
tag = $1
sub(FS".*","")
$0 = tag OFS vals[tag]
}
1' "$placeLocation" "$vehicleHeading" "$message" "$file" > "$tmp" && mv "$tmp" "$file"
I am trying to run a django manage.py task via cron, on AWS Elasticbeanstalk (EB).
*/10 * * * * root /usr/local/bin/notification_cron.sh > /dev/null
The notification_cron.sh script calls a django manage.py task.
Django needs EB's environment variables (like RDS_PORT, RDS_DB_NAME, RDS_PASSWORD etc). So I am saving these environment variables into a file at deployment, and reloading those in the bash script that also calls the manage.py task.
This is part of my deployment config in .ebextensions:
commands:
001_envvars_to_bash_source_file:
command: |
# source our elastic beanstalk environment variables
/opt/elasticbeanstalk/bin/get-config --output YAML environment|perl -ne "/^\w/ or next; s/: /=/; print qq|\$_|" > /usr/local/bin/envvars
chmod 755 /usr/local/bin/envvars
files:
"/usr/local/bin/notification_cron.sh":
mode: "000755"
owner: root
group: root
content: |
#!/usr/bin/env bash
AWS_CONFIG_FILE="/home/ec2-user/.aws/config"
set -o allexport
# Loading environment data
source /usr/local/bin/envvars
set +o allexport
cd /opt/python/current/app/
source /opt/python/run/venv/bin/activate
python manage.py my_management_task
The problem is caused by the line
/opt/elasticbeanstalk/bin/get-config --output YAML environment|perl -ne "/^\w/ or next; s/: /=/; print qq|\$_|" > /usr/local/bin/envvars
or the alternate sed equivalent
/opt/elasticbeanstalk/bin/get-config environment --output yaml | sed -n '1!p' | sed -e 's/^\(.*\): /\1=/g' > /usr/local/bin/envvars
The contents of /usr/local/bin/envvars are not always in quotes:
PYTHONPATH="/opt/python/current/app/mydjangoapp:$PYTHONPATH"
DJANGO_SETTINGS_MODULE=mydjangoapp.settings
AWS_ACTIVE='true'
RDS_PORT='5432'
RDS_HOSTNAME=hostname.host.us-east-1.rds.amazonaws.com
RDS_USERNAME=master
RDS_DB_NAME=ebdb
RDS_PASSWORD=My&Password
This causes trouble where an environment variable has the "&" character.
'RDS_PASSWORD': My&Password
Bash splits this up at the "&" character when I source /usr/local/bin/envvars to import them back into my script before calling django's manage.py.
Phew. My question is:
How do I get RDS_PASSWORD="My&Password" in the /usr/local/bin/envvars file (note the double quotes required) without breaking the other lines like RDS_PORT='5432'?
Using GNU sed you can do something like below,
sed -r 's/RDS_PASSWORD=([[:graph:]]+)/RDS_PASSWORD="\1"/' /usr/local/bin/envvars
PYTHONPATH="/opt/python/current/app/mydjangoapp:$PYTHONPATH"
DJANGO_SETTINGS_MODULE=mydjangoapp.settings
AWS_ACTIVE='true'
RDS_PORT='5432'
RDS_HOSTNAME=hostname.host.us-east-1.rds.amazonaws.com
RDS_USERNAME=master
RDS_DB_NAME=ebdb
RDS_PASSWORD="My&Password"
You can then add the -i flag to sed to in-place substitution. I have used the character class [[:graph:]] which is
‘[:graph:]’
Graphical characters: ‘[:alnum:]’ and ‘[:punct:]’.
The special characters as part of [:punct:] are
! " # $ % & ' ( ) * + , - . / : ; < = > ? # [ \ ] ^ _ ` { | } ~
So the sed can handle the substitution if any of these above characters constitute the RDS_PASSWORD variable.
The syntax you can use, is
/opt/elasticbeanstalk/bin/get-config environment --output yaml | \
sed -n '1!p' | sed -e 's/^\(.*\): /\1=/g' | \
sed -r 's/RDS_PASSWORD=([[:graph:]]+)/RDS_PASSWORD="\1"/' > /usr/local/bin/envvars
I'm trying to use sed to change a variable in the site.js file on my server.
Here is the line: var url = "page.php"; I'm looking to just substitute page.php for whatever.php.
I thought this would be pretty simple and I figured this would work with no issues:
sed -i "s/\url = \".*\"/\url = \"page2.php\"/" /home/site.js
It works okay except instead of getting: var url = "page2.php"; I get: var R1 = "page2.php";
Why is the url value being changed to R1 when I use sed here?
You don't need \ before url.
sed -i -r 's#url\s*=\s*"[^"]+"#url = "page2.php"#' /home/site.js
Extra escaping of " can be eliminated by enclosing sed expression with ' instead of "
It's better to use different separator than / (here #) when the strings themselves may contain /
Try doing this :
sed -i -r 's#(var\s+url\s*=\s*")[^"]+"#\1whatever.php"#' file.js
/ is not mandatory as delimiter, I've picked up # there.
Here's another example: Took me while to figure that you change the / for delimiter and not the / in the directory path.
Use # instead of / for sed delimiter if you have dir path names.
First I tried this:
[root#ip-172-35-24-37 ec2-user]# egrep -q "^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$" /etc/fstab && sed -ri "s/^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$/\1/dev/shm\2nodev\3\4/" /etc/fstab
And got this error:
sed: -e expression #1, char 20: unknown option to `s'
So then I used # for the sed delimiter instead of /:
[root#ip-172-35-24-37 ec2-user]# egrep -q "^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$" /etc/fstab && sed -ri "s#^(\s*\S+\s+)/dev/shm(\s+\S+\s+\S+)(\s+\S+\s+\S+)(\s*#.*)?\s*$#\1/dev/shm\2nodev\3\4##" /etc/fstab
[root#ip-172-35-24-37 ec2-user]#
And it worked.
You can use something else besides # for a delimiter like ! or ? or %. Just don't use / if you have dir paths.
I adapted Jan Goyvaerts's e-mail regex to a bash function to be used in pipes to anonymize e-mail addresses:
function remove_emails {
sed -r "s|\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b|email.address#removed.com|gI";
}
which I'm using in a bash pipe:
mysqldump \
-uuser \
-ppass \
db_name \
| remove_emails \
| gzip -c \
| cat \
> tmp.sql.gz
works fine but now, I'd like to have different random e-mails, I'd be satisfied with:
email.address1#removed.com
email.address2#removed.com
or
eiyyzhupzftrvjwehbqp#removed.com
kwmbrshzmxqlrqatqpff#removed.com
or anything that differs and is unique
I'm quite comfortable with bash but using counters, process substitution and so fails as sed is invoked only once, so
sed "s,sth,$(echo $RANDOM),g"
and similar won't work,
Is there anything to generate random stuff or counters in sed itself?
This might work for you (GNU sed):
<<<'Here is a random number.' sed 's/random number/& $RANDOM/;s/.*/echo "&"/e'
or if you prefer:
<<<'Here is a random number.' sed 's/random number/& $RANDOM/;s/.*/echo "&"/' | sh
I experimented with potong's correct answer and found a way to implement an iterator which answers the other part of my question:
remove_emails() {
sed -r 's|\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b|test$(( iterator++ ))#example.com|gI;s|.*|echo "&"|' | bash
}
iterator=0
test_data='some.e.mail.address.#domain.com\nsome.other#email.co.uk\nwhatever#man.biz\nsed#sed.com\n'
echo -e "before:\n${test_data}"
echo -e "after: \n${test_data}" | remove_emails
You could do it by repeatedly invoking sed in a while loop as shown below:
remove_emails() {
while read line
do
sed -r "s|\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b|email.address${RANDOM}#removed.com|gI" <<< "$line"
done
}
I have the following log file which contains lines like this
1345447800561|FINE|blah#13|txReq
1345447800561|FINE|blah#13|Req
1345447800561|FINE|blah#13|rxReq
1345447800561|FINE|blah#14|txReq
1345447800561|FINE|blah#15|Req
I am trying extract the first field from each line and depending on whether it belongs to blah#13 or blah#14, blah#15 i am creating the corresponding files using the following script, which seems quite in-efficient in terms of the number of temp files creates. Any suggestions on how I can optimize it ?
cat newLog | grep -i "org.arl.unet.maca.blah#13" >> maca13
cat newLog | grep -i "org.arl.unet.maca.blah#14" >> maca14
cat newLog | grep -i "org.arl.unet.maca.blah#15" >> maca15
cat maca10 | grep -i "txReq" >> maca10TxFrameNtf_temp
exec<blah10TxFrameNtf_temp
while read line
do
echo $line | cut -d '|' -f 1 >>maca10TxFrameNtf
done
cat maca10 | grep -i "Req" >> maca10RxFrameNtf_temp
while read line
do
echo $line | cut -d '|' -f 1 >>maca10TxFrameNtf
done
rm -rf *_temp
Something like this ?
for m in org.arl.unet.maca.blah#13 org.arl.unet.maca.blah#14 org.arl.unet.maca.blah#15
do
grep -i "$m" newLog | grep "txReq" | cut -d' ' -f1 > log.$m
done
I've found it useful at times to use ex instead of grep/sed to modify text files in place without using temps ... saves the trouble of worrying about uniqueness and writability to the temp file and its directory etc. Plus it just seemed cleaner.
In ksh I would use a code block with the edit commands and just pipe that into ex ...
{
# Any edit command that would work at the colon prompt of a vi editor will work
# This one was just a text substitution that would replace all contents of the line
# at line number ${NUMBER} with the word DATABASE ... which strangely enough was
# necessary at one time lol
# The wq is the "write/quit" command as you would enter it at the vi colon prompt
# which are essentially ex commands.
print "${NUMBER}s/.*/DATABASE/"
print "wq"
} | ex filename > /dev/null 2>&1