I have a script to replace a specific email address in various files. The replacement address is the first parameter to the script:
#!/bin/bash
perl -pi -e s/'name\#domain\.org'/$1/ file-list
This doesn't work, as the # character in $1 is substituted by perl. Is there a straightforward fix for this? Running the script as subst foo\#bar.com, subst foo\\#bar.com, subst "foo#bar.com", and so on, doesn't work. Is there a sed script that could handle this more easily?
Instead of expanding a shell variable directly in the perl code you can pass it as an argument by setting the s switch:
#!/usr/bin/env bash
perl -i -spe 's/name\#domain\.org/$replacement/' -- -replacement="$1" file1.txt file2.txt
In perl's s///, without the use of e or ee modifiers, variables in the replacement part are treated as literals so you don't need to escape them.
This works, but needs you to pass the new mail address to the script with the # character preceded by \\:
#!/bin/bash
perl -pi -e "s/name\#domain.org/$1/" file-list
If the script is subst, run as:
subst newname\\#example.com
This is a better alternative, which uses sed to carry out the escaping:
#!/bin/bash
ADR="$(echo "$1" | sed -e 's/#/\\\#/')"
perl -pi -e "s/name\#domain.org/$ADR/" file-list
Of course, in this case it's probably better to use sed to do the whole thing.
Related
I have being trying to write a bash script that can search recursively in a directory and replace multiple strings e.g. #{DEMO_STRING_1} etc with an environment variable e.g. $sample1.
Full script:
#!/bin/sh
find /my/path/here -type f -name '*.js' -exec sed -i \
-e 's/#{DEMO_STRING_1}/'"$sample1"'/g' \
-e 's/#{DEMO_STRING_2}/'"$sample2"'/g' \
-e 's/#{DEMO_STRING_3}/'"$sample3"'/g' \
-e 's/#{DEMO_STRING_4}/'"$sample4"'/g' \
-e 's/#{DEMO_STRING_5}/'"$sample5"'/g' \
-e 's/#{DEMO_STRING_6}/'"$sample6"'/g' \
-e 's/#{DEMO_STRING_7}/'"$sample7"'/g' \
-e 's/#{DEMO_STRING_8}/'"$sample8"'/g' \
{} +
I can not figure out how to replace strings with hashtag with curly brackets.
I tried this example: sed find and replace with curly braces or Environment variable substitution in sed but I can not figure out how to combine them.
What I am missing? I searched also for characters that need to be escaped e.g. What characters do I need to escape when using sed in a sh script? but again not the characters that I need.
The specific format is throwing the following error:
sed: bad option in substitution expression
Where am I going so wrong?
Update: Sample of environment variables:
https://www.example.com
/sample string/
12345-abcd-54321-efgh
base64 string
All the cases above are environment variables that I would like to replace. All environment variables are within double quotes.
It is important to understand that the environment variable references are expanded by the shell, as it prepares to execute the command, not by the command itself (sed in this case). The command sees only the results of the expansions.
In your case, that means that if any of the environment variables' values contain characters that are meaningful to sed in context, such as unescaped (to sed) slashes (/), then sed will attribute special significance to them instead of interpreting them as ordinary characters. For example, given a sed command such as
sed -e "s/X/${var}/" <<EOF
Replacement: X
EOF
, if the value of $var is Y then the output will be
Replacement: Y
, but if the value of $var is /path/to/Y then sed will fail with the same error you report. This happens because the sed command actually run is the same as if you had typed
sed -e s/X//path/to/Y
, which contains an invalid s instruction. Probably the best alternative would be to escape the replacement-string characters that otherwise would be significant to sed. You can do that by interposing a shell function:
escape_replacement() {
# replace all \ characters in the first argument with double backslashes.
# Note that in order to do that here, we need to escape them from the shell
local temp=${1//\\/\\\\}
# Replace all & characters with \&
temp=${temp//&/\\&}
# Replace all / characters with \/, and write the result to standard out.
# Use printf instead of echo to avoid edge cases in which the value to print
# is interpreted to be or start with an option.
printf -- "%s" "${temp//\//\\/}"
}
Then the script would use it like this:
find /my/path/here -type f -name '*.js' -exec sed -i \
-e 's/#{DEMO_STRING_1}/'"$(escape_replacement "$sample1")"'/g' \
...
Note that you probably also want to use a shebang line that explicitly specifies a shell that supports substitution references (${parameter/pattern/replacement}), because these are not required by POSIX, and you might run into a system where /bin/sh is a shell that does not support them. If you're willing to rely on Bash then that should be reflected in your shebang line. Alternatively, you could prepare a version of the escape_replacement function that does not rely on substitution references.
If you use perl - you don't need to escape anything.
With your shell variable exported you can access it via $ENV{name} inside perl.
examples:
samples=(
https://www.example.com
'/sample string/'
12345-abcd-54321-efgh
'base64 string'
$'multi\nline'
)
for sample in "${samples[#]}"
do
echo '---'
export sample
echo 'A B #{DEMO_STRING_1} C' |
perl -pe 's/#{DEMO_STRING_1}/$ENV{sample}/g'
done
echo '---'
Output:
---
A B https://www.example.com C
---
A B /sample string/ C
---
A B 12345-abcd-54321-efgh C
---
A B base64 string C
---
A B multi
line C
---
To add the -i option you can: perl -pi -e 's///'
I have been trying to replace some text in a .txt file using a shell script and perl.
oldKey=123
trimmedNewKey=456
#Export shell variables, so that they can be used by perl
export oldKey
export trimmedNewKey
#Search and Replace
perl -pi -e 's/$ENV{oldKey}/$ENV{trimmedNewKey}/g' AppConstants.txt
This fails, but on the other hand, if I use string directly for the search param, it works:
perl -pi -e 's/123/$ENV{trimmedNewKey}/g' AppConstants.txt.
I am not a shell guy and can't figure out why my "search" param can't be evaluated via a variable. Help !
Try changing your code into
oldKey=123
trimmedNewKey=456
perl -pi -e "s/$oldKey/$trimmedNewKey/g" AppConstants.txt
This should work, because in a string enclosed in single quotes, there is no variable and wildcard expansion, as it is the case in a string enclosed in double quotes.
It is possible perhaps to pass some variables to perl so that the expansion happens inside the perl program, but why to make it complex when you can solve it in an easier way?
I am trying to find and replace a specific text content using the sed command and to run it via a shell script.
Below is the sample script that I am using:
fp=/asd/filename.txt
fd="sed -i -E 's ($2).* $2:$3 g' ${fp}"
eval $fd
and executing the same by passing the arguments:
./test.sh update asd asdfgh
But if the argument string contains $ , it breaks the commands and it is replacing with wrong values, like
./test.sh update asd $apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.
How can I make sure that the values inside the variables are not expanded because of the $?
Updated
sh file test.sh
set -xv
fp="/asd/filename.txt"
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
text file filename.txt
hello:world
Outputs
1)
./test.sh update hello WORLD
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:WORLD/g' /asd/filename.txt
2)
./test.sh update hello '$apr1$hosgaxyv$D0KXp5dCyZ2BUYCS9BmHu1'
sed -iE "s/(${2//'$'/'\$'}).*/${2//'$'/'\$'}:${3//'$'/'\$'}/g" "$fp"
++ sed -iE 's/(hello).*/hello:'\''$'\''apr1'\''$'\''hosgaxyv'\''$'\''D0KXp5dCyZ2BUYCS9BmHu1/g' /asd/filename.txt
In both the case , its not replacing the content
You don't need eval here at all:
fp=/asd/filename.txt
sed -i -E "s/(${2//'$'/'\$'}).*/\1:${3//'$'/'\$'}/g" "$fp"
The whole sed command is in double quotes so variables can expand.
I've replaced the blank as the s separator with / (doesn't really matter in the example).
I've used \1 to reference the first capture group instead of repeating the variable in the substitution.
Most importantly, I've used ${2//'$'/'\$'} instead of $2 (and similar for $3). This escapes every $ sign as \$; this is required because of the double quoting, or the $ get eaten by the shell before sed gets to see them.
When you call your script, you must escape any $ in the input, or the shell tries to expand them as variable names:
./test.sh update asd '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1.'
Put the command-line arguments that are filenames in single quotes:
./test.sh update 'asd' '$apr1$HnIF6bOt$9m3NzAwr.aG1Yp.t.bpIS1'
must protect all the script arguments with quotes if having space and special shell char, and escape it if it's a dollar $, and -Ei instead of -iE even better drop it first for test, may add it later if being really sure
I admit i won't understant your regex so let's just get in the gist of solution, no need eval;
fp=/asd/filename.txt
sed -Ei "s/($2).*/$2:$3/g" $fp
./test.sh update asd '\$apr1\$HnIF6bOt\$9m3NzAwr.aG1Yp.t.bpIS1.'
I am writing a shell script to replace a variable content which is an integer with another, using perl in a shell script.
#!/bin/sh
InitialFileStep="$1"
CurrentStage=$((($i*(9*4000))+$InitialFileStep))
PreviousStage=$((($(($(($i-1))*(9*4000)))) + (InitialFileStep)))
perl -pi -e 's/$PreviousStage/$CurrentStage/g' file.txt
echo "Hey!"
It seems it cannot find the variable content in the file.
I don't know what is the problem, is it the because the variables are integers and not strings?
The shell does not interpret anything inside single quote. '$ThisIsASimpleStringForTheShell' so try:
perl -pi -e 's/'"$PreviousStage"'/'"$CurrentStage"'/g' file.txt
The double quotes prevents possible spaces to mess up your command.
Mixing single and double quotes gives you the possibility to add regex operator to your command, preventing shell to interpret them before perl. This command substitute the contents of $PreviousStage with the contents of $CurrentStage only if the former is alone in a single line:
perl -pi -e 's/^'"$PreviousStage"'$/'"$CurrentStage"'/g' file.txt
The variables only exist in your shell script; you can't directly use them in your Perl script.
I hate attempting to generate Perl code from the shell as the previous solutions suggest. That way madness lies (though it works here since you're just dealing with integers). Instead, pass the values as arguments or some other way.
perl -i -pe'BEGIN { $S = shift; $R = shift; } s/\Q$S/$R/g' \
"$PreviousStage" "$CurrentStage" file.txt
or
export PreviousStage
export CurrentStage
perl -i -pe's/\Q$ENV{PreviousStage}/$ENV{CurrentStage}/g' file.txt
or
S="$PreviousStage" R="$CurrentStage" perl -i -pe's/\Q$ENV{S}/$ENV{R}/g' file.txt
In the below shell script I try to print A2D(Vlog-Ams-#Cross) with special characters escaped. For example replace ( with \( but sed won't have any effect.
#! /bin/sh
parameter="A2D(Vlog-Ams-#Cross)"
echo $parameter
parameterz=`echo "$parameter" | sed 's/(/\\(/g'`
echo $parameterz
The output is
A2D(Vlog-Ams-#Cross)
A2D(Vlog-Ams-#Cross)
If I do the same on my c-shell terminal, it works fine.
Any ideas?
You use backslashs within a backtick command and that's tricky. If the sed command didn't occur within backticks, it would work correctly. When the shell looks for the closing backtick, however, it removes one level of backslash quoting, so you get
sed 's/(/\(/g'
and that's a no-op. If your shell permits it, use $(...) instead of backticks; in this way you avoid these quoting problems.
In your replacement \\( the first \ escapes the second \. But you must escape the (, too:
$ echo 'A2D(Vlog-Ams-#Cross)' | sed -e 's/(/\\\(/g' -e 's/)/\\\)/g'
A2D\(Vlog-Ams-#Cross\)