I am reading some configs from a text file and storing them into a variable. THey are pipe separated like this:
a,b|c,d|e,f|g,h
I want to pass these into a sed command to substitute the left value from the right value in a text file and output that to another file.
I have:
#!/bin/bash
test="a,b|c,d|e,f|j,l"
arr=(`echo $test | sed 's/|/\n/g'`)
sub=""
for i in "${arr[#]}"
do
echo "$i"
array=(`echo $i | sed 's/,/\n/g'`)
sub+="s/${array[0]}/${array[1]}/g; "
done
sed $sub ${FILE} > output.json
When I do this it runs a sed command for each element in arr. So after the first sed it says:
No such file or directory
Now if I don't pass in $sub as a variable and just put the entire string in it does it in one sed command and it works:
sed "s/a/b/g; s/c/d/g; s/e/f/g; s/g/h/g;" ${FILE} > output.json
How can I pass in my matchers as a variable like in my first example, but get it to do it all in one sed command like the second example?
How can I pass in my matchers as a variable like in my first example, but get it to do it all in one sed command like the second example?
Shellcheck will hint you:
Line 12:
sed $sub ${FILE} > output.json
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
Did you mean: (apply this, apply all SC2086)
sed "$sub" "${FILE}" > output.json
Because there is a space after ; the pattern is word splitted. Quote it to prevent expansion. When $sub is not quoted, the s/c/d/g; is parsed as a filename, so sed exits with no such file or directory.
Related
I have a list like this:
> echo $candidates
ENV-NONPROD-SANDBOX
ENV-NONPROD-SANDBOX-SECRETS
ENV-NONPROD-DEMO
ENV-NONPROD-DEMO-SECRETS
ENV-PROD-EU
ENV-PROD-EU-SECRETS
ENV-PROD-US
ENV-PROD-US-SECRETS
I also have a dynamically created list of expressions which I want to apply as filters (AND) to narrow that list to possible candidates:
$ filters=('/-SECRETS/!d' '/-NONPROD/!d') # static exanmple
Then I concatenate this and try to apply, but that does not work:
$ filterParam=$(printf "-e '%s' " "${filters[#]}")
$ echo $filterParam
-e "/-SECRETS/!d" -e "/-NONPROD/!d"
$ echo "$candidates" | sed $filterParam
sed: 1: " '/-SECRETS/\!d' ...": invalid command code '
The strange thing: If I execute it manually, it works!
> echo "$candidates" | sed -e "/-SECRETS/!d" -e "/-NONPROD/!d"
ENV-NONPROD-SANDBOX-SECRETS
ENV-NONPROD-DEMO-SECRETS
I execute this on macOS and zsh 5.8.1 (x86_64-apple-darwin21.0)
filterParam=$(printf "-e '%s' "
No, you can't store command line arguments in variables. Read https://mywiki.wooledge.org/BashFAQ/050 .
You can use bash arrays, which you already use to store filters, so just use them:
sedargs=()
for i in "${filters[#]}"; do
sedargs+=(-e "$i")
done
sed "${sedargs[#]}"
But sed is sed, just join array elements with a newline or a semicolon, which delimits sed expressions:
sed "$(printf "%s\n" "${filters[#]}")"
When you do a
sed $filterParam
in zsh, sed is invoked with one single argument, which is the content of the variable filterParam. sed does not know how to handle this.
If you would type the parameters explicitly, i.e.
sed -e "/-SECRETS/!d" -e "/-NONPROD/!d"
sed is invoked with four arguments, and this is what sed understands.
In bash, in the command
sed $filterParam
the value of filterParam would be split at the spaces and each "word" would be passed as a separate argument. In your concrete setting, this would make have sed receive 4 arguments.
If I run these commands from a script:
#my.sh
PWD=bla
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
xxx
bla
it is fine.
But, if I run:
#my.sh
sed 's/xxx/'$PWD'/'
...
$ ./my.sh
$ sed: -e expression #1, char 8: Unknown option to `s'
I read in tutorials that to substitute environment variables from shell you need to stop, and 'out quote' the $varname part so that it is not substituted directly, which is what I did, and which works only if the variable is defined immediately before.
How can I get sed to recognize a $var as an environment variable as it is defined in the shell?
Your two examples look identical, which makes problems hard to diagnose. Potential problems:
You may need double quotes, as in sed 's/xxx/'"$PWD"'/'
$PWD may contain a slash, in which case you need to find a character not contained in $PWD to use as a delimiter.
To nail both issues at once, perhaps
sed 's#xxx#'"$PWD"'#'
In addition to Norman Ramsey's answer, I'd like to add that you can double-quote the entire string (which may make the statement more readable and less error prone).
So if you want to search for 'foo' and replace it with the content of $BAR, you can enclose the sed command in double-quotes.
sed 's/foo/$BAR/g'
sed "s/foo/$BAR/g"
In the first, $BAR will not expand correctly while in the second $BAR will expand correctly.
Another easy alternative:
Since $PWD will usually contain a slash /, use | instead of / for the sed statement:
sed -e "s|xxx|$PWD|"
You can use other characters besides "/" in substitution:
sed "s#$1#$2#g" -i FILE
一. bad way: change delimiter
sed 's/xxx/'"$PWD"'/'
sed 's:xxx:'"$PWD"':'
sed 's#xxx#'"$PWD"'#'
maybe those not the final answer,
you can not known what character will occur in $PWD, / : OR #.
if delimiter char in $PWD, they will break the expression
the good way is replace(escape) the special character in $PWD.
二. good way: escape delimiter
for example:
try to replace URL as $url (has : / in content)
x.com:80/aa/bb/aa.js
in string $tmp
URL
A. use / as delimiter
escape / as \/ in var (before use in sed expression)
## step 1: try escape
echo ${url//\//\\/}
x.com:80\/aa\/bb\/aa.js #escape fine
echo ${url//\//\/}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//\//\/}"
x.com:80\/aa\/bb\/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s/URL/${url//\//\\/}/"
URL
echo $tmp | sed "s/URL/${url//\//\/}/"
URL
OR
B. use : as delimiter (more readable than /)
escape : as \: in var (before use in sed expression)
## step 1: try escape
echo ${url//:/\:}
x.com:80/aa/bb/aa.js #escape not success
echo "${url//:/\:}"
x.com\:80/aa/bb/aa.js #escape fine, notice `"`
## step 2: do sed
echo $tmp | sed "s:URL:${url//:/\:}:g"
x.com:80/aa/bb/aa.js
With your question edit, I see your problem. Let's say the current directory is /home/yourname ... in this case, your command below:
sed 's/xxx/'$PWD'/'
will be expanded to
sed `s/xxx//home/yourname//
which is not valid. You need to put a \ character in front of each / in your $PWD if you want to do this.
Actually, the simplest thing (in GNU sed, at least) is to use a different separator for the sed substitution (s) command. So, instead of s/pattern/'$mypath'/ being expanded to s/pattern//my/path/, which will of course confuse the s command, use s!pattern!'$mypath'!, which will be expanded to s!pattern!/my/path!. I’ve used the bang (!) character (or use anything you like) which avoids the usual, but-by-no-means-your-only-choice forward slash as the separator.
Dealing with VARIABLES within sed
[root#gislab00207 ldom]# echo domainname: None > /tmp/1.txt
[root#gislab00207 ldom]# cat /tmp/1.txt
domainname: None
[root#gislab00207 ldom]# echo ${DOMAIN_NAME}
dcsw-79-98vm.us.oracle.com
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: ${DOMAIN_NAME}/g'
--- Below is the result -- very funny.
domainname: ${DOMAIN_NAME}
--- You need to single quote your variable like this ...
[root#gislab00207 ldom]# cat /tmp/1.txt | sed -e 's/domainname: None/domainname: '${DOMAIN_NAME}'/g'
--- The right result is below
domainname: dcsw-79-98vm.us.oracle.com
VAR=8675309
echo "abcde:jhdfj$jhbsfiy/.hghi$jh:12345:dgve::" |\
sed 's/:[0-9]*:/:'$VAR':/1'
where VAR contains what you want to replace the field with
I had similar problem, I had a list and I have to build a SQL script based on template (that contained #INPUT# as element to replace):
for i in LIST
do
awk "sub(/\#INPUT\#/,\"${i}\");" template.sql >> output
done
If your replacement string may contain other sed control characters, then a two-step substitution (first escaping the replacement string) may be what you want:
PWD='/a\1&b$_' # these are problematic for sed
PWD_ESC=$(printf '%s\n' "$PWD" | sed -e 's/[\/&]/\\&/g')
echo 'xxx' | sed "s/xxx/$PWD_ESC/" # now this works as expected
for me to replace some text against the value of an environment variable in a file with sed works only with quota as the following:
sed -i 's/original_value/'"$MY_ENVIRNONMENT_VARIABLE"'/g' myfile.txt
BUT when the value of MY_ENVIRONMENT_VARIABLE contains a URL (ie https://andreas.gr) then the above was not working.
THEN use different delimiter:
sed -i "s|original_value|$MY_ENVIRNONMENT_VARIABLE|g" myfile.txt
I have a list of substitutions that I would like to perform with sed. Instead of combining the substitutions into a single sed command, I would like to perform each substitution in an iteration of a bash loop. For example:
cat ${input} |
for subst in "${substlist}"; do
sed 's/'${subst}'/modified_'${subst}'/g'
done > ${output}
I would expect that each iteration modifies the entire stream but I'm only seeing that the first iteration gets the input.
Is this pattern possible in bash?
Create an array of -e options to pass to sed.
filters=()
for subst in ${substlist}; do
filters+=(-e "s/$subst/modified_$subst/")
done
sed "${filters[#]}" "$input" > "$output"
(The question of iterating over an unquoted parameter expansion and dynamically creating each sed filter is beyond the scope of this answer.)
Here is one way to do it as a single stream. Building up the sed arguments from ${substlist} and calling sed once:
#!/bin/sh
cat ${input} |
sed `for subst in ${substlist}; do
echo " -e s/${subst}/modified_${subst}/g "
done` > ${output}
Depending on what is in ${substlist} you may need to do additional escaping.
Copy the input file to the output file, then perform the sed substitutions using the -i option to keep overwriting that file.
cp "$input" "$output"
for subst in $substlist
do
sed -i "s/$subst/modified_$subst/g" "$output"
done
With BSD/OSX sed that needs to be:
sed -i '' "s/$subst/modified_$subst/g" "$output"
I have a file pattern.txt which is composed of one very long line of complicated code (~8200 chars).
This code can be found in multiple files inside multiple directories.
I can easily identify a list of these files using
grep -rli 'uniquepartofthecode' *
My concern is how do I replace it with the exact text from within the file ?
I tried to do:
var=$(cat pattern.txt)
sed -i "s/$var//g" targetfile.txt
but I got the following error :
sed: -e expression #1, char 96: unknown option to `s'
sed is interpreting my $var content as a regular expression, I would like it to just match the exact text.
The pattern.txt content could be more or less any combination of characters so I'm afraid I cannot escape every characters efficiently.
Is there a solution using sed ? Or should I use another tool for that ?
EDIT:
I tried using this solution to make a proper regex pattern from my text file.
Is it possible to escape regex metacharacters reliably with sed
the overall process is:
var=$(cat pattern.txt)
searchEscaped=$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$var")
sed -n "s/$searchEscaped/foo/p" <<<"$var" # if ok, echoes 'foo'
This last command displays "foo". $searchEscaped seems to be properly escaped.
Though, this is not returning anything (it should display foo + the rest of the file without the matched part):
sed -n "s/$searchEscaped/foo/p" targetfile.txt
I think that the best solution is to not use regular expressions at all and resort to string replacement.
One way to do this is using perl:
$ echo "$string_to_replace"
some other stuff abc$^%!# some more
$ echo "$search"
abc$^%!#
$ perl -spe '$len = length $search;
while (($pos = index($_, $search, $n)) > -1) {
substr($_, $pos, $len) = "replacement";
$n = $pos + $len;
}' <<<"$string_to_replace" -- -search="$search"
some other stuff replacement some more
The -p switch tells perl to loop through each line of the variable $string_to_replace (which could easily be replaced by a file). -s allows options to be passed to the script - in this case, I've passed a shell variable containing the search string.
For each line of the file, the while loop runs through all of the matches of the search string. substr is used on the left hand of the assignment to replace a substring of $_, which refers to the current line being processed.
I am having trouble using sed to substitute values and write to a new file. It writes to a new file, but fails to change any values. Here is my code:
cd/mydirectory
echo "Enter file name:"
read file_input
file1= "$file_input"
file1= "$file1.b"
file2= "$file_input"
file2= "${file2}Ins.b"
sed "/\!cats!/s/\!cats!.*/cats!300!/g $file1>$file2
I simply want to substitute whatever text was after cats with the value 300. Whenever I run this script it doesn't overwrite the previous value with 300. Any suggestions?
Try changing
sed "/\!cats!/s/\!cats!.*/cats!300!/g $file1>$file2
to
sed "s/cats.*/cats300/g" $file1 > $file2
To replace text, you often have to use sed like sed "s/foo/bar/g" file_in > file_out, to change all occurrences of foo with bar in file_in, redirecting the output to file_out.
Edit
I noticed that you are redirecting the output to the same file - you can't do that. You have 2 options:
Redirect the results to another file, with a different filename. e.g.:
sed "s/cats.*/cats300/g" $file1 > $file2.tmp
Note the .tmp after $file2
Use the -i flag (if using GNU sed):
sed -i "s/cats.*/cats300/g" $file1
The i stands for inline replacement.
I think this modified version of your script should work:
echo "Enter file name:"
read file_input
file1="$file_input" # No space after '='
file1="$file1.b" # No space after '='
file2="$file_input" # No space after '='
file2="${file2}Ins.b" # No space after '='
sed 's/!cats!.*/!cats!300!/g' "$file1" > "$file2"
Note the single quotes around sed expression: with them, there's no need to escape the !s in your expression. Note also the double quotes around "$file1" and "$file2": if one of those variables contain spaces, this will prevent your command from breaking.
Some further remarks:
As pointed by jim, you may want to use the GNU sed -i option.
Your regex will currently replace everything after !cats! in matching lines. If they were several occurences of !cats! on your line, only one will remain. If instead you just want to replace the value between two ! delimiters, you may consider use following sed command instead:
sed 's/!cats![^!]*/!cats!300/g'