I am fighting with sed to do a substitution where the substitute string contains slashes. This general topic has been discussed on stack overflow before. But, AFAICT, I have anew wrinkle that hasn't been addressed in previous questions.
Let's say I have a file, ENVIRO.tpml, which has several lines, one of which is
Loaded modules: SUPPLY_MODULES_HERE
I want to replace SUPPLY_MODULES_HERE in an automated fashion with a list of loaded modules. (At this point, if anyone has a better way to do this than sed, please let me know!) My first effort here is to define an environment variable and use sed to put it into the file:
> modules=$(module list 2>&1)
> sed "s/SUPPLY_MODULES_HERE/${modules}/" ENVIRO.tmpl > ENVIRO.txt
(The 2>&1 being needed because module list sends its output to STDERR, for reasons I can't begin to understand.) However, as is often the case, the modules have slashes in them. For example
> echo ${modules}
gcc/9.2.0 mpt/2.20
The slashes kill my command because sed can't understand the expression and thinks my substitution command is "unterminated".
So I do the usual thing and use some other character for the command delimiter:
> modules=$(module list 2>&1)
> sed "s|SUPPLY_MODULES_HERE|${modules}|" ENVIRO.tmpl > ENVIRO.txt
and I still get an "unterminated 's'" error.
So I replace double quotes with single quotes:
> sed 's|SUPPLY_MODULES_HERE|${modules}|' ENVIRO.tmpl > ENVIRO.txt
and now I get no error, but the line in ENVIRO.txt looks like
Loaded modules: ${modules}
Not what I was hoping for.
So, AFAICT, I need double quotes to expand the variable, but I need single quotes to make the alternative delimiters work. But I need both at the same time. How do I get this?
UPDATE: Gordon Davisson's comment below got to the root of the matter: "echo ${modules} can be highly misleading". Examining $modules with declare -p shows that it actually has a newline (or, more generally, some kind of line break) in it. What I did was add an extra step to extract newlines out of the variable. With that change, everything worked fine. An alternative would be to convince sed to expand the variable with line breaks and substitute it as such into the text, but I haven't been able to make that work. Any takers?
sed is not the best tool here due to use of regex and delimiters.
Better to use awk command that doesn't require any regular expression.
awk -v kw='SUPPLY_MODULES_HERE' -v repl="$(module list 2>&1)" '
n = index($0, kw) {
$0 = substr($0, 1, n-1) repl substr($0, n+length(kw))
} 1
' file
index function uses plain string search in awk.
substr function is used to get substring before and after the search keyword.
Related
I have 2 bash script variables defined:
THELINENUMBER="14" # an arbitrary line number, comes from a separate command
NEWLINE="a line/ with# special! characters<" # arbitrary line with special characters, comes from separate command
I need to use the line number ${THELINENUMBER} to replace a line in a file called after.txt with ${NEWLINE}.
How do I do that?
These are some examples I have tried:
sed -i '${THELINENUMBER}s#.*#"/"${NEWLINE}"/"' after.txt
sed -i "${THELINENUMBER}s#.*#"/"${NEWLINE}"/"" after.txt
sed -i "${THELINENUMBER}s/.*/'${NEWLINE}'" after.txt
sed -i '${THELINENUMBER}s,.*,${NEWLINE}' after.txt
I am told that the delimitter is usually a /, but those are present in my line replacement variable, so I can't use those. I tried with # and , but the desired behavior did not change. I am also told that " and ' are supposed to be used to turn off escaping in text (use literal string), but I have not been able to get that to work either. How do I pass in a string parameter into sed that has special characters? I am wondering if I should pass the variable ${NEWLINE} into another built-in function call to add escape characters or something before passing it into sed. Is sed the right tool for the job? I did not find much helpful information looking at the CLI manpages. I use Ubuntu 18.04.
I have referred to these sources in my internet search:
https://stackoverflow.com/questions/11145270/how-to-replace-an-entire-line-in-a-text-file-by-line-number
https://askubuntu.com/questions/76808/how-do-i-use-variables-in-a-sed-command
https://stackoverflow.com/questions/37372047/find-a-line-with-a-string-and-replace-entire-line-with-another-line
Use the c (change) command.
By the way, the naming convention for regular shell variables is NOT ALLCAPS, as that may result in accidental collisions with special variables like PATH.
sed "$linenumber c\\
$newline" file
Try
sed -i "${THELINENUMBER}s#.*#${NEWLINE}#" after.txt
this works because:
You require " enclosing the entire sed command instead of backtick so that the variables are expanded
No other quotes or backticks are needed to escape " in the variables as there aren't any: there are no literal (escaped) quotes inside the variables
An alternate separator (such as #) is required due to the / inside the NEWLINE variable.
I'm trying to continue to extract and isolate sections of text within my wordpress config file via bash script. Can someone help me figure out my sytax?
The lineof code in the wp-config.php file is:
$table_prefix = 'xyz_';
This is what I'm trying to use to extract the xyz_ portion.
prefix=$(sed -n "s/$table_prefix = *'[^']*'/p" wp-config.php)
echo -n "$prefix"
There's something wrong with my characters obviously. Any help would be much appreciated!
Your sed command is malformed. You can use s/regex/replacement/p to print your sed command. Yours, as written, will give unterminated 's' command. If you want to print your whole line out, you can use the capture group \0 to match it as s/<our_pattern>/\0/p
Bash interpets $table_prefix as a variable, and because it is in double quotes, it tries to expand it. Unless you set this variable to something, it expands to nothing. This would cause your sed command to match much more liberally, and we can fix it by escaping the $ as \$table_prefix.
Next, this won't actually match. Your line has multiple spaces before the =, so we need another wildcard there as in ...prefix *= *...
Lastly, to extract the xyz_ portion alone, we'll need to do some things. First, we have to make sure our pattern matches the whole line, so that when we substitute, the rest of the line won't be kept. We can do this by wrapping our pattern to match in ^.* ... .*\$. Next, we want to wrap the target section in a capture group. In sed, this is done with \(<stuff>\). The zeroth capture group is the whole line, and then capture groups are numbered in the order the parentheses appear. this means we can do \([^']*\) to grab that section, and \1 to output it:
All that gives us:
prefix=$(sed -n "s/^.*\$table_prefix *= *'\([^']*\)'.*\$/\1/p" wp-config.php)
The only issue with the regex is that the '$' character specifies that you are using a bash variable and since the pattern is wrapped in double quotes (", bash will attempt to expand the variable. You can mitigate this by either escapping the $ or wrapping the pattern in single quotes and escaping the single quotes in the pattern
Lastly, you are using the sed command s which stands for subsitute. It takes a pattern and replaces the matches with text in the form of s/<pattern>/<replace>/. You can omit the 's' and leave the 'p' or print command at the end. After all your command should look something like:
sed -n "/\$table_prefix = *'[^']*'/p" wp-config.php
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
So I have seen this question: Replace complete line getting number from variable
which is pretty similar, but in my case I am trying to use multiple variables: one for the line number [lineNo], one for the text to replace [transFormatted] and the file to which it should be looking in [OUTPUT_FILE] I've tried dozens of combinations to try to get it to recognize all these variables but nothing seems to work. It's unhappy no matter which way I try. What am I doing wrong?
sed -e '${lineNo}s/.*/${transFormatted}/' < $OUTPUT_FILE
Single quotes inhibit parameter expansion.
sed -e "${lineNo}s/.*/$transFormatted/" < "$OUTPUT_FILE"
You have to use double quotes for the variables to be expanded from the shell environment.
Currently, I have a fully functional sed command that adds an item to a list in a specific text file in the following way ...
ITEMS="$ITEM1 $ITEM2 $ITEM3"
becomes the following when we wish to insert $ITEM4 ...
ITEMS="$ITEM1 $ITEM2 $ITEM3 $ITEM4"
The number of items in the list is not known; it is dynamic. And the dollar signs and quotes in that text are to be taken literally.
I use the following command to accomplish the addition to the list (variable $itemNum is assigned elsewhere) ...
sed "/^\s*ITEMS=/s/\"$/ \$ITEM$itemNum&/" file.txt
All this does, of course, is find the line starting with ITEMS= (and possibly with leading spaces) and replace the last double quote in that line with a space plus the desired item plus a double quote (to put the replaced double quote back in place). However, I have an additional case where the ITEMS list could be empty like so ...
ITEMS=""
and in this case, my command would insert $ITEM4 at the end of the list like so ...
ITEMS=" $ITEM4"
However, I want my command to be able to account for an empty list and not put the leading space when the list is indeed empty, so it just looks like ...
ITEMS="$ITEM4"
How could I alter my existing command to best accomplish this?
Taking your subject as a guide, here is one way to do it conditionally with sed:
sed -e '/^\s*ITEMS=/{s/=""/="$ITEM'"$itemNum"'"/;t;s/"$/ $ITEM'"$itemNum&/;}" file.txt
The main change is we try to substitute empty quotes ("") first. If that succeeds, the conditional branch t sends us to the end of the script, bypassing the other substitution. The second substitution only occurs if the first fails. I've changed the quoting to get rid of some of the "leaning toothpicks", but it's about 66% your original command.
Edit: The "leaning toothpicks" thing by OP's request
The original put the entire sed script line in double quotes so all " and most $ had to be escaped with backslashes. Rewriting my version like that looks like this:
sed -e "/^\s*ITEMS=/{s/=\"\"/=\"\$ITEM$itemNum\"/;t;s/\"$/ \$ITEM$itemNum&/;}" file.txt
I just find all those backslashes distracting, so I put as much of it into single quotes as I could, leaving only $itemNum in double quotes. In a shell script, if two strings are adjacent, they get stuck together so var1=QRS; var2='ABC'$var1'XYZ' leaves var2 set to 'ABCQRSXYZ' - if you think there's a chance the variable could contain a space, then it is best to quote it: var1=QRS; var2='ABC'"$var1"'XYZ' (no space between '" and "' in there.
in this situation, awk is more flexible:
awk -F'"' -vn="$itemNum" '/^\s*ITEMS=/{sub(/"$/,($2?" ":"")"$ITEM"n"&")}1' file
test:
kent$ itemNum=4
kent$ echo 'ITEMS=""'|awk -F'"' -vn="$itemNum" '/^\s*ITEMS=/{sub(/"$/,($2?" ":"")"$ITEM"n"&")}1'
ITEMS="$ITEM4"
kent$ echo 'ITEMS="$ITEM1 $ITEM2 $ITEM3"'|awk -F'"' -vn="$itemNum" '/^\s*ITEMS=/{sub(/"$/,($2?" ":"")"$ITEM"n"&")}1'
ITEMS="$ITEM1 $ITEM2 $ITEM3 $ITEM4"