I have a bash script that lists subprograms/processes that could if the user chooses to, insert startupflags to a specific program. I want to match strings in the below formats and depending on which pgm the user chooses I want to insert/replace the string with the new flag infront of {PGMPATH}/pgm. The existing programs are listed in a startupfile according to something like this:
start -existingFlag ${PGMPATH}/pgm
start -existingFlag -anotherExistingFlag ${PGMPATH}/anotherPgm
start -existingFlag -anotherFlag ${PGMPATH}/yetAnotherPgm otherStuff
But to start with I try to match toward a hardcoded string (in the future toward the lines in the startup file):
start -existingFlag ${PGMPATH}\/pgm*
and replace it with a new line looking like this:
*start -existingFlag -newFlag ${PGMPATH}\/pgm*
From script:
existingString="start -existingFlag ${PGMPATH}\/pgm"
newString="start -existingFlag -newFlag ${PGMPATH}\/pgm"
sed 's/$replaceString/$newString/g' $STARTUPCONFFILE
This works (the string is replaced) as long as there is no '$' (just before {PGMPATH}) in the strings, but as soon as I add '$' as in ${PGMPATH} SED doesn't replace. I have tried a lot but I can't get it to work.
Suggestions?
You need double quotes for the shell to expand variables:
$ set newString=1
$ set replaceString=one
# using single quotes: no expansion -> no replacement!
$ echo one | sed 's/$replaceString/$newString/g'
one
# using double quotes: expansion -> replacement!
$ echo one | sed "s/$replaceString/$newString/g"
1
What does echo $PATH print? Are you aware that PATH is usually a colon separated list of directories? Is this really what you want? It would expand to, e.g.
start -existingFlag /usr/bin:/bin:/usr/local/bin/pgm
which is most certainly not what you expect. Maybe you have a variable name clash and should use another name than PATH.
I think, there will be slashes in $PGMPATH. They will interfere with sed syntax.
You can use other character, like | or % as separator, instead of usual /.
e.g. try:
sed "s|$replaceString|$newString|g"
Alternately, you can use \%regexp% syntax, given in sed manual here. (I have not used it myself though..)
Another alternate option, is to escape all the slashes in $PGMPATH, using another line of sed; but that would be more difficult.
Also, as pointed to by sudo_O, I have changed the single quote to double quotes, since the variables won't expand, when quoted with single quotes.
Related
I've written a bunch of environment variables in Docker format, but now I want to use them outside of that context. How can I source them with one line of bash?
Details
Docker run and compose have a convenient facility for importing a set of environment variables from a file. That file has a very literal format.
The value is used as is and not modified at all. For example if the value is surrounded by quotes (as is often the case of shell variables), the quotes are included in the value passed
Lines beginning with # are treated as comments and are ignored
Blank lines are also ignored.
"If no = is provided and that variable is…exported in your local environment," docker "passes it to the container"
Thankfully, whitespace before the = will cause the run to fail
so, for example, this env-file:
# This is a comment, with an = sign, just to mess with us
VAR1=value1
VAR2=value2
USER
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
#VAR7 =would fail
VAR8= but what about this?
VAR9="and this?"
results in these env variables in the container:
user=ubuntu
VAR1=value1
VAR2=value2
VAR3=is going to = trouble
VAR4=this $sign will mess with things
VAR5=var # with what looks like a comment
VAR8= but what about this?
VAR9="and this?"
The bright side is that once I know what I'm working with, it's pretty easy to predict the effect. What I see is what I get. But I don't think bash would be able to interpret this in the same way without a lot of changes. How can I put this square Docker peg into a round Bash hole?
tl;dr:
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
You're probably going to want to source a file, whose contents
are executed as if they were printed at the command line.
But what file? The raw docker env-file is inappropriate, because it won't export the assigned variables such that they can be used by child processes, and any of the input lines with spaces, quotes, and other special characters will have undesirable results.
Since you don't want to hand edit the file, you can use a stream editor to transform the lines to something more bash-friendly. I started out trying to solve this with one or two complex Perl 5 regular expressions, or some combination of tools, but I eventually settled on one sed command with one simple and two extended regular expressions:
sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list
This does a lot.
The first expression prepends export to any line whose first character is anything but #.
As discussed, this makes the variables available to anything else you run in this session, your whole point of being here.
The second expression simply inserts a single-quote after the first = in a line, if applicable.
This will always enclose the whole value, whereas a greedy match could lop off some of (e.g.) VAR3, for example
The third expression appends a second quote to any line that has at least one =.
it's important here to match on the = again so we don't create an unmatched quotation mark
Results:
# This is a comment, with an =' sign, just to mess with us'
export VAR1='value1'
export VAR2='value2'
export USER
export VAR3='is going to = trouble'
export VAR4='this $sign will mess with things'
export VAR5='var # with what looks like a comment'
#VAR7 ='would fail'
export VAR8=' but what about this?'
export VAR9='"and this?"'
Some more details:
By wrapping the values in single-quotes, you've
prevented bash from assuming that the words after the space are a command
appropriately brought the # and all succeeding characters into the VAR5
prevented the evaluation of $sign, which, if wrapped in double-quotes, bash would have interpreted as a variable
Finally, we'll take advantage of process substitution to pass this stream as a file to source, bring all of this down to one line of bash.
source <(sed -E -e "s/^([^#])/export \1/" -e "s/=/='/" -e "s/(=.*)$/\1'/" env.list)
Et voilà!
I have a tag
<string name="currencysym">$</string>
in string.xml And what to change the $ symbol dynamically. Use the below command but didn't work:
currencysym=₹
sed -i '' 's|<string name="currencysym">\(.*\)<\/string>|<string name="currencysym">'"<!\[CDATA\[${currencysym}\]\]>"'<\/string>|g'
Getting OUTPUT:
<string name="currencysym"><![CDATA[<string name="currencysym">$</string>#x20B9;]]></string>
" & " Has Removed...
But I need:
<string name="currencysym"><![CDATA[₹]]></string>
using xml-parser/tool to handle xml is first choice
instead of <..>\(.*\)<..> better use <..>\([^<]*\)<..> in case you have many tags in one line
& in replacement has special meaning, it indicates the whole match (\0) of the pattern. That's why you see <...>..</..> came to your output. If you want it to be literal, you should escape it -> \&
First problem is the line
currencysym=₹
This actually reads as "assign empty to currencysym and start no process in the background":
In bash you can set an environment variable (or variables) just or one run of a process by doing VAR=value command. This is how currencysym= is being interpreted.
The & symbol means start process in the background, except there is no command specified, so nothing happens.
Everything after # is interpreted as a comment, so #x20B9; is just whitespace from Bash's point of view.
Also, ; is a command separator, like &, which means "run in foreground". It is not used here because it is commented out by #.
You have to either escape &, # and ;, or just put your string into single quotes: currencysym=\&\#x20B9\; or currencysym='₹'.
Now on top of that, & has a special meaning in sed, so you will need to escape it before using it in the sed command. You can do this directly in the definition like currencysym=\\\&\#x20B9\; or currencysym='\₹', or you can do it in your call to sed using builtin bash functionality. Instead of accessing ${currencysym}, reference ${currencysym/&/\&}.
You should use double-quotes around variables in your sed command to ensure that your environment variables are expanded, but you should not double-quote exclamation marks without escaping them.
Finally, you do not need to capture the original currency symbol since you are going to replace it. You should make your pattern more specific though since the * quantifier is greedy and will go to the last closing tag on the line if there is more than one:
sed 's|<string name="currencysym">[^<]*</string>|<string name="currencysym"><![CDATA['"${currencysym/&/\&}"']]></string>|' test.xml
Yields
<string name="currencysym"><![CDATA[₹]]></string>
EDIT
As #fedorqui points out, you can use this example to show off correct use of capture groups. You could capture the parts that you want to repeat exactly (the tags), and place them back into the output as-is:
sed 's|\(<string name="currencysym">\)[^<]*\(</string>\)|\1<![CDATA['"${currencysym/&/\&}"']]>\2|' test.xml
sed -i '' 's|\(<string name="currencysym">\)[^<]*<|\1<![CDATA[\₹]]><|g' YourFile
the group you keep in buffer is the wrong one in your code, i keep first part not the &
grouping a .* is not the same as all untl first < you need. especially with the goption meaning several occurence could occur and in this case everithing between first string name and last is the middle part (your group).
carrefull with & alone (not escaped) that mean 'whole search pattern find' in replacement part
I am trying to execute script with commands:
sed -i "USER/c\export USER=${signumid}" .bashrc
sed -i "DEVENVHOME=$/c\export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x}" .bashrc
I want to replace the line with string "USER" in .bashrc with export USER=${signumid} where $signumid variable is being provided through Cygwin prompt. Similarly I want to replace line with string DEVENVHOME=$ with export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x} in .bashrc file, where $signumid variable is provided through Cygwin prompt.
But I am getting following errors on Cygwin termminal.:
sed: -e expression #1, char 1: unknown command: `U'
sed: -e expression #1, char 3: extra characters after command
The general syntax of a sed script is a sequence of address command arguments statements (separated by newline or semicolon). The most common command is the s substitution command, with an empty address, so we can perhaps assume that that is what you want to use here. You seem to be attempting to interpolate a shell variable $signumid which adds a bit of a complication to this exposition.
If your strings were simply static text, it would make sense to use single quotes; then, the shell does not change the text within the quotes at all. The general syntax of the s command is s/regex/replacement/ where the slash as the argument separator is just a placeholder, as we shall soon see.
sed -i 's/.*USER.*/export USER=you/
s% DEVENVHOME=\$%export DEVENVHOME=${DEVENVHOME:-/home/you/CPM_WORKAREA/devenv.x}%' .bashrc
This will find any line with USER and substitute the entire line with export USER=you; then find any line which contains DEVENVHOME=$ (with a space before, and a literal dollar character) and replace the matched expression with the long string. Because the substitution string uses slashes internally, we use a different regex separator % -- alternatively, we could backslash-escape the slashes which are not separators, but as we shall see, that quickly becomes untenable when we add the following twist. Because the dollar sign has significance as the "end of line" metacharacter in regular expressions, we backslash-escape it.
I have ignored the c\ in your attempt on the assumption that it is simply a misunderstanding of sed syntax. If it is significant, what do you hope to accomplish with it? c\export is not a valid Bash command, so you probably mean something else, but I cannot guess what.
Now, to interpolate the value of the shell variable signumid into the replacement, we cannot use single quotes, because those inhibit interpolation. You have correctly attempted to use double quotes instead (in your edited question), but that means we have to make some additional changes. Inside double quotes, backslashes are processed by the shell, so we need to double all backslashes, or find alternative constructs. Fortunately for us, the only backslash is in \$ which can equivalently be expressed as [$], so let's switch to that notation instead. Also, where a literal dollar sign is wanted in the replacement string, we backslash-escape it in order to prevent the shell from processing it.
sed -i "s/.*USER.*/export USER=$signumid/
s% DEVENVHOME=[$]%export DEVENVHOME=\${DEVENVHOME:-/home/$signumid/CPM_WORKAREA/devenv.x}%" .bashrc
Equivalenty, you could use single quotes around the parts of the script which are meant to be untouched by the shell, and then put an adjacent double-quoted string around the parts which need interpolation, like
'un$touched*by$(the!shell)'"$signumid"'more$[complex]!stuff'
This final script still rests on a number of lucky or perhaps rather unlucky guesses about what you actually want. On the first line, I have changed just USER to a regular expression which matches the entire line -- maybe that's not what you want? On the other hand, the second line makes the opposite assumption, just so you can see the variations -- it only replaces the actual text we matched. Probably one or the other needs to be changed.
Finally, notice how the two separate sed commands have been conflated into a single script. Many newcomers do not realize that sed is a scripting language which accepts an arbitrary number of commands in a script, and simply treat it as a "replace" program with a funny syntax.
Another common source of confusion is the evaluation order. The shell processes the double-quoted string even before sed starts to execute, so if you have mistakes in the quoting, you can easily produce syntax errors in the sed script which lead to rather uninformative error messages (because what sed tells you in the error message is based on what the script looks like after the shell's substutions). For example, if signumid contains slashes, it will produce syntax errors, because sed will see those as terminating separators for the s/// command. An easy workaround is to switch to a separator which does not occur in the value of signumid.
I have a variable with a path, like this:
SET "somevar=D:\tree\path\nonsens\oink.txt"
And I have a file, where somethink like this is written
VAR=moresonsense
Now I want to replace the word morenonsense to D:\tree\path\nonsens\oink.txt. This should be the result
VAR=D:\tree\path\nonsens\oink.txt
For this, I am using the tool sed for windows. But using sed in windows gives me the following:
VAR=D: ree/path/nonsens/oink.txt
The spaces between the colon and ree is a tab. I thought, I could fix it with the following line before calling sed:
SET "somevar=%somevar:\\=\\\\%"
But no, this line is not working. So I have some questions:
Is there a possibility, to prevent sed from changing \t to a tab and prevent changing two backslashed \ to a slash /?
Is there another easy way to replace a string with another string within a file with BATCH?
Does someone has another idea how to resolve this problem?
You should not \-escape the \ instances in the variable expansion; use the following:
SET "somevar=%somevar:\=\\%"
I don't know whether that solves all your problems, but SET "somevar=%somevar:\\=\\\\%" definitely does not work as intended, because it'll only match two consecutive \ chars in the input, resulting in a no-op with your input.
I want to do a search and replace on a line with specific line number. However, I want to be able to use a variable for the Line Number itself.
For instance, if I wanted to replace the number 4 with a number 5 on line 180. I would use the following code.
sed '180 s/4/5/' file
My Question is how do I use a variable for the line number?
sed '$variable s/4/5/' file
#gniourf_gniourf's comment contains the crucial pointer: use double quotes around your sed program in order to reference shell variables (the shell doesn't interpret (expand) single-quoted strings in any way).
Note that sed programs are their own world - they have NO concept of variables, so the only way to use variables is to use a double-quoted string evaluated by the shell containing references to shell variables.
As a result, you must \-escape characters that you want the shell to ignore and pass through to sed to see, notably $ as \$.
In your specific case, however, nothing needs escaping.
Thus, as #gniourf_gniourf states in his comment, use:
sed "$variable s/4/5/" file
Afterthought:
Alternatively, the core of your sed program can remain single-quoted, with only the shell-variable references spliced in as double-quoted strings; note that no spaces are allowed between the string components, as the entire expression must evaluate to a single string:
sed "$variable"' s/4/5/' file
While in this specific case you could get away without the double quotes around the variable reference, it's generally safer to use them, so as to avoid unwanted shell expansions (such as word splitting) that could alter or even break the command.
You could just leave the variable outside of the quotes
sed $variable's/4/5/' file
Note that there cannot be a space between the variable and beginning quote though
You can do it with awk
awk 'NR==l {sub(/4/,"5")}1' l="$variable" file