I have inherited a shell script. One of the things it does is parsing of a list of filenames. For every filename in the list it does following command:
fs_item="`echo ${fs_item%/}`"
This command (a part from doing it's job which in this case, I think, is to remove everything after last slash) replaces spaces in filename with one space:
in: aa bbbb ccc
out: aa bbbb ccc
From this point filename is broken.
So, the question is: can I somehow tell bash not to replace spaces?
Get rid of the backticks and the echo command. It is worse than useless in this situation because it adds nothing, and causes the problem you are trying to solve here.
fs_item="${fs_item%/}"
Is the echo really necessary? You could simply remove it:
fs_item="${fs_item%/}"
If your actual problem is something different, and you cannot get rid of the echo (or some other command invocation), adding some quotes should work:
fs_item="`echo \"${fs_item%/}\"`"
The spaces vanish when running the backticked echo command. The internal field separator includes the space character, so words separated by a sequence of one or more spaces will be passed as separate arguments to echo. Then, echo just prints it's arguments separated by a single space.
Since we're on the internal field separator subject, changing the IFS should also work (but usually has other possibly undesirable effects elsewhere in your script):
IFS=$'\n'
This sets the internal field separator to the newline character. After this, the spaces are no longer considered to be separators for lists. The echo command will receive just one argument (unless you have file names with the newline character in them) and spaces will stay intact.
Try setting IFS to something else, e.g. IFS=","
Related
#!/bin/bash
documents=("/datalog/AB errors.txt");
/usr/bin/bash /home/user/download.sh "${documents[*]}"
the download.sh SCPs into a server and downloads documents but in the case above there is a space in the file name which leads it to seek /datalog/AB instead of AB errors.txt file.
I thought surrounding in double quotes fixes the space in filename issue. I also tried AB\ errors.txt but that caused the entire bash script to not run.
The form
documents=("/datalog/AB errors.txt")
is bad form (turning string into an array !!!). The correct form for assignment is
documents="/datalog/AB errors.txt"
Always assume full-path variables can have spaces or other characters, and always double-quote such variable instantiations, i.e.
ls "${variable}" .
As Gordon Davisson said, you must code in this fashion at every level of scripting/programming to ensure the embedded special characters don't lead to unexpected results.
Suppose there's a text file with the following line:
export MYSQL_ADMIN=''
I want to insert text inside that single quote using the sed command, so that it changes to something like this for example:
export MYSQL_ADMIN='abc1'
What is the appropriate sed command for that in Linux?
I tried
sed -i -e ''/MYSQL_ADMIN/s/''/'abc1'/g"
but it didn't work.
Something like sed -i "s;export MYSQL_ADMIN=.*;export MYSQL_ADMIN='abc1';" /path/to/file.ext
-i modify file in place
s means substitute,
First block is what you are matching as an regular expression - the .* matches everything to the end of the line, this ensures you don't keep any text on that line after the substitue - and second block is what you are replacing with that match.
Always check the file after each run of sed if there is no error and check what changed.
To get the single quotes to print you may have to do ""'"" like ""'""abc1""'""
It is important to understand that although
I want to insert text inside that single quote using the sed command
is a perfectly good characterization of the effect you want to achieve, it does not map directly onto operations from sed's repertoire. With sed, the appropriate tool for most line modifications is the s command, which substitutes specified text for one or more matches to a specified regular expression. That would be the most natural thing to use for your case.
Additionally, it is important with sed to understand how and when to bind commands to specific lines. If you don't do that for a given command then it is applied to all lines. Sometimes that's fine, but other times it will produce unwanted results.
I tried
sed -i -e ''/MYSQL_ADMIN/s/''/'abc1'/g"
but it didn't work.
The two leading single quotes in that sed expression match each other, leaving the trailing double quote unmatched. Also, you do not specify the name of the file to modify. This variation would at least be valid shell syntax, and it would have the desired effect on the specified line appearing in file my_script:
sed -i -e "/MYSQL_ADMIN/s/''/'abc1'/g" my_script
That might also make other, unwanted changes, however.
You need to make some assumptions about the content of the file in order to do such a thing at all. The above depends on the text MYSQL_ADMIN and '' to appear on the same line only in the line(s) you want to modify. That may turn out to hold, but it seems unnecessarily risky. An assumption more likely to hold in general would be that there will be only one assignment to variable MYSQL_ADMIN, or that it is acceptable to modify all such assignments that assign a single-quote-delimited empty value.
Going with the latter, one might end up with this:
sed -i -e "s/\<MYSQL_ADMIN=''\(\s\|$\)/MYSQL_ADMIN='abc1'\1/g" my_script
The pattern \<MYSQL_ADMIN=''\(\s\|$\) improves on your plain MYSQL_ADMIN in these significant ways:
the \< causes it to match only immediately after a word boundary -- start of line, whitesepace, or punctuation. This prevents substitutions for other variables whose names happen to end with MYSQL_ADMIN. If you prefer, it would be even stronger to instead anchor the match to the beginning of the line with ^.
including the ='' in the pattern distinguishes between MYSQL_ADMIN and variables whose names contain that as an initial substring. It also ensures that the '' that gets replaced, if any, goes with the variable and does not merely appear somewhere else on the line.
the \(\s\|$\) both matches and captures either a whitespace character or the empty string at the end of a line. This distinguishes between assignments of an empty value and assignments of values that are merely prefixed by '' (which is valid if the file is a shell script). Having included it in the match, the capture allows the matched text, if any, to be preserved in the output (via the \1 in the replacement).
Because that matches the whole assignment, a complete assignment must appear in the replacement, too. On the other hand, this means that (probably) you can apply the command to every line, as shown, with no particular loss of efficiency relative to the previous command.
Even that might produce changes you didn't want, however, such as in comment lines or quoted text.
Essentially what I am trying to do is take a string with a bunch of text and if it has a substring of "$$" to replace it with a substring of "$$$"
ex:
string="abcde\$\$fghi"
# Modify string
echo $string
# ^ should give "abcde$$$fghi"
I have been at this for like 2 hours now and it seems like a very simple thing, so if anyone could provide some help then I would greatly appreciate it. Thanks!
EDIT: Changed original string in the question from "abcde$$fghi" to "abcde\$\$fghi"
$$ is a special variable in the shell, it contains the ID of the current process. The variables are expanded in double quotes, therefore string does not contain $$ but a number (the PID of shell) instead.
Enclose the string in apostrophes (single quotes) to get $$ inside it.
The replacement you need can be done in multiple ways. The simplest way (probably) and also the fastest way (for sure) is to use / in the parameter expansion of $string:
echo "${string/'$$'/'$$$'}"
To make it work you have to use the same trick as before: wrap $$ and $$$ in single quotes to prevent the shell replace them with something else. The quotes around the entire expression are needed to preserve the space characters contained by $string, otherwise the line is split to words by whitspaces and and echo outputs these words separated by one space character.
Check it online.
If you quote the string with single quote marks (i.e. string='abcde$$fghi') you can do the replacement with echo "${string/'$$'/'$$$'}"
Edit: this is basically what #axiac said in their comment
I have a tag
<string name="currencysym">$</string>
in string.xml And what to change the $ symbol dynamically. Use the below command but didn't work:
currencysym=₹
sed -i '' 's|<string name="currencysym">\(.*\)<\/string>|<string name="currencysym">'"<!\[CDATA\[${currencysym}\]\]>"'<\/string>|g'
Getting OUTPUT:
<string name="currencysym"><![CDATA[<string name="currencysym">$</string>#x20B9;]]></string>
" & " Has Removed...
But I need:
<string name="currencysym"><![CDATA[₹]]></string>
using xml-parser/tool to handle xml is first choice
instead of <..>\(.*\)<..> better use <..>\([^<]*\)<..> in case you have many tags in one line
& in replacement has special meaning, it indicates the whole match (\0) of the pattern. That's why you see <...>..</..> came to your output. If you want it to be literal, you should escape it -> \&
First problem is the line
currencysym=₹
This actually reads as "assign empty to currencysym and start no process in the background":
In bash you can set an environment variable (or variables) just or one run of a process by doing VAR=value command. This is how currencysym= is being interpreted.
The & symbol means start process in the background, except there is no command specified, so nothing happens.
Everything after # is interpreted as a comment, so #x20B9; is just whitespace from Bash's point of view.
Also, ; is a command separator, like &, which means "run in foreground". It is not used here because it is commented out by #.
You have to either escape &, # and ;, or just put your string into single quotes: currencysym=\&\#x20B9\; or currencysym='₹'.
Now on top of that, & has a special meaning in sed, so you will need to escape it before using it in the sed command. You can do this directly in the definition like currencysym=\\\&\#x20B9\; or currencysym='\₹', or you can do it in your call to sed using builtin bash functionality. Instead of accessing ${currencysym}, reference ${currencysym/&/\&}.
You should use double-quotes around variables in your sed command to ensure that your environment variables are expanded, but you should not double-quote exclamation marks without escaping them.
Finally, you do not need to capture the original currency symbol since you are going to replace it. You should make your pattern more specific though since the * quantifier is greedy and will go to the last closing tag on the line if there is more than one:
sed 's|<string name="currencysym">[^<]*</string>|<string name="currencysym"><![CDATA['"${currencysym/&/\&}"']]></string>|' test.xml
Yields
<string name="currencysym"><![CDATA[₹]]></string>
EDIT
As #fedorqui points out, you can use this example to show off correct use of capture groups. You could capture the parts that you want to repeat exactly (the tags), and place them back into the output as-is:
sed 's|\(<string name="currencysym">\)[^<]*\(</string>\)|\1<![CDATA['"${currencysym/&/\&}"']]>\2|' test.xml
sed -i '' 's|\(<string name="currencysym">\)[^<]*<|\1<![CDATA[\₹]]><|g' YourFile
the group you keep in buffer is the wrong one in your code, i keep first part not the &
grouping a .* is not the same as all untl first < you need. especially with the goption meaning several occurence could occur and in this case everithing between first string name and last is the middle part (your group).
carrefull with & alone (not escaped) that mean 'whole search pattern find' in replacement part
I am trying to execute script with commands:
sed -i "USER/c\export USER=${signumid}" .bashrc
sed -i "DEVENVHOME=$/c\export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x}" .bashrc
I want to replace the line with string "USER" in .bashrc with export USER=${signumid} where $signumid variable is being provided through Cygwin prompt. Similarly I want to replace line with string DEVENVHOME=$ with export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x} in .bashrc file, where $signumid variable is provided through Cygwin prompt.
But I am getting following errors on Cygwin termminal.:
sed: -e expression #1, char 1: unknown command: `U'
sed: -e expression #1, char 3: extra characters after command
The general syntax of a sed script is a sequence of address command arguments statements (separated by newline or semicolon). The most common command is the s substitution command, with an empty address, so we can perhaps assume that that is what you want to use here. You seem to be attempting to interpolate a shell variable $signumid which adds a bit of a complication to this exposition.
If your strings were simply static text, it would make sense to use single quotes; then, the shell does not change the text within the quotes at all. The general syntax of the s command is s/regex/replacement/ where the slash as the argument separator is just a placeholder, as we shall soon see.
sed -i 's/.*USER.*/export USER=you/
s% DEVENVHOME=\$%export DEVENVHOME=${DEVENVHOME:-/home/you/CPM_WORKAREA/devenv.x}%' .bashrc
This will find any line with USER and substitute the entire line with export USER=you; then find any line which contains DEVENVHOME=$ (with a space before, and a literal dollar character) and replace the matched expression with the long string. Because the substitution string uses slashes internally, we use a different regex separator % -- alternatively, we could backslash-escape the slashes which are not separators, but as we shall see, that quickly becomes untenable when we add the following twist. Because the dollar sign has significance as the "end of line" metacharacter in regular expressions, we backslash-escape it.
I have ignored the c\ in your attempt on the assumption that it is simply a misunderstanding of sed syntax. If it is significant, what do you hope to accomplish with it? c\export is not a valid Bash command, so you probably mean something else, but I cannot guess what.
Now, to interpolate the value of the shell variable signumid into the replacement, we cannot use single quotes, because those inhibit interpolation. You have correctly attempted to use double quotes instead (in your edited question), but that means we have to make some additional changes. Inside double quotes, backslashes are processed by the shell, so we need to double all backslashes, or find alternative constructs. Fortunately for us, the only backslash is in \$ which can equivalently be expressed as [$], so let's switch to that notation instead. Also, where a literal dollar sign is wanted in the replacement string, we backslash-escape it in order to prevent the shell from processing it.
sed -i "s/.*USER.*/export USER=$signumid/
s% DEVENVHOME=[$]%export DEVENVHOME=\${DEVENVHOME:-/home/$signumid/CPM_WORKAREA/devenv.x}%" .bashrc
Equivalenty, you could use single quotes around the parts of the script which are meant to be untouched by the shell, and then put an adjacent double-quoted string around the parts which need interpolation, like
'un$touched*by$(the!shell)'"$signumid"'more$[complex]!stuff'
This final script still rests on a number of lucky or perhaps rather unlucky guesses about what you actually want. On the first line, I have changed just USER to a regular expression which matches the entire line -- maybe that's not what you want? On the other hand, the second line makes the opposite assumption, just so you can see the variations -- it only replaces the actual text we matched. Probably one or the other needs to be changed.
Finally, notice how the two separate sed commands have been conflated into a single script. Many newcomers do not realize that sed is a scripting language which accepts an arbitrary number of commands in a script, and simply treat it as a "replace" program with a funny syntax.
Another common source of confusion is the evaluation order. The shell processes the double-quoted string even before sed starts to execute, so if you have mistakes in the quoting, you can easily produce syntax errors in the sed script which lead to rather uninformative error messages (because what sed tells you in the error message is based on what the script looks like after the shell's substutions). For example, if signumid contains slashes, it will produce syntax errors, because sed will see those as terminating separators for the s/// command. An easy workaround is to switch to a separator which does not occur in the value of signumid.