I want to replace a string from a file using sed in bash script, but that string is present at multiple places in that file.
Is there any way to replace the string using a WHERE clause so I can replace the string only where I want?
Using a line number won't work because I need a script that is more flexible than that allows. Here what I'm trying to do.
I stored the desired piece of code in a variable. Can I use that variable in a sed command? For example,
sed -i "s/condition: succeeded('Fair_PreProd')/condition: succeeded('Fair_UAT')/g" $folder_path/$file_name
Here is the original file:
-stage: Moto_Dev
dependsOn: Build
condition: and(succeeded(), eq(variables.isDevelop, true))
- stage: Unity_Dev
dependsOn: Build
condition: and(succeeded(), eq(variables.isUnityDevelop, true))
- stage: QA
dependsOn: Dev
condition: succeeded('Dev')
- stage: UAT
dependsOn: Build
condition: and(succeeded(), eq(variables.isStaging, true))
There are 3 places where dependsOn: Build is present. I want to replace only the one in the -stage:MotoDev section. How can I do that?
Is there any way to replace the string using a WHERE clause so I can replace the string only where I want?
sed does not have SQL-style WHERE clauses, but commands can have "addresses" that define subsets of input lines to operate upon. These can take several forms. Regular expressions are perhaps the most common, but there are also line numbers, and a couple of special forms. You can also have inclusive ranges built from simple addresses. An address range would be a reasonably good way to address the problem you present.
For example,
sed -i '/^\s*-\s*stage:\s*Moto_Dev/,/^\s*-/ s/dependsOn: Build/dependsOn: Test/' input
Explanation:
The -i command-line flag tells sed to work "in-place", which really means that it will replace the original file with one containing sed's output.
The /^\s*-\s*stage:\s*Moto_Dev/,/^\s*-/ is a range address, consisting of a regex for the range start (/^\s*-\s*stage:\s*MotoDev/) and one for the range end (/^\s*-/).
/^\s*-\s*stage:\s*Moto_Dev/ matches the beginning of the section in which you want the change to be made, with some flexibility around the exact amount of whitespace at certain positions. For brevity and clarity, it uses \s to represent a single space or tab character. That is a GNU extension, but if you cannot depend on GNU sed then there are other ways to express the same thing.
/^\s*-/ matches the beginning of the next section, as you have presented the input. It could be made more specific if it were necessary to be more selective.
The range includes its endpoints, but that does not appear to be a problem for the task at hand.
There is only one such range in the input presented, and that range contains the line you want to modify. The specified substitution, s/dependsOn: Build/dependsOn: Test/, is performed on each line in the range, but only the one contains a match to be replaced. All others in the range will be unaffected.
No commands at all are specified for lines outside the range, so they too will be unaffected.
You also asked,
I stored the desired piece of code in a variable. Can I use that
variable in a sed command? For example,
sed -i "s/condition: succeeded('Fair_PreProd')/condition: succeeded('Fair_UAT')/g" $folder_path/$file_name
sed does not expand shell-style parameter references, but you don't need it to do. The variable references in that command are expanded by the shell itself, before it executes the resulting command, so
yes, you may use them, and
it's not a question of using shell variables with sed in particular.
Suggesting an awk solution that reads bash variables.
filter : An awk RegExp filtering correct lines.
target : An awk RegExp to identify the string to be replaced
replacement: A string not RegExp to replace target RegExp.
With provided example:
awk '$0~filter && $0~target{gsub(target,replacement)}1' filter="block4" target="a=c" replacement="this = that" input.file
# same command, less readable, but shorter:
awk '$0~f && $0~t{gsub(t,r)}1' f="block4" t="a=c" r="this = that" input.file
Advantages of this approach:
More flexible than sed.
Generic: can add more filters and filtering logic.
Disadvantages of this approach:
Cannot do in-place replacement on the input file. Like sed -i
Therefore need to specify all files explicitly one by one.
sample output:
awk '$0~filter && $0~target{gsub(target,replacement)}1' filter="block4" target="a=c" replacement="this = that" input.1.txt
block1{ a=c }
block2{ v=c }
block3{ w=c }
block4{ this = that }
block5{ a=c }
Related
Suppose there's a text file with the following line:
export MYSQL_ADMIN=''
I want to insert text inside that single quote using the sed command, so that it changes to something like this for example:
export MYSQL_ADMIN='abc1'
What is the appropriate sed command for that in Linux?
I tried
sed -i -e ''/MYSQL_ADMIN/s/''/'abc1'/g"
but it didn't work.
Something like sed -i "s;export MYSQL_ADMIN=.*;export MYSQL_ADMIN='abc1';" /path/to/file.ext
-i modify file in place
s means substitute,
First block is what you are matching as an regular expression - the .* matches everything to the end of the line, this ensures you don't keep any text on that line after the substitue - and second block is what you are replacing with that match.
Always check the file after each run of sed if there is no error and check what changed.
To get the single quotes to print you may have to do ""'"" like ""'""abc1""'""
It is important to understand that although
I want to insert text inside that single quote using the sed command
is a perfectly good characterization of the effect you want to achieve, it does not map directly onto operations from sed's repertoire. With sed, the appropriate tool for most line modifications is the s command, which substitutes specified text for one or more matches to a specified regular expression. That would be the most natural thing to use for your case.
Additionally, it is important with sed to understand how and when to bind commands to specific lines. If you don't do that for a given command then it is applied to all lines. Sometimes that's fine, but other times it will produce unwanted results.
I tried
sed -i -e ''/MYSQL_ADMIN/s/''/'abc1'/g"
but it didn't work.
The two leading single quotes in that sed expression match each other, leaving the trailing double quote unmatched. Also, you do not specify the name of the file to modify. This variation would at least be valid shell syntax, and it would have the desired effect on the specified line appearing in file my_script:
sed -i -e "/MYSQL_ADMIN/s/''/'abc1'/g" my_script
That might also make other, unwanted changes, however.
You need to make some assumptions about the content of the file in order to do such a thing at all. The above depends on the text MYSQL_ADMIN and '' to appear on the same line only in the line(s) you want to modify. That may turn out to hold, but it seems unnecessarily risky. An assumption more likely to hold in general would be that there will be only one assignment to variable MYSQL_ADMIN, or that it is acceptable to modify all such assignments that assign a single-quote-delimited empty value.
Going with the latter, one might end up with this:
sed -i -e "s/\<MYSQL_ADMIN=''\(\s\|$\)/MYSQL_ADMIN='abc1'\1/g" my_script
The pattern \<MYSQL_ADMIN=''\(\s\|$\) improves on your plain MYSQL_ADMIN in these significant ways:
the \< causes it to match only immediately after a word boundary -- start of line, whitesepace, or punctuation. This prevents substitutions for other variables whose names happen to end with MYSQL_ADMIN. If you prefer, it would be even stronger to instead anchor the match to the beginning of the line with ^.
including the ='' in the pattern distinguishes between MYSQL_ADMIN and variables whose names contain that as an initial substring. It also ensures that the '' that gets replaced, if any, goes with the variable and does not merely appear somewhere else on the line.
the \(\s\|$\) both matches and captures either a whitespace character or the empty string at the end of a line. This distinguishes between assignments of an empty value and assignments of values that are merely prefixed by '' (which is valid if the file is a shell script). Having included it in the match, the capture allows the matched text, if any, to be preserved in the output (via the \1 in the replacement).
Because that matches the whole assignment, a complete assignment must appear in the replacement, too. On the other hand, this means that (probably) you can apply the command to every line, as shown, with no particular loss of efficiency relative to the previous command.
Even that might produce changes you didn't want, however, such as in comment lines or quoted text.
I did check the ABS, but it was hard to find a reference to my problem/question there.
Here it is. Consider the following code (Which extracts the first character of OtherVar and then converts MyVar to uppercase):
OtherVar=foobar
MyChar=${OtherVar:0:1} # get first character of OtherVar string variable
MyChar=${MyChar^} # first character to upper case
Could I somehow condense the second and third line into one statement?
P.S.: As was pointed out below, not needs to have a named variable. I should add, I would like to not add any sub-shells or so and would also accept a somehow hacky way to achieve the desired result.
P.P.S.: The question is purely educational.
You could do it all-in-one without forking sub-shell or running external command:
printf -v MyChar %1s "${OtherVar^}"
Or:
read -n1 MyChar <<<"${OtherVar^}"
Another option:
declare -u MyChar=${OtherVar:0:1}
But I can't see the point in such optimization in a bash script.
There are more suitable text processing interpreters, like awk, sed, even perl or python if performance matters.
You could use the cut command and put it in a complex expression to get it on one line, but I'm not sure it makes the code too much clearer:
OtherVar=foobar
MyChar=$(echo ${OtherVar^} | cut -c1-1) # uppercase first character and cut string
I'm trying to extract a tag value of an HTML node that I already have in a variable.
I'm currently using Zsh but I'm trying to make it work in Bash as well.
The current variable has the value:
<span class="alter" fill="#ffedf0" data-count="0" data-more="none"/>
and I would like to get the value of data-count (in this case 0, but could be any length integer).
I have tried using cut, sed and the variables expansion as explained in this question but I haven't managed to adapt the regexs, or maybe it has to be done differently for Zsh.
There is no reason why sed would not work in this situation. For your specific case, I would do something like this:
sed 's/.*data-count="\([0-9]*\)".*/\1/g' file_name.txt
Basically, it just states that sed is looking for the a pattern that contains data-count=, then saves everything within the paranthesis \(...\) into \1, which is subsequently printed in place of the match (full line due to the .*)
Could you please try following.
awk 'match($0,/data-count=[^ ]*/){print substr($0,RSTART+12,RLENGTH-13)}' Input_file
Explanation: Using match function of awk to match regex data-count=[^ ]* means match everything from data-count till a space comes, if this regex is TRUE(a match is found) then out of the box variables RSTART and RLENGTH will be set. Later I am printing current line's sub-string as per these variables values to get only value of data-count.
With sed could you please try following.
sed 's/.*data-count=\"\([^"]*\).*/\1/' Input_file
Explanation: Using sed's capability of group referencing and saving regex value in first group after data-count=\" which is its length, then since using s(substitution) with sed so mentioning 1 will replace all with \1(which is matched regex value in temporary memory, group referencing).
As was said before, to be on the safe side and handle any syntactically valid HTML tag, a parser would be strongly advised. But if you know in advance, what the general format of your HTML element will look like, the following hack might come handy:
Assume that your variable is called "html"
html='<span class="alter" fill="#ffedf0" data-count="0" data-more="none"/>'
First adapt it a bit:
htmlx="tag ${html%??}"
This will add the string tag in front and remove the final />
Now make an associative array:
declare -A fields
fields=( ${=$(tr = ' ' <<<$htmlx)} )
The tr turns the equal sign into a space and the ${= handles word splitting. You can now access the values of your attributes by, say,
echo $fields[data-count]
Note that this still has the surrounding double quotes. Yuo can easily remove them by
echo ${${fields[data-count]%?}#?}
Of course, once you do this hack, you have access to all attributes in the same way.
I have a simple sed script and I am replacing a bunch of lines in my application dynamically with a variable, the variable is a list of strings.My function works but does not keep the original indentation.the function deletes the line if it contains the certain string and replaces the line with a completely new line, I could not do a replace due to certain syntax restrictions.
How do I keep my original indentation when the line is replaced
Can I capitalize my variable and remove the underscore on the fly, i.e. the title is a capitalize and underscore removed version of the variableName, the list of items in the variable array is really long so I am trying to do this in one shot.
Ex: I want report_type -> Report Type done mid process
Is there a better way to solve this with sed? Thanks for any inputs much appreciated.
sed function is as follows
variableName=$1
sed -i "/name\=\"${variableName}\.name\" value\=model\.${variableName}\.name options\=\#lists\./c\\{\{\> \_dropdown title\=\"${variableName}\" required\=true name\=\"${variableName}\"\}\}" test
SAMPLE INPUT
{{> _select title="Report Type" required=true name="report_type.name" value=model.report_type.name options=#lists.report_type}}
SAMPLE EXPECTED OUPUT
{{> _dropdown title="Report Type" required=true name="report_type" value=model.report_type.name}}
sample input variable
report_type
Try this:
sed -E "s/^(\s+).*name\=\"(report_type)\.name\" value\=model\.report_type\.name options\=\#lists\..*$/\1\{\{\> \_dropdown title\=\"\2\" required\=true name\=\"\2\"\}\}/;T;s/\"(\w+)_(\w+)\"/\"\u\1 \u\2\"/g" input.txt > output.txt
I used "report_type" instead of ${variableName} for testing as an sed one-liner.
Please change back to ${variableName}.
Then go back to using -i (in addition to -E, which is for extended regex).
I am not sure whether I can do it without extended regex, let me know if that is necessary.
use s/// to replace fine tuned line
first capture group for the white space making the indentation
second capture group for the variable name
stop if that did not replace anything, T;
another s///
look for something consisting of only letters between "",
with a "_" between two parts,
seems safe enough because this step is only done on the already replaced line
replace by two parts, without "_"
\u for making camel case
Note:
Doing this on your sample input creates two very similar lines.
I assume that is intentional. Otherwise please provide desired output.
Using GNU sed version 4.2.1.
Interesting line of output:
{{> _dropdown title="Report Type" required=true name="Report Type"}}
I have a shell script where I have a statement:
isPartial = $searchCurl| grep -Po '\"partialSearch\":(true|false)'|sed 's/\\\"partialSearch\\\"://'
now, if I just echo the RHS
$searchCurl| grep -Po '\"partialSearch\":(true|false)'|sed 's/\\\"partialSearch\\\"://'
it prints "partialSearch":true, but the variable isPartial doesn't get initialized .
Why is this happening and how can I fix it ?
Since the number of backslashes in your examples varies, it is not clear to me if the double quotes are already escaped in the input text. I’ll assume they are not, i.e. the input text looks something like:
sometext... "partialSearch":true ... sometext...
..bla bla bla... "partialsearch":false ...
and my examples below will work under this assumption.
There are a number of points to be made.
You seem to be trying to parse JSON input with regular expressions. While this could be acceptable for quick-and-dirty one-time jobs where you know the exact format of the data being processed, in general it is a very bad idea. You should use a JSON parser like jq.
You obviously have stored some bash code in the variable searchCurl. This is considered bad practice. Instead of searchCurl="... code ..." you should do function searchCurl () { ... code ... } and call searchCurl without prefixing it with a dollar sign. Variables are for values, functions are for code.
In most cases, if you are going to use sed, it’s better to use it for everything without invoking grep. Sometimes it can be simpler to have both. See below for an example.
To assign the output of a command to a variable, you have to use command substitution.
In short, if in your input text you have only one match of '"partialSearch":(true|false)', this is what you want:
isPartial=$(searchCurl|sed -rn 's/^.*"partialSearch":(true|false).*$/\1/p')
If you have more and the input text is one big line as I suppose, usage of grep -o might simplify the task of splitting the input into one match per line, so that
isPartial=$(searchCurl|grep -Po '"partialSearch":(true|false)'|sed -e 's/^.*://')
might be what you want (and in this case, isPartial will hold a space-separated list of true and false).