Related
Suppose there's a text file with the following line:
export MYSQL_ADMIN=''
I want to insert text inside that single quote using the sed command, so that it changes to something like this for example:
export MYSQL_ADMIN='abc1'
What is the appropriate sed command for that in Linux?
I tried
sed -i -e ''/MYSQL_ADMIN/s/''/'abc1'/g"
but it didn't work.
Something like sed -i "s;export MYSQL_ADMIN=.*;export MYSQL_ADMIN='abc1';" /path/to/file.ext
-i modify file in place
s means substitute,
First block is what you are matching as an regular expression - the .* matches everything to the end of the line, this ensures you don't keep any text on that line after the substitue - and second block is what you are replacing with that match.
Always check the file after each run of sed if there is no error and check what changed.
To get the single quotes to print you may have to do ""'"" like ""'""abc1""'""
It is important to understand that although
I want to insert text inside that single quote using the sed command
is a perfectly good characterization of the effect you want to achieve, it does not map directly onto operations from sed's repertoire. With sed, the appropriate tool for most line modifications is the s command, which substitutes specified text for one or more matches to a specified regular expression. That would be the most natural thing to use for your case.
Additionally, it is important with sed to understand how and when to bind commands to specific lines. If you don't do that for a given command then it is applied to all lines. Sometimes that's fine, but other times it will produce unwanted results.
I tried
sed -i -e ''/MYSQL_ADMIN/s/''/'abc1'/g"
but it didn't work.
The two leading single quotes in that sed expression match each other, leaving the trailing double quote unmatched. Also, you do not specify the name of the file to modify. This variation would at least be valid shell syntax, and it would have the desired effect on the specified line appearing in file my_script:
sed -i -e "/MYSQL_ADMIN/s/''/'abc1'/g" my_script
That might also make other, unwanted changes, however.
You need to make some assumptions about the content of the file in order to do such a thing at all. The above depends on the text MYSQL_ADMIN and '' to appear on the same line only in the line(s) you want to modify. That may turn out to hold, but it seems unnecessarily risky. An assumption more likely to hold in general would be that there will be only one assignment to variable MYSQL_ADMIN, or that it is acceptable to modify all such assignments that assign a single-quote-delimited empty value.
Going with the latter, one might end up with this:
sed -i -e "s/\<MYSQL_ADMIN=''\(\s\|$\)/MYSQL_ADMIN='abc1'\1/g" my_script
The pattern \<MYSQL_ADMIN=''\(\s\|$\) improves on your plain MYSQL_ADMIN in these significant ways:
the \< causes it to match only immediately after a word boundary -- start of line, whitesepace, or punctuation. This prevents substitutions for other variables whose names happen to end with MYSQL_ADMIN. If you prefer, it would be even stronger to instead anchor the match to the beginning of the line with ^.
including the ='' in the pattern distinguishes between MYSQL_ADMIN and variables whose names contain that as an initial substring. It also ensures that the '' that gets replaced, if any, goes with the variable and does not merely appear somewhere else on the line.
the \(\s\|$\) both matches and captures either a whitespace character or the empty string at the end of a line. This distinguishes between assignments of an empty value and assignments of values that are merely prefixed by '' (which is valid if the file is a shell script). Having included it in the match, the capture allows the matched text, if any, to be preserved in the output (via the \1 in the replacement).
Because that matches the whole assignment, a complete assignment must appear in the replacement, too. On the other hand, this means that (probably) you can apply the command to every line, as shown, with no particular loss of efficiency relative to the previous command.
Even that might produce changes you didn't want, however, such as in comment lines or quoted text.
I have the following code:
script_list=$'1 a\n2\n3'
old_IFS="${IFS}"
IFS=$'\n'
for line in "${script_list}"; do
echo "__${line}__";
done
IFS="${old_IFS}"
which shall produce:
__1 a__
__2__
__3__
However, instead it gives me:
__1 a
2
3__
I double quoted "${script_list}" because it can contain spaces (see: https://stackoverflow.com/a/10067297). But I believe this is also where the problem lies, because when I remove the double quotes around it, it works as expected.
What am I missing?
edit:
As Cyrus suggested, I ran the code at ShellCheck and it tells me:
$ shellcheck myscript
Line 5:
for line in "${script_list}"; do
^-- SC2066: Since you double quoted this, it will not word split, and the loop will only run once.
Is it safe to simply remove the double quotes or do I need to be careful with that?
I double quoted "${script_list}" because it can contain spaces
This is only done to prevent the shell from splitting the string at spaces. However you explicitly tell the shell (by setting IFS) that your IFS is now a newline, not a space. The shell would split by default here on newlines, not on spaces, unless you quote it. Hence, remove the quotes.
I am trying to execute script with commands:
sed -i "USER/c\export USER=${signumid}" .bashrc
sed -i "DEVENVHOME=$/c\export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x}" .bashrc
I want to replace the line with string "USER" in .bashrc with export USER=${signumid} where $signumid variable is being provided through Cygwin prompt. Similarly I want to replace line with string DEVENVHOME=$ with export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x} in .bashrc file, where $signumid variable is provided through Cygwin prompt.
But I am getting following errors on Cygwin termminal.:
sed: -e expression #1, char 1: unknown command: `U'
sed: -e expression #1, char 3: extra characters after command
The general syntax of a sed script is a sequence of address command arguments statements (separated by newline or semicolon). The most common command is the s substitution command, with an empty address, so we can perhaps assume that that is what you want to use here. You seem to be attempting to interpolate a shell variable $signumid which adds a bit of a complication to this exposition.
If your strings were simply static text, it would make sense to use single quotes; then, the shell does not change the text within the quotes at all. The general syntax of the s command is s/regex/replacement/ where the slash as the argument separator is just a placeholder, as we shall soon see.
sed -i 's/.*USER.*/export USER=you/
s% DEVENVHOME=\$%export DEVENVHOME=${DEVENVHOME:-/home/you/CPM_WORKAREA/devenv.x}%' .bashrc
This will find any line with USER and substitute the entire line with export USER=you; then find any line which contains DEVENVHOME=$ (with a space before, and a literal dollar character) and replace the matched expression with the long string. Because the substitution string uses slashes internally, we use a different regex separator % -- alternatively, we could backslash-escape the slashes which are not separators, but as we shall see, that quickly becomes untenable when we add the following twist. Because the dollar sign has significance as the "end of line" metacharacter in regular expressions, we backslash-escape it.
I have ignored the c\ in your attempt on the assumption that it is simply a misunderstanding of sed syntax. If it is significant, what do you hope to accomplish with it? c\export is not a valid Bash command, so you probably mean something else, but I cannot guess what.
Now, to interpolate the value of the shell variable signumid into the replacement, we cannot use single quotes, because those inhibit interpolation. You have correctly attempted to use double quotes instead (in your edited question), but that means we have to make some additional changes. Inside double quotes, backslashes are processed by the shell, so we need to double all backslashes, or find alternative constructs. Fortunately for us, the only backslash is in \$ which can equivalently be expressed as [$], so let's switch to that notation instead. Also, where a literal dollar sign is wanted in the replacement string, we backslash-escape it in order to prevent the shell from processing it.
sed -i "s/.*USER.*/export USER=$signumid/
s% DEVENVHOME=[$]%export DEVENVHOME=\${DEVENVHOME:-/home/$signumid/CPM_WORKAREA/devenv.x}%" .bashrc
Equivalenty, you could use single quotes around the parts of the script which are meant to be untouched by the shell, and then put an adjacent double-quoted string around the parts which need interpolation, like
'un$touched*by$(the!shell)'"$signumid"'more$[complex]!stuff'
This final script still rests on a number of lucky or perhaps rather unlucky guesses about what you actually want. On the first line, I have changed just USER to a regular expression which matches the entire line -- maybe that's not what you want? On the other hand, the second line makes the opposite assumption, just so you can see the variations -- it only replaces the actual text we matched. Probably one or the other needs to be changed.
Finally, notice how the two separate sed commands have been conflated into a single script. Many newcomers do not realize that sed is a scripting language which accepts an arbitrary number of commands in a script, and simply treat it as a "replace" program with a funny syntax.
Another common source of confusion is the evaluation order. The shell processes the double-quoted string even before sed starts to execute, so if you have mistakes in the quoting, you can easily produce syntax errors in the sed script which lead to rather uninformative error messages (because what sed tells you in the error message is based on what the script looks like after the shell's substutions). For example, if signumid contains slashes, it will produce syntax errors, because sed will see those as terminating separators for the s/// command. An easy workaround is to switch to a separator which does not occur in the value of signumid.
I am new to writing in bash and I just finished this long script but I made the mistake of not adding quotation marks to all the variables beginning with the unary operator $. Adding all the quotation marks by hand is going to take a while. Is there a short cut I can use so all the words in the text file beginning with $ get quotation marks around them? So if a line in the file looks like:
python myProgram.py $car1 $car2 $speed1 $speed2
Then after the shortcut it will appear as
python myProgram.py "$car1" "$car2" "$speed1" "$speed2"
I am writing the script using nano.
Use global search and replace with the expression (\$\w+).
Switch to search and replace mode with C-\.
Switch to regex mode with Alt-R.
Type the expression (\$\w+). Hit Enter.
Type in the replacement expression "\1" replace the captured expression with quotations. Hit Enter.
On the match, hit A for All.
Given your need, it doesn't seem mandatory to provide a solution based on that editor. If you have access to a shell you might try this simple sed command:
sed -i.bak -r 's/\$\w+/"&"/g' my-script.sh
This is far from being perfect but should do the job in your particular case. If the above command:
-i.bak will perform the replacement "in place" -- that is modifying the original file, making a backup with the .bak extension
s/..../..../g is the usual sed command to search and replace using a pattern. The search pattern is between the first two \. The replacement is between the last two /
\$\w+ this pattern correspond to a $ followed by one or more letters (\w+). The backslash before $ is needed because that character normally has special meaning in a search pattern.
"&" is the replacement string. In there, the & is replaced by the string found in the search pattern. Broadly speaking this put quotes arround any string matching the search pattern.
I have a huge file, a really huge file (some 600+MB of text). In fact they are jsons. Each json is on a new line and only comes in a few flavours.
They look like:
{"text":{"some nested words":"Some more","something else":"Yeah more stuff","some list":["itemA","ItemB","itemEtc"]},"One last object":{"a thing":"and it's value"}}
And what I want is it go through with sed, suck out the text, and for each nexted pair put in some indent, so we get:
{
-{
--[]
-}
--{}
-}
}
(I'm not 100% sure I got the nesting right on the output, I think it's right)
Is this possible? I saw this, which was the closest I could imagine it being, but that gets rid of the brackets two.
I've noticed the answer there uses braching, so I think I need that, and I'll need to do some kind of s/pattern/newline+tab/space/g type command but I can't figure out how or what to make that...
Could someone help please? It needn't be pure sed but that is prefered.
This will not be pretty... =) Here is my solution as a sed script. Notice that it requires that the first line notifies the shell how to invoke sed to execute our script. As you can see, the "-n" flag is used so we force sed only to print what we explicitly command it to through the "p" or "P" commands. The "-f" option tells sed to read the commands from a file, with the name following the option. As the file name of the script is concatenated by the shell into the final command, it will properly read commands from the script (ie. if you run "./myscript.sed" the shell will execute "/bin/sed -nf myscript.sed").
#!/bin/sed -nf
s/[^][{}]//g
t loop
: loop
t dummy
: dummy
s/^\s*[[{]/&/
t open
s/^\s*[]}]/&\
/
t close
d
: open
s/^\(\s*\)[[]\s*[]]/\1[]\
/
s/^\(\s*\)[{]\s*[}]/\1{}\
/
t will_loop
b only_open
: will_loop
P
s/.*\n//
b loop
: only_open
s/^\s*[[{]/&\
/
P
s/.*\n//
s/[][{}]/ &/g
b loop
: close
s/ \([][{}]\)/\1/g
P
s/.*\n//
b loop
Before we start, we must first strip everything into brackets and square brackets. That's the responsibility of the first "s" command. It tells sed to replace every character that isn't a bracket or a square bracket with nothing, ie. remove it. Notice that the square brackets in the match represent a group of characters to match, but when the first character inside them is a "^", it will actually match any character except the ones specified after the "^". Because we want to match the closing square bracket and we need to close with a square bracket the group of characters to ignore, we tell that a closing square bracket should be included in the group by making it the first character following the "^". We can then specify the rest of the characters: opening square bracket, open bracket and close bracket (group of ignored characters: "][{}"), and then close the group with the closing square bracket. I tried to detail more here because this can be confusing.
Now for the actual logic. The algorithm is pretty simple:
while line isn't empty
if line starts with optional spaces followed by [ or {
if after the [ or { there are optional spaces followed by a respective ] or }
print the respective pair, with only the indentation spaces, followed by a newline
else
print the opening square or normal bracket, followed by a newline
remove what was printed from the pattern space (a.k.a. the buffer)
add a space before every open or close bracket (normal or square)
end-if
else
remove a space before every open or close bracket (normal or square)
print the closing square or normal bracket, followed by a newline
remove what was printed from the pattern space
end-if
end-while
But there are a couple of quirks. First of all, sed doesn't support a "while" loop or an "if" statement directly. The closest we can get to is the "b" and "t" commands. The "b" command branches (jumps) to a predefined label, similar to a C goto statement. The "t" also branches to a predefined label, but only if a substitution has happened since the start of the script running on the current line or since the last "t" command. Labels are written with the ":" command.
Because it is very likely that the first command actually performs at least one substitution, the first "t" command that follows it will cause a branch. Because we need to test for some other substitutions, we need to make sure that the next "t" command won't automatically succeed because of that first command. That is why we start with a "t" command to a line just above it (ie. if it branches or not, it will still continue at the same point), so we can "reset" the internal flag used by "t" commands.
Because the "loop" label will be branched to from at least one "b" command, it is possible that the same flag will be set when the "b" is executed, because only "t" commands can clear it. Therefore, we need to do the same workaround to reset the flag, this time by using a "dummy" label.
We now start the algorithm by checking for the presence of an open square bracket or an open close bracket. Because we only want to test for their presence, we must replace the match with itself, which is what "&" represents, and sed will automatically set the internal flag for the "t" command if the match succeeds. If the match succeeds, we use the "t" command to branch into into "open" label.
If it doesn't succeed, we need to see if we match a close square or normal the bracket. The command is nearly identical, but now we append a newline after the closing bracket. We do this by adding an escaped newline (ie. a backslash followed by an actual newline) after where we place the match (ie. after the "&"). Similarly to above, we use the "t" command to branch to the "close" label if the match succeeds. If it doesn't succeed, we will consider the line as invalid, and promptly empty the pattern space (buffer) and restart the script on the next line, all with the single "d" command.
Entering the "open" label, we will first handle the case of a pair of matching open and close brackets. If we do match them, we will print them with the indentation spaces preceding them, without any spaces between them, and ending with a newline. There is one specific command for each type of bracket pair (square or normal), but they are analogous. Because we have to keep track of how many indentation spaces there are we must store them in a special "variable". We do this by using the group capture, which will store the part of the match that starts after the "(" and ends before the ")". Therefore, we use it to capture the spaces after the start of the line and before the open bracket. We then proceed to match the open bracket followed by spaces and the respective close bracket. When we write the replacement, we make sure to reinsert the spaces by using the special variable "\1", which contains the data stored by the first group capture in the match. We then write the respective pair of open and close brackets and the escaped newline.
If we managed to do any of the replacements, we must print what we have just written, remove it from the pattern space and restart the loop with the remaining characters of the line. Because of this, we first branch with the "t" command to the "will_loop" label. Otherwise, we branch to the "only_open" label, which will handle the case of only an open bracket, without the consecutive respective close bracket.
Inside the "will_loop" label, we just print everything in the pattern space up to the first newline (which we manually added) with the "P" command. We then manually remove everything up to that first newline, so we can proceed with the rest of the line. This is similar to what the "D" command does, but without restarting the execution of the script. Finally we branch to the start of the loop again.
Inside the "only_open" label, we match an open bracket in a similar fashion as previously, but now we rewrite it appended with a newline. We then print that line and remove it from the pattern space. Now we replace all brackets (open or close, square or normal) with itself preceded by a single space character. This is so we can increment the indentation. Finally we branch to the beginning of the loop again.
The final label "close" will handle a closing bracket. We first remove every single space before a bracket, effectively decrementing the indentation. To do this, we need to use captures, because although we want to match the space and the bracket that follows, we only want to write back the bracket. Finally, print everything up to the newline that we manually added before entering the "close" label, remove what we printed from pattern space and restart the loop.
Some observations:
This doesn't check for the syntactic correctness of the code (ie. {{[}] would be accepted)
It will add and remove indentation as brackets are encountered, regardless of their type. This means that when it adds an indentation, it will remove it even if the encountered close bracket is not of the same type.
Hope this helps, and sorry for the long post =)
This might work for you (GNU sed):
sed 's/[^][{}]//g;s/./&\n/g;s/.$//' file |
sed -r '/[[{]/{G;s/(.)\n(.*)/\2\1/;x;s/^/\t/;x;b};x;s/.//;x;G;s/(.)\n(.*)/\2\1/' |
sed -r '$!N;s/((\{).*(\}))|((\[).*(\]))/\2\5\3\6/;P;D'
Explanation:
The first sed command produces a stream of curly/square brackets each on its own line
The second sed command indents each bracket
The third sed command reduces those paired brackets to a single line
If your happy with correctly indented brackets, the third command can be omitted.
I think you're expected output should look like:
{
-{
--[]
-}
-{
-}
}
Here's one way using GNU awk:
awk -f script.awk file.txt
Contents of script.awk:
BEGIN {
FS=""
flag = 0
}
{
for (i=1; i<=NF; i++) {
if ($i == "{" || $i == "[") {
flag = flag + 1
build_tree(flag, $i)
printf (flag <=2) ? "\n" : ""
}
if ($i == "}" || $i == "]") {
flag = flag - 1
printf (flag >= 2) ? $i : \
build_tree(flag + 1, $i); \
printf "\n"
}
}
}
function build_tree (num, brace) {
for (j=1; j<=num - 1; j++) {
printf "-"
}
printf brace
}
I know this is an ancient thread and nobody is looking anyway, but there's a simpler way now.
cat file.txt | jq '.' | sed 's/ /-/g' | tr -dc '[[]{}()]\n-' | sed '/^-*$/d'
There are 2 spaces in the first sed.