I have a file like this
parameters:
database_host: 127.0.0.1
database_port: null
database_name: myDb
database_user: root
database_password: null
mailer_transport: smtp
mailer_host: 127.0.0.1
mailer_user: null
mailer_password: null
secret: ea64f518be08e0d5895335990f10d984c22f400c
name_db: newDB
This is a yml file, and it has a particular format
I Want to create a file sh that find and replace the value after name_db with a parameter
Look for the key (name_db:) and replace the value after it. Assuming a sed that supports the -i option:
sed -i.bak "/^[[:space:]]*name_db:/ s/:.*/: $newname/" file.yml
The regular expression /^[[:space:]]*name_db:/ looks for a line that starts with zero or more spaces followed by name_db:. When that line is found, the s/:.*/: $newname/ substitute is executed. It will replace the colon and whatever follows with colon, space, and the value in $newname. This will fail if $newname contains a slash. If that's a possibility, choose a different character (other than /) as the marker. In case of doubt, Control-A is quite useful and unlikely to be part of a valid name_db value.
This doesn't care what the old value is. You can simply wrap this whole expression in double quotes, which interpolates the value in $newname. Nevertheless, using single quotes around sed scripts is generally a good idea.
monkeyUser commented:
sed: 1: "file.yml": extra characters at the end of p command
Note that the command line shown (using -i.bak) works on both Linux (GNU sed) and Mac OS X (BSD sed). GNU sed allows an optional backup suffix which must be attached to the -i option if it is present; BSD sed requires a suffix which must either be attached to the -i option as shown or can be the next argument. If you want no backup with GNU sed, specify just -i …; with BSD sed, specify '' as an empty argument after the -i option: -i '' …. Given the error message sed: 1: "file.yml": extra characters at the end of p command, I'm suspicious that the code was run on a system with BSD sed, though I'm not quite sure why p was mentioned (I suspect some editing of the actual error message). That is: sed -i /something/ file.yml would, with BSD sed, treat /something/ as the backup file suffix and then find problems with file.yml treated as a sed script…except I think the file name began with p rather than f, since f isn't a sed command (but p is).
Related
I want to remove the first two characters of a column in a text file.
I am using the below but this is also truncating the headers.
sed -i 's/^..//' file1.txt
Below is my file:
FileName,Age
./Acct_Bal_Tgt.txt,7229
./IDQ_HB1.txt,5367
./IDQ_HB_LOGC.txt,5367
./IDQ_HB.txt,5367
./IGC_IDQ.txt,5448
./JobSchedule.txt,3851
I want the ./ to be removed from each line in the file name.
Transferring comments to an answer, as requested.
Modify your script to:
sed -e '2,$s/^..//' file1.txt
The 2,$ prefix limits the change to lines 2 to the end of the file, leaving line 1 unchanged.
An alternative is to remove . and / as the first two characters on a line:
sed -e 's%^[.]/%%' file1.txt
I tend to use -e to specify that the script option follows; it isn't necessary unless you split the script over several arguments (so it isn't necessary here where there's just one argument for the script). You could use \. instead of [.]; I'm allergic to backslashes (as you would be if you ever spent time working out whether you needed 8 or 16 consecutive backslashes to get the right result in a troff document).
Advice: Don't use the -i option until you've got your script working correctly. It overwrites your file with the incorrect output just as happily as it will with the correct output. Consequently, if you're asking about how to write a sed script on SO, it isn't safe to be using the -i option. Also note that the -i option is non-standard and behaves differently with different versions of sed (when it is supported at all). Specifically, on macOS, the BSD sed requires a suffix specified; if you don't want a backup, you have to use two arguments: -i ''.
Use this Perl one-liner:
perl -pe 's{^[.]/}{}' file1.txt > output.txt
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-p : Loop over the input one line at a time, assigning it to $_ by default. Add print $_ after each loop iteration.
s{^[.]/}{} : Replace a literal dot ([.]) followed by a slash ('/'), found at the beginning of the line (^), with nothing (delete them). This does not modify the header since it does not match the regex.
If you prefer to modify the file in-place, you can use this:
perl -i.bak -pe 's{^[.]/}{}' file1.txt
This creates the backup file file1.txt.bak.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc perlrequick: Perl regular expressions quick start
Using file structure
foo_11: "Марія"
foo_112: "Superman"
FOOTLONG: "Subway"
foo_13: "Юлія"
I want to remove all strings that don't have at least one character from Ukrainian alphabet.
Script:
for i in *.txt;
do
sed '/[^А-ЯЄЇІа-яєїі]+/d' $i >$i.out
mv $i.out $i
done
doesn't do anything. What is wrong?
Using mac bash.
Assuming that your character class defining Ukrainian letters is correct, the following should work:
sed '/[А-ЯЄЇІа-яєїі]/!d' file
[А-ЯЄЇІа-яєїі] matches a Ukrainian letter anywhere on the line.
Note that even the letters that look like ASCII letters A I a i are actually Ukrainian (Cyrillic) letters with Unicode codepoints U+410 U+406 U+430 U+456.
! negates the match, meaning that only lines not containing at least 1 Ukrainian letter match.
d deletes those lines.
To put it all together:
for f in *.txt; do
sed -i '' '/[А-ЯЄЇІа-яєїі]/!d' "$f" # -i '' is BSD Sed syntax; GNU sed takes just -i
done
As for what you've tried:
As #StefanHegny points out in a comment on the question, + isn't supported when sed is not run with -E in order to enable extended regular expressions; without -E, the cumbersome \{1,\} must be used. (\+ is only supported by GNU sed, not by the BSD version of sed that macOS comes with).
However, even the fixed version of your command, sed '/[^А-ЯЄЇІа-яєїі]\{1,\}/d', doesn't do what you want: it deletes all lines that contain at least one non-Ukrainian-letter character, which eliminates all of your input lines, given that they all have ASCII-based field names and contain :.
You should double-quote variable references such as $i to protect them from shell expansions: "$i"
BSD Sed does support in-place updating with -i, but - unlike GNU Sed - it requires that an empty option-argument (indicating that no backup of the input file should be made) be specified as a separate argument: -i ''.
Your write-to-a-temp-file-first-then-replace-the-original approach works too, but it's generally better to use the following idiom: sed ... file > file.tmp && mv file.tmp file. Separating the mv command with && ensures that the original file is only replaced if the sed command succeeded.
That said, that doesn't help with logic errors as in the case at hand: despite outputting nothing, sed reports success in this case.
This code would achieve what you want (if I understood your question correctly):
grep -i "Я\|Є\|Ї\|І" /folder/file >> /tmp/result
The result is stored on /tmp/result
Note: I don't know Ukranian, so I'm sure I did not included all Ukranian characters, please add/delete Ukranian characters you want to match to the construction above.
Note2: this code is case insensitive thanks to grep -i so you only need to add the character once (lowercase or capital).
To put it on your loop it could be:
for i in *.txt;
do
grep -i "Я\|Є\|Ї\|І" "$i" > "$i".out
mv "$i".out "$i"
done
Edit: I edited this answer to make it simpler, and to add a loop to it.
I have file which contain following context like
abc...
include /home/user/file.txt'
some text
I need to remove include and also complete path after include.
I have used following command which remove include but did not remove path.
sed -i -r 's#include##g' 'filename'
I am also trying to understand above command but did not understand following thing ( copy paste from somewhere)
i - modify file change
r - read file
s- Need input
g - Need input
Try this,
$ sed '/^include /s/.*//g' file.txt
abc...
some text
It remove all the texts in a line which starts with include. s means substitute. so s/.*//g means replace all the texts with null.g means global. The substitution will be applied globally.
OR
$ sed '/^include /d' file.txt
abc...
some text
d means delete.
It deletes the line which starts with include. To save the changes made(inline edit), your commands should be
sed -i '/^include /s/.*//g' file.txt
sed -i '/^include /d' file.txt
I your case if you just want to delete the second line, you can use:
sed -i '2d' file
If you want to explore something about linux commands then man pages are there for you.
Just go to terminal and type:
man sed
as per your question, The above command without -i will show the file content on terminal by deleting the second line from the input file. However, the input file remains unchanged. To update the original file or to make the changes permanently in the source file, use the -i option.
-i[SUFFIX], --in-place[=SUFFIX] :
edit files in place (makes backup if extension supplied)
-r or --regexp-extended :
option is to use extended regular expressions in the script.
s/regexp/replacement/ :
Attempt to match regexp against the pattern space. If success‐
ful, replace that portion matched with replacement. The
replacement may contain the special character & to refer to that
portion of the pattern space which matched, and the special
escapes \1 through \9 to refer to the corresponding matching
sub-expressions in the regexp.
g G : Copy/append hold space to pattern space.
grep -v
This is not about learning sed, but as an alternative (and short) solution, there is:
grep -v '^include' filename_in
Or with output redirection:
grep -v '^include' filename_in > filename_out
-v option for grep inverts matching (hence printing non-matching lines).
For simple deletion that's what I'd use; if you have to modify your path after the include, stick with sed instead.
You can use awk to just delete the line:
awk '/^include/ {next}1' file
sed -i -r 's#include##g' 'filename'
-i: you directly modify the treated file, by default, sed read a file, modify the content via stdout (the original file stay the same).
-r: use of extended regular expression (and not reduce to POSIX limited one).This is not necessary in this case due to simple POSIX compliant action in action list (the s### string).
s#pattern#NewValue#: substitute in current line the pattern (Regular Expression) with "Newvalue" (that also use internal buffer or specific value). The traditionnal form is s/// but in this case, using / in path (pattern or new value) an alternate form is used to avoid to escape all / in pattern or new value
g: is an option of s### that specify change EVERY occurence and not the first (by default)
so here it replace ANY occurence of include by nothing (remove) directly into your file
As per the Avinash Raj solution you got what you want but you want some explaination about some parameter used in sed command
First one is
command: s for substitution
With the sed command the substitute command s changes all occurrences of the regular expression into a new value. A simple example is changing "my" in the "file1" to "yours" in the "file2" file:
sed s/my/yours/ file1 >file2
The character after the s is the delimiter. It is conventionally a slash, because this is what ed, more, and vi use. It can be anything you want, however. If you want to change a pathname that contains a slash - say /usr/local/bin to /common/bin - you could use the backslash to quote the slash:
sed 's/\/usr\/local\/bin/\/common\/bin/' <old >new
/g - Global replacement
Replace all matches, not just the first match.
If you tell it to change a word, it will only change the first occurrence of the word on a line. You may want to make the change on every word on the line instead of the first then add a g after the last delimiter and use the work-around:
Delete with d
Delete the pattern space; immediately start next cycle.
You can delete line by specifying the line number. like
sed '$d' filename.txt
It will remove last line of file
sed '2 d' file.txt
It will delete second line of file.
-i option
This option specifies that files are to be edited in-place. GNU sed does this by creating a temporary file and sending output to this file rather than to the standard output.
To modify file actully you can use -i option without it sed command repressent changes on stdout not actual file. You can take backup of original file before modification by using -i.bak option.
-r option
--regexp-extended
Use extended regular expressions rather than basic regular expressions. Extended regexps are those that egrep accepts; they can be clearer because they usually have less backslashes, but are a GNU extension and hence scripts that use them are not portable.
I have a hash file that takes the form of:
SHA1(disk.iso)= 43798473890473280573920473902472083947320
I need to replace the old hash with the new hash.
I've been trying to modify some old code with no luck:
sed -i 's/SHA1(disk.iso)"[^+]*"/"'" $HASH"'"/' manifest
Any thoughts here?
* UPDATE *
The sting listed above is correct:
SHA1(disk.iso)= (some SHA1 hash here. Note the space after the equal sign.)
Here is the current code:
sed -i "s/\(SHA1(disk.iso)=\).*/\1 $HASH/" manifest
but still nothing. This does not modify the line in question.
* SOLUTION *
THIS WORKS:
sed -i "s/\(SHA1(disk.iso)=\).*/\1 $HASH/" manifest
I just had the file name wrong. Thank you Janos
Here you go:
sed -i "s/\(SHA1(disk.iso)=\).*/\1 $HASH/" manifest
That is:
Capture the filename within \(...\), and match the rest of the line with .*
Replace the pattern (the entire line) with the captured filename \1, and append the $HASH
The whole thing within double quotes, so that shell variables are expanded.
Here's another variation to do the same thing:
sed -i "/^SHA1(disk.iso)/ s/=.*/= $HASH/" manifest
That is:
For lines starting with SHA1(disk.iso)
Replace the = sign and everything after it with = $HASH
You regular expression seems to be strange.
You use to many quotes.
You can just do (if you now the hashes):
sed -i "s/$OLDHASH/$NEWHASH" manifest
And if you don't know them and just want to replace any line with SHA1(disk.iso),
you can write:
sed -i "s/\(SHA1(disk.iso)=\).*/\1 $HASH/"
\(\) here mean backreferences; that means that you save the line in a register, that will be later used using \1. Of course, you could write directly:
sed -i "s/SHA1(disk.iso)=.*/SHA1(disk.iso) $HASH/"
but in this case it would be impossible to write something like disk[123].iso
to match several ISOs at once.
Two steps:
find the right line
replace the number in that line with a new number
Typically you do this with
cat myHashFile.txt | sed '/SHA1(disk.iso)/ {s/\d+/'$HASH'/}' > newHashFile.txt
The first term in /.../ in general takes a regular expression "apply what follows to lines meeting this condition"
The second part in {..} is the command:
s substitute
\d any digit
\d+ one or more digits (greedy)
$HASH replace with the contents of the $HASH variable
For some reason I can't seem to find a straightforward answer to this and I'm on a bit of a time crunch at the moment. How would I go about inserting a choice line of text after the first line matching a specific string using the sed command. I have ...
CLIENTSCRIPT="foo"
CLIENTFILE="bar"
And I want insert a line after the CLIENTSCRIPT= line resulting in ...
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Try doing this using GNU sed:
sed '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
if you want to substitute in-place, use
sed -i '/CLIENTSCRIPT="foo"/a CLIENTSCRIPT2="hello"' file
Output
CLIENTSCRIPT="foo"
CLIENTSCRIPT2="hello"
CLIENTFILE="bar"
Doc
see sed doc and search \a (append)
Note the standard sed syntax (as in POSIX, so supported by all conforming sed implementations around (GNU, OS/X, BSD, Solaris...)):
sed '/CLIENTSCRIPT=/a\
CLIENTSCRIPT2="hello"' file
Or on one line:
sed -e '/CLIENTSCRIPT=/a\' -e 'CLIENTSCRIPT2="hello"' file
(-expressions (and the contents of -files) are joined with newlines to make up the sed script sed interprets).
The -i option for in-place editing is also a GNU extension, some other implementations (like FreeBSD's) support -i '' for that.
Alternatively, for portability, you can use perl instead:
perl -pi -e '$_ .= qq(CLIENTSCRIPT2="hello"\n) if /CLIENTSCRIPT=/' file
Or you could use ed or ex:
printf '%s\n' /CLIENTSCRIPT=/a 'CLIENTSCRIPT2="hello"' . w q | ex -s file
Sed command that works on MacOS (at least, OS 10) and Unix alike (ie. doesn't require gnu sed like Gilles' (currently accepted) one does):
sed -e '/CLIENTSCRIPT="foo"/a\'$'\n''CLIENTSCRIPT2="hello"' file
This works in bash and maybe other shells too that know the $'\n' evaluation quote style. Everything can be on one line and work in
older/POSIX sed commands. If there might be multiple lines matching the CLIENTSCRIPT="foo" (or your equivalent) and you wish to only add the extra line the first time, you can rework it as follows:
sed -e '/^ *CLIENTSCRIPT="foo"/b ins' -e b -e ':ins' -e 'a\'$'\n''CLIENTSCRIPT2="hello"' -e ': done' -e 'n;b done' file
(this creates a loop after the line insertion code that just cycles through the rest of the file, never getting back to the first sed command again).
You might notice I added a '^ *' to the matching pattern in case that line shows up in a comment, say, or is indented. Its not 100% perfect but covers some other situations likely to be common. Adjust as required...
These two solutions also get round the problem (for the generic solution to adding a line) that if your new inserted line contains unescaped backslashes or ampersands they will be interpreted by sed and likely not come out the same, just like the \n is - eg. \0 would be the first line matched. Especially handy if you're adding a line that comes from a variable where you'd otherwise have to escape everything first using ${var//} before, or another sed statement etc.
This solution is a little less messy in scripts (that quoting and \n is not easy to read though), when you don't want to put the replacement text for the a command at the start of a line if say, in a function with indented lines. I've taken advantage that $'\n' is evaluated to a newline by the shell, its not in regular '\n' single-quoted values.
Its getting long enough though that I think perl/even awk might win due to being more readable.
A POSIX compliant one using the s command:
sed '/CLIENTSCRIPT="foo"/s/.*/&\
CLIENTSCRIPT2="hello"/' file
Maybe a bit late to post an answer for this, but I found some of the above solutions a bit cumbersome.
I tried simple string replacement in sed and it worked:
sed 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
& sign reflects the matched string, and then you add \n and the new line.
As mentioned, if you want to do it in-place:
sed -i 's/CLIENTSCRIPT="foo"/&\nCLIENTSCRIPT2="hello"/' file
Another thing. You can match using an expression:
sed -i 's/CLIENTSCRIPT=.*/&\nCLIENTSCRIPT2="hello"/' file
Hope this helps someone
The awk variant :
awk '1;/CLIENTSCRIPT=/{print "CLIENTSCRIPT2=\"hello\""}' file
I had a similar task, and was not able to get the above perl solution to work.
Here is my solution:
perl -i -pe "BEGIN{undef $/;} s/^\[mysqld\]$/[mysqld]\n\ncollation-server = utf8_unicode_ci\n/sgm" /etc/mysql/my.cnf
Explanation:
Uses a regular expression to search for a line in my /etc/mysql/my.cnf file that contained only [mysqld] and replaced it with
[mysqld]
collation-server = utf8_unicode_ci
effectively adding the collation-server = utf8_unicode_ci line after the line containing [mysqld].
I had to do this recently as well for both Mac and Linux OS's and after browsing through many posts and trying many things out, in my particular opinion I never got to where I wanted to which is: a simple enough to understand solution using well known and standard commands with simple patterns, one liner, portable, expandable to add in more constraints. Then I tried to looked at it with a different perspective, that's when I realized i could do without the "one liner" option if a "2-liner" met the rest of my criteria. At the end I came up with this solution I like that works in both Ubuntu and Mac which i wanted to share with everyone:
insertLine=$(( $(grep -n "foo" sample.txt | cut -f1 -d: | head -1) + 1 ))
sed -i -e "$insertLine"' i\'$'\n''bar'$'\n' sample.txt
In first command, grep looks for line numbers containing "foo", cut/head selects 1st occurrence, and the arithmetic op increments that first occurrence line number by 1 since I want to insert after the occurrence.
In second command, it's an in-place file edit, "i" for inserting: an ansi-c quoting new line, "bar", then another new line. The result is adding a new line containing "bar" after the "foo" line. Each of these 2 commands can be expanded to more complex operations and matching.