I am new to this platform. Just had a requirement which I have been working over sometime but not able to find it.
If this pattern was to occur in the middle of a line. How to handle it. Suppose the line is like. aaaa ---- bbbb. If i want to erase the ----bbbb part how to do it. But I want to keep the aaaa part as it is in the file.
Thanks
You can do it easily with sed:
sed -r 's/^--.*//' inputfile > outputfile
Or in place:
sed -r -i.bak 's/^--.*//' inputfile
This will create an inputfile.bak as a backup before modifying the file
Here is a good old bash solution:
while read -r line; do
echo "${line/#--*/}"
done < inputFile > outputFile
One way using awk:
awk '/^--/{$0=" ";}1' file
This will repalce the line with a space when it begins with --
Its not clear from your problem statement what the criteria (limitations) of the solution is.
What you are looking for is something that will support regular expressions. There are a lot of UNIX/Linux tools that can be used to solve this problem.
One simple solution is:
# cat file.txt | sed -e "{s/^--.*/ /}"
The regular expression "^--." will match any line beginning "^" with "--" followed
by any number of characters ".". "s" is the sed substitution command.
so "s/^--.*/ /" means, substitute all lines that start with -- and are followed by any
number of characters with a single space.
Related
I have a csv file, with student name and marks. I want to update "marks" of a student with name "jack"(the only person in the csv). the data in csv file looks as below.
student,marks
jack,10
peter,20
rick,10
I found this awk '$1 == "Audrey" {print $2}' numbers.txt, but iam not sure on how to modify the file.
awk 'BEGIN{FS=OFS=","} $1=="jack"{$2=27} 1' foo.csv > tmp && mv tmp foo.csv
It worked for me with
sed -ir "s/^\(jack\),.*/\1,$new_grade/"
input.csv. with argument "r" or else i get the "error sed: 1: "input.csv": command i expects \ followed by text".
ed is usually better for in-place editing of files than sed:
printf "%s\n" "/^jack,/c" "jack,${new_grade}" "." w | ed -s input.csv
or using a heredoc to make it easier to read:
ed -s input.csv <<EOF
/^jack,/c
jack,${new_grade}
.
w
EOF
At the first line starting with jack,, change it to jack,XX where XX is the value of the new_grade variable, and write the new contents of the file.
You could use sed:
new_grade=9
sed -i'' "s/^\(jack\),.*/\1,$new_grade/"
The pattern ^\(jack\),.* matches the beginning of the line ^ followed by jack by a comma and the rest of the line .*. The replacement string \1,$new_mark contains the first captured group \1 (in this case jack) followed by a comma and the new mark.
Alternatively you could loop over the file and use a pattern substitution:
new_grade=9
while read -s line; do
echo ${line/jack,*/jack,$new_grade}
done < grades.txt > grades2.txt
Another approach with sed is to anchor the replacement to the digits at the end of the line with:
sed '/^jack,/s/[0-9][0-9]*$/12/' file
This uses the form sed '/find/s/match/replace' where find locates at the beginning of the line '^' the word "jack," eliminating all ambiguity with, e.g. jackson,33. Then the normal substitution form of 's/match/replace/' where match locates at least one digit at the end of the line (anchored by '$') and replaces it with the 12 (or whatever you choose).
Example Use/Output
With your example file in file, you would have:
$ sed '/^jack,/s/[0-9][0-9]*$/12/' file
student,marks
jack,12
peter,20
rick,10
(note: the POSIX character class of [[:digit:]] is equivalent to [0-9] which is another alternative)
The equivalent expression using the POSIX character class would be:
sed '/^jack,/s/[[:digit:]][[:digit:]]*$/12/' file
You can also use Extended Regular Expression which provides the '+' repetition operator to indicate one-or-more compared to the basic repetition designator of '*' to indicate zero-or-more. The equivalent ERE would be sed -E '/^jack,/s/[0-9]+$/12/' file
You can add the -i option to edit in-place and/or using it as -i.bak to create a backup of the original with the .bak extension before modifying the original.
I have a text file and I would like to only delete the first character of the text file, is there a way to do this in shell script?
I'm new to writing scripts so I really don't know where to start. I understand that the main command most people use is "sed" but I can only find how to use that as a find and replace tool.
All help is appreciated.
You can use the tail command, telling it to start from character 2:
tail -c +2 infile > outfile
You can use sed
sed '1s/^.//' startfile > endfile
1s means match line 1, in substitution mode (s)
^. means at the beginning of the line (^), match any character (.)
There's nothing between the last slashes, which means substitute with nothing (remove)
I used to use cut command to do this.
For example:
cat file|cut -c2-80
Will show characters from column 2 to 80 only.
In your case you can use:
cat file|cut -c2-10000 > newfile
I hope this help you.
[]s
You can also use the 0,addr2 address-range to limit replacements to the first substitution, e.g.
sed '0,/./s/^.//' file
That will remove the 1st character of the file and the sed expression will be at the end of its range -- effectively replacing only the 1st occurrence.
To edit the file in place, use the -i option, e.g.
sed -i '0,/./s/^.//' file
or simply redirect the output to a new file:
sed '0,/./s/^.//' file > newfile
A few other ideas:
awk '{print (NR == 1 ? substr($0,2) : $0)}' file
perl -0777 -pe 's/.//' file
perl -pe 's/.// unless $done; $done = 1' file
ed file <<END
1s/.//
w
q
END
dd allows you to specify an offset at which to start reading:
dd ibs=1 seek=1 if="$input" of="$output"
(where the variables are set to point to your input and output files, respectively)
I am trying to learn some bash scripting and i can't understand how to use grep in order to split a URL link for example :
blabla1.com
blabla2.gov
blabla3.fr
I just want to keep com , gov and fr ( without the '.' character) ignore whats before '.'
Thanks in advance ..
Grep is a tool for matching text. You need something else if you want to transform text. If you have the values in question in a bash variable, then what you ask is pretty easy:
authority=blabla.com
# Here's the important bit:
domain=${authority/*./}
echo $domain
The funny syntax in the middle evaluates to the result of a pattern substitution on the value of variable temp.
If you're trying to do this on lines of a file, then the sed program is your friend:
sed 's/.*\.//' < input.file
This is again a pattern substitution, but sed uses regular expression patterns, whereas bash uses shell glob patterns.
grep -E -o '[^.]+$' < input
-o instructs grep to print only the matching part of the line
-E switches on extended regexp which is needed for + quantifier
[^.]+$ means any character which is not a dot at the end of the line
Try this way:
grep -o -E '[a-z]{2,3}\b' input > output
-o, --only-matching: Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.
$ cat input
blabla1.com
blabla2.gov
blabla3.fr
$ cat output
com
gov
fr
$ cut -d. -f2 file
com
gov
fr
If that's not all you need, post some more truly representative input and expected output so we can help you find the right solution.
I have pattern like below
hi
hello
hallo
greetings
salutations
no more hello for you
I am trying to replace all newlines with tab using the following command
sed -e "s_/\n_/\t_g"
but it's not working.
Could anybody please help? I'm looking for a solution in sed/awk.
tr is better here, I think:
tr "\n" "\t" < newlines
As Nifle suggested in a comment, newlines here is the name of the file holding the original text.
Because sed is so line-oriented, it's more complicated to use in a case like this.
not sure about output you want
# awk -vRS="\n" -vORS="\t" '1' file
hi hello hallo greetings salutations no more hello for you
sed '$!{:a;N;s/\n/\t/;ta}' file
You can't replace newlines on a line-by-line basis with sed. You have to accumulate lines and replace the newlines between them.
text abc\n <- can't replace this one
text abc\ntext def\n <- you can replace the one after "abc" but not the one at the end
This sed script accumulates all the lines and eliminates all the newlines but the last:
sed -n '1{x;d};${H;x;s/\n/\t/g;p};{H}'
By the way, your sed script sed -e "s_/\n_/\t_g" is trying to say "replace all slashes followed by newlines with slashes followed by tabs". The underscores are taking on the role of delimiters for the s command so that slashes can be more easily used as characters for searching and replacing.
paste -s
-s Concatenate all of the lines of each separate input file in
command line order. The newline character of every line
except the last line in each input file is replaced with the
tab character, unless otherwise specified by the -d option.
You are almost there with your sed script, you'd just need to change it to:
sed -e "s/\n/\t/g"
The \ is enough for escape, you don't need to add _
And you need to add the / before g at the end to let sed know that this is the last part of the script.
How to insert a newline before a pattern within a line?
For example, this will insert a newline behind the regex pattern.
sed 's/regex/&\n/g'
How can I do the same but in front of the pattern?
Given this sample input file, the pattern to match on is the phone number.
some text (012)345-6789
Should become
some text
(012)345-6789
This works in bash and zsh, tested on Linux and OS X:
sed 's/regexp/\'$'\n/g'
In general, for $ followed by a string literal in single quotes bash performs C-style backslash substitution, e.g. $'\t' is translated to a literal tab. Plus, sed wants your newline literal to be escaped with a backslash, hence the \ before $. And finally, the dollar sign itself shouldn't be quoted so that it's interpreted by the shell, therefore we close the quote before the $ and then open it again.
Edit: As suggested in the comments by #mklement0, this works as well:
sed $'s/regexp/\\\n/g'
What happens here is: the entire sed command is now a C-style string, which means the backslash that sed requires to be placed before the new line literal should now be escaped with another backslash. Though more readable, in this case you won't be able to do shell string substitutions (without making it ugly again.)
Some of the other answers didn't work for my version of sed.
Switching the position of & and \n did work.
sed 's/regexp/\n&/g'
Edit: This doesn't seem to work on OS X, unless you install gnu-sed.
In sed, you can't add newlines in the output stream easily. You need to use a continuation line, which is awkward, but it works:
$ sed 's/regexp/\
&/'
Example:
$ echo foo | sed 's/.*/\
&/'
foo
See here for details. If you want something slightly less awkward you could try using perl -pe with match groups instead of sed:
$ echo foo | perl -pe 's/(.*)/\n$1/'
foo
$1 refers to the first matched group in the regular expression, where groups are in parentheses.
On my mac, the following inserts a single 'n' instead of newline:
sed 's/regexp/\n&/g'
This replaces with newline:
sed "s/regexp/\\`echo -e '\n\r'`/g"
echo one,two,three | sed 's/,/\
/g'
You can use perl one-liners much like you do with sed, with the advantage of full perl regular expression support (which is much more powerful than what you get with sed). There is also very little variation across *nix platforms - perl is generally perl. So you can stop worrying about how to make your particular system's version of sed do what you want.
In this case, you can do
perl -pe 's/(regex)/\n$1/'
-pe puts perl into a "execute and print" loop, much like sed's normal mode of operation.
' quotes everything else so the shell won't interfere
() surrounding the regex is a grouping operator. $1 on the right side of the substitution prints out whatever was matched inside these parens.
Finally, \n is a newline.
Regardless of whether you are using parentheses as a grouping operator, you have to escape any parentheses you are trying to match. So a regex to match the pattern you list above would be something like
\(\d\d\d\)\d\d\d-\d\d\d\d
\( or \) matches a literal paren, and \d matches a digit.
Better:
\(\d{3}\)\d{3}-\d{4}
I imagine you can figure out what the numbers in braces are doing.
Additionally, you can use delimiters other than / for your regex. So if you need to match / you won't need to escape it. Either of the below is equivalent to the regex at the beginning of my answer. In theory you can substitute any character for the standard /'s.
perl -pe 's#(regex)#\n$1#'
perl -pe 's{(regex)}{\n$1}'
A couple final thoughts.
using -ne instead of -pe acts similarly, but doesn't automatically print at the end. It can be handy if you want to print on your own. E.g., here's a grep-alike (m/foobar/ is a regex match):
perl -ne 'if (m/foobar/) {print}'
If you are finding dealing with newlines troublesome, and you want it to be magically handled for you, add -l. Not useful for the OP, who was working with newlines, though.
Bonus tip - if you have the pcre package installed, it comes with pcregrep, which uses full perl-compatible regexes.
In this case, I do not use sed. I use tr.
cat Somefile |tr ',' '\012'
This takes the comma and replaces it with the carriage return.
To insert a newline to output stream on Linux, I used:
sed -i "s/def/abc\\\ndef/" file1
Where file1 was:
def
Before the sed in-place replacement, and:
abc
def
After the sed in-place replacement. Please note the use of \\\n. If the patterns have a " inside it, escape using \".
Hmm, just escaped newlines seem to work in more recent versions of sed (I have GNU sed 4.2.1),
dev:~/pg/services/places> echo 'foobar' | sed -r 's/(bar)/\n\1/;'
foo
bar
echo pattern | sed -E -e $'s/^(pattern)/\\\n\\1/'
worked fine on El Captitan with () support
In my case the below method works.
sed -i 's/playstation/PS4/' input.txt
Can be written as:
sed -i 's/playstation/PS4\nplaystation/' input.txt
PS4
playstation
Consider using \\n while using it in a string literal.
sed : is stream editor
-i : Allows to edit the source file
+: Is delimiter.
I hope the above information works for you 😃.
in sed you can reference groups in your pattern with "\1", "\2", ....
so if the pattern you're looking for is "PATTERN", and you want to insert "BEFORE" in front of it, you can use, sans escaping
sed 's/(PATTERN)/BEFORE\1/g'
i.e.
sed 's/\(PATTERN\)/BEFORE\1/g'
You can also do this with awk, using -v to provide the pattern:
awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' file
This checks if a line contains a given pattern. If so, it appends a new line to the beginning of it.
See a basic example:
$ cat file
hello
this is some pattern and we are going ahead
bye!
$ awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' file
hello
this is some
pattern and we are going ahead
bye!
Note it will affect to all patterns in a line:
$ cat file
this pattern is some pattern and we are going ahead
$ awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' d
this
pattern is some
pattern and we are going ahead
sed -e 's/regexp/\0\n/g'
\0 is the null, so your expression is replaced with null (nothing) and then...
\n is the new line
On some flavors of Unix doesn't work, but I think it's the solution to your problem.
echo "Hello" | sed -e 's/Hello/\0\ntmow/g'
Hello
tmow
This works in MAC for me
sed -i.bak -e 's/regex/xregex/g' input.txt sed -i.bak -e 's/qregex/\'$'\nregex/g' input.txt
Dono whether its perfect one...
After reading all the answers to this question, it still took me many attempts to get the correct syntax to the following example script:
#!/bin/bash
# script: add_domain
# using fixed values instead of command line parameters $1, $2
# to show typical variable values in this example
ipaddr="127.0.0.1"
domain="example.com"
# no need to escape $ipaddr and $domain values if we use separate quotes.
sudo sed -i '$a \\n'"$ipaddr www.$domain $domain" /etc/hosts
The script appends a newline \n followed by another line of text to the end of a file using a single sed command.
In vi on Red Hat, I was able to insert carriage returns using just the \r character. I believe this internally executes 'ex' instead of 'sed', but it's similar, and vi can be another way to do bulk edits such as code patches. For example. I am surrounding a search term with an if statement that insists on carriage returns after the braces:
:.,$s/\(my_function(.*)\)/if(!skip_option){\r\t\1\r\t}/
Note that I also had it insert some tabs to make things align better.
Just to add to the list of many ways to do this, here is a simple python alternative. You could of course use re.sub() if a regex were needed.
python -c 'print(open("./myfile.txt", "r").read().replace("String to match", "String to match\n"))' > myfile_lines.txt
sed 's/regexp/\'$'\n/g'
works as justified and detailed by mojuba in his answer .
However, this did not work:
sed 's/regexp/\\\n/g'
It added a new line, but at the end of the original line, a \n was added.