I have tabs (five) between these words:
cat dog
and I want this output:
cat
dog
I tried this: sed 's/\t/\n/g; /^$/d' pets
and the output was the same as with sed 's/\t/\n/g' pets:
cat
dog
I had to execute sed two times to get what I wanted. Like this sed 's/\t/\n/g' pets>temp after sed '/^$/d' temp
Is there a way to get the desired output with one command?
Continuing from my comment. The problem with sed 's/\t/\n/g is you will replace each '\t' with a '\n'. You want to replace a sequence of tabs with a single newline. For that you need:
sed 's/\t\t*/\n/g'
or if you like explicitly enclosing the '\t' in [ ] that's fine as well.
The expression '\t' matches a single tab, when followed by '\t\t*' it matches a tab and zero or more tabs that follow replacing the sequence with a single '\n', resulting in your desired output:
cat
dog
The g (globally) at the end will just replace each sequence with a single newline, e.g.
"cat dog fish"
(separated by tabs), becomes
cat
dog
fish
Let me know if you have any questions.
If the number of tabs is important, BRE will let you specify bounds by escaping the curly braces, like this:
$ sed $'s/\t\{5\}/\\\n/' <<<$'one\t\t\t\t\ttwo'
one
two
Note that you haven't specified an operating system so it's unknown whether you're using GNU sed, which would let you include things like \t in your regex. (I use FreeBSD and macOS, where sed does not have this capability.) But you HAVE mentioned that you're using bash, which supports "format expansion". You can use this bash feature to insert "literal" special characters into your script.
I need to convert a series of text files that are formatted with line breaks to single lines separated by newlines (\n). For example:
This is an example text file
where the contents are separated
by line breaks
What I want this to look like is:
This is an example text file\nwhere the contents are separated\nby line breaks\n
I'm open to using awk, sed, or any builtin POSIX commands.
Please try this solution:
awk 'BEGIN{RS="\n";ORS="\\n"}1' file.txt
What we are doing is detect the Record Separator like '\n', and when we print we use '\n', the double slash implies it must print '\n', to force the printing we use the pattern 1 with the default action (print the whole record).
If you have any problem let me know, I don't have an awk available to try it.
It's not clear when you say "line break" if you you mean Carriage Return, Line Feed, or Newline or something else, nor is it clear if you want to replace newlines with the string \n or if you just want to strip Carriage Returns from newlines or something else, but if its the latter then all you need is:
dos2unix file
If you don't have dos2unix you can do it with any awk:
$ printf 'foo\r\nbar\r\n' | cat -v
foo^M
bar^M
$ printf 'foo\r\nbar\r\n' | awk '{sub(/\r$/,"")}1' | cat -v
foo
bar
You can't do it robustly with tr since it can't tell when a \r is at the end of a line or not, and you can't do it portably with sed.
This might work for you (GNU sed):
sed '1h;1!H;$!d;x;s/\n/\\n/g' file
Slurp the file into memory and quote newlines.
This question already has answers here:
Bash - how to put each line within quotation
(8 answers)
Closed 6 years ago.
I have a file with one word on each line, I want to add a " before the word, and a ", after the word with Bash.
Example:
Hello
Sleep
Try
And I want:
"Hello",
"Sleep",
"try",
Hope anyone can help me
sed 's/.*/"&",/' file
In the replacement part of a substitution, & is replaced with whatever matched the original regular expression. .* matches the whole line.
Assuming you've got one word per line, I'd use the following GNU sed command :
sed -i 's/.*/"\0",/' <your_file>
Where you replace <your_file> by the path to your file.
Explanation :
sed means Stream EDitor ; it specialise in filtering and transforming text streams
-i is it's in-place option : it means it will write the result of its transformation in the file it's working on
s/search/replace/ is sed's search & replace command : it search for a pattern written as regular expression, and replace the matched text by something else
.* is a regex pattern that will match as much as it can. Since sed works line by line, it will match a whole line
\0 is a back-reference to the whole matched string
"\0", is the whole matched string, enclosed in quotes, followed by a comma.
s/.*/"\0",/ is a sed search&replace command that will match a whole line and enclose it in quotes, following it by a comma
Already answered here
Using awk
awk '{ print "\""$0"\","}' inputfile
Using pure bash
while read FOO; do
echo -e "\"$FOO\","
done < inputfile
where inputfile would be a file containing the lines without quotes.
If your file has empty lines, awk is definitely the way to go:
awk 'NF { print "\""$0"\","}' inputfile
NF tells awk to only execute the print command when the Number of Fields is more than zero (line is not empty).
sed (the Stream EDitor) is one choice for this manipulation:
sed -e 's/^/"/;s/$/",/' file
There are two parts to the regular expression, separated by ;:
s/^/"/ means add a " to the beginning of each line.
s/$/",/ means add a ", to the end of each line.
The magic being the anchors for start of line (^) and end of line ($). The man page has more details. It's worth learning sed.
Just to be different: perl
perl -lne 'print qq{"$_",}' file
This is a simple one liner:
sed -i 's/^/"/;s/$/",/' file_name
> echo $word
word
> echo "\""$word"\","
"word",
See How to escape double quotes in bash?
I am new to this platform. Just had a requirement which I have been working over sometime but not able to find it.
If this pattern was to occur in the middle of a line. How to handle it. Suppose the line is like. aaaa ---- bbbb. If i want to erase the ----bbbb part how to do it. But I want to keep the aaaa part as it is in the file.
Thanks
You can do it easily with sed:
sed -r 's/^--.*//' inputfile > outputfile
Or in place:
sed -r -i.bak 's/^--.*//' inputfile
This will create an inputfile.bak as a backup before modifying the file
Here is a good old bash solution:
while read -r line; do
echo "${line/#--*/}"
done < inputFile > outputFile
One way using awk:
awk '/^--/{$0=" ";}1' file
This will repalce the line with a space when it begins with --
Its not clear from your problem statement what the criteria (limitations) of the solution is.
What you are looking for is something that will support regular expressions. There are a lot of UNIX/Linux tools that can be used to solve this problem.
One simple solution is:
# cat file.txt | sed -e "{s/^--.*/ /}"
The regular expression "^--." will match any line beginning "^" with "--" followed
by any number of characters ".". "s" is the sed substitution command.
so "s/^--.*/ /" means, substitute all lines that start with -- and are followed by any
number of characters with a single space.
How to insert a newline before a pattern within a line?
For example, this will insert a newline behind the regex pattern.
sed 's/regex/&\n/g'
How can I do the same but in front of the pattern?
Given this sample input file, the pattern to match on is the phone number.
some text (012)345-6789
Should become
some text
(012)345-6789
This works in bash and zsh, tested on Linux and OS X:
sed 's/regexp/\'$'\n/g'
In general, for $ followed by a string literal in single quotes bash performs C-style backslash substitution, e.g. $'\t' is translated to a literal tab. Plus, sed wants your newline literal to be escaped with a backslash, hence the \ before $. And finally, the dollar sign itself shouldn't be quoted so that it's interpreted by the shell, therefore we close the quote before the $ and then open it again.
Edit: As suggested in the comments by #mklement0, this works as well:
sed $'s/regexp/\\\n/g'
What happens here is: the entire sed command is now a C-style string, which means the backslash that sed requires to be placed before the new line literal should now be escaped with another backslash. Though more readable, in this case you won't be able to do shell string substitutions (without making it ugly again.)
Some of the other answers didn't work for my version of sed.
Switching the position of & and \n did work.
sed 's/regexp/\n&/g'
Edit: This doesn't seem to work on OS X, unless you install gnu-sed.
In sed, you can't add newlines in the output stream easily. You need to use a continuation line, which is awkward, but it works:
$ sed 's/regexp/\
&/'
Example:
$ echo foo | sed 's/.*/\
&/'
foo
See here for details. If you want something slightly less awkward you could try using perl -pe with match groups instead of sed:
$ echo foo | perl -pe 's/(.*)/\n$1/'
foo
$1 refers to the first matched group in the regular expression, where groups are in parentheses.
On my mac, the following inserts a single 'n' instead of newline:
sed 's/regexp/\n&/g'
This replaces with newline:
sed "s/regexp/\\`echo -e '\n\r'`/g"
echo one,two,three | sed 's/,/\
/g'
You can use perl one-liners much like you do with sed, with the advantage of full perl regular expression support (which is much more powerful than what you get with sed). There is also very little variation across *nix platforms - perl is generally perl. So you can stop worrying about how to make your particular system's version of sed do what you want.
In this case, you can do
perl -pe 's/(regex)/\n$1/'
-pe puts perl into a "execute and print" loop, much like sed's normal mode of operation.
' quotes everything else so the shell won't interfere
() surrounding the regex is a grouping operator. $1 on the right side of the substitution prints out whatever was matched inside these parens.
Finally, \n is a newline.
Regardless of whether you are using parentheses as a grouping operator, you have to escape any parentheses you are trying to match. So a regex to match the pattern you list above would be something like
\(\d\d\d\)\d\d\d-\d\d\d\d
\( or \) matches a literal paren, and \d matches a digit.
Better:
\(\d{3}\)\d{3}-\d{4}
I imagine you can figure out what the numbers in braces are doing.
Additionally, you can use delimiters other than / for your regex. So if you need to match / you won't need to escape it. Either of the below is equivalent to the regex at the beginning of my answer. In theory you can substitute any character for the standard /'s.
perl -pe 's#(regex)#\n$1#'
perl -pe 's{(regex)}{\n$1}'
A couple final thoughts.
using -ne instead of -pe acts similarly, but doesn't automatically print at the end. It can be handy if you want to print on your own. E.g., here's a grep-alike (m/foobar/ is a regex match):
perl -ne 'if (m/foobar/) {print}'
If you are finding dealing with newlines troublesome, and you want it to be magically handled for you, add -l. Not useful for the OP, who was working with newlines, though.
Bonus tip - if you have the pcre package installed, it comes with pcregrep, which uses full perl-compatible regexes.
In this case, I do not use sed. I use tr.
cat Somefile |tr ',' '\012'
This takes the comma and replaces it with the carriage return.
To insert a newline to output stream on Linux, I used:
sed -i "s/def/abc\\\ndef/" file1
Where file1 was:
def
Before the sed in-place replacement, and:
abc
def
After the sed in-place replacement. Please note the use of \\\n. If the patterns have a " inside it, escape using \".
Hmm, just escaped newlines seem to work in more recent versions of sed (I have GNU sed 4.2.1),
dev:~/pg/services/places> echo 'foobar' | sed -r 's/(bar)/\n\1/;'
foo
bar
echo pattern | sed -E -e $'s/^(pattern)/\\\n\\1/'
worked fine on El Captitan with () support
In my case the below method works.
sed -i 's/playstation/PS4/' input.txt
Can be written as:
sed -i 's/playstation/PS4\nplaystation/' input.txt
PS4
playstation
Consider using \\n while using it in a string literal.
sed : is stream editor
-i : Allows to edit the source file
+: Is delimiter.
I hope the above information works for you 😃.
in sed you can reference groups in your pattern with "\1", "\2", ....
so if the pattern you're looking for is "PATTERN", and you want to insert "BEFORE" in front of it, you can use, sans escaping
sed 's/(PATTERN)/BEFORE\1/g'
i.e.
sed 's/\(PATTERN\)/BEFORE\1/g'
You can also do this with awk, using -v to provide the pattern:
awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' file
This checks if a line contains a given pattern. If so, it appends a new line to the beginning of it.
See a basic example:
$ cat file
hello
this is some pattern and we are going ahead
bye!
$ awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' file
hello
this is some
pattern and we are going ahead
bye!
Note it will affect to all patterns in a line:
$ cat file
this pattern is some pattern and we are going ahead
$ awk -v patt="pattern" '$0 ~ patt {gsub(patt, "\n"patt)}1' d
this
pattern is some
pattern and we are going ahead
sed -e 's/regexp/\0\n/g'
\0 is the null, so your expression is replaced with null (nothing) and then...
\n is the new line
On some flavors of Unix doesn't work, but I think it's the solution to your problem.
echo "Hello" | sed -e 's/Hello/\0\ntmow/g'
Hello
tmow
This works in MAC for me
sed -i.bak -e 's/regex/xregex/g' input.txt sed -i.bak -e 's/qregex/\'$'\nregex/g' input.txt
Dono whether its perfect one...
After reading all the answers to this question, it still took me many attempts to get the correct syntax to the following example script:
#!/bin/bash
# script: add_domain
# using fixed values instead of command line parameters $1, $2
# to show typical variable values in this example
ipaddr="127.0.0.1"
domain="example.com"
# no need to escape $ipaddr and $domain values if we use separate quotes.
sudo sed -i '$a \\n'"$ipaddr www.$domain $domain" /etc/hosts
The script appends a newline \n followed by another line of text to the end of a file using a single sed command.
In vi on Red Hat, I was able to insert carriage returns using just the \r character. I believe this internally executes 'ex' instead of 'sed', but it's similar, and vi can be another way to do bulk edits such as code patches. For example. I am surrounding a search term with an if statement that insists on carriage returns after the braces:
:.,$s/\(my_function(.*)\)/if(!skip_option){\r\t\1\r\t}/
Note that I also had it insert some tabs to make things align better.
Just to add to the list of many ways to do this, here is a simple python alternative. You could of course use re.sub() if a regex were needed.
python -c 'print(open("./myfile.txt", "r").read().replace("String to match", "String to match\n"))' > myfile_lines.txt
sed 's/regexp/\'$'\n/g'
works as justified and detailed by mojuba in his answer .
However, this did not work:
sed 's/regexp/\\\n/g'
It added a new line, but at the end of the original line, a \n was added.