Replace spaces between two strings with symbol using sed - bash

I have string like this:
20.07.2010|Berlin|id 100|bd-22.10.94|Marry Scott Robinson|msc#gmail.com
I need to replace whitespaces only between "Marry Scott Robinson" with "|". So to have bd-22.10.94|Marry|Scott|Robinson|
There many of such rows, so problem is in replace whitespace only between "bd-" and vertical line after name.

I'll assume that the name is always on the fifth column :
awk 'BEGIN{FS=OFS="|"}{gsub(/ /,OFS,$5)}1' file
If it is not the case, you can do :
awk 'BEGIN{FS=OFS="|"}{for(i=1;i<=NF;i++){if($i ~ /bd-/){break}};gsub(/ /,OFS,$(i+1))}1' file
Returns :
20.07.2010|Berlin|id 100|bd-22.10.94|Marry|Scott|Robinson|msc#gmail.com

Perl to the rescue!
perl -lne '($before, $change, $after) = /(.*\|bd-.*?\|)(.*?)(\|.*)/;
print $before, $change =~ s/ /|/gr, $after' -- file
-n reads the input line by line, running the code for each line
-l removes newlines from input and adds them to output
the first line populates three variables by values captured from the line. $before contains verything up to the first | after bd-; $change contains what follows up to the next |, and $after contains the rest.
s/ /|/gr replaces spaces by pipes (/g for "all of them") and returns (/r) the result.

This might work for you (GNU sed):
sed 's/[^|]*/\n&\n/5;:a;s/\(\n[^\n ]*\) /\1\|/;ta;s/\n//g' file
Sometimes to fix a problem we must erect scaffolding, then fix the original problem and finally remove the scaffolding.
Here we need to isolate the field by surrounding it by newlines.
Remove the spaces between the newlines by looping until failure.
Finally, remove the scaffolding i.e. the introduced newlines.

Another perl version:
$ perl -F'\|' -ne '$F[4] =~ tr/ /|/; print join("|", #F)' foo.txt
20.07.2010|Berlin|id 100|bd-22.10.94|Marry|Scott|Robinson|msc#gmail.com
Same basic idea as Corentin's first awk example. Split each line into columns based on |, replace spaces in the 5th one with |'s, print the re-joined lines.

Related

Replace single character in fasta header with awk or sed

I am working in bash with a fasta file with headers that begin with a ">" and end with either a "C" or a "+". Like so:
>chr1:35031657-35037706+
GGTGGACTAGCCAGTGAATGTCAACGCGTCCCTA
CCTAAGGCGATATCCGCAGCCGCCCGCGTCCCTA
>chr1:71979382-71985425C
agattaaatgaactattacacataaagtgcttac
ttacacataaagtgcttacgaactattacaggga
I'd like to use awk (gsub?) or sed to change the last character of the header to a "+" if it is a "C". Basically I want all of the sequences to end in "+". No C's.
Desired output:
>chr1:35031657-35037706+
GGTGGACTAGCCAGTGAATGTCAACGCGTCCCTA
CCTAAGGCGATATCCGCAGCCGCCCGCGTCCCTA
>chr1:71979382-71985425+
agattaaatgaactattacacataaagtgcttac
ttacacataaagtgcttacgaactattacaggga
Nothing needs to change with the sequences. I think this is pretty straight forward, but I'm struggling to use other posts to do this myself. I know that awk '/^>/ && /C$/{print $0}' will print the headers than begin with ">" and end with "C", but I'm not sure how to replace all of those "C"s with "+"s.
Thanks for your help!
I think this would be easier to do in sed:
sed '/^>/ s/C$/+/'
Translation: on lines starting with ">", replace "C" at the end of the line with "+". Note that if the "C" isn't matched, there isn't an error, it just doesn't replace anything. Also, unlike awk, sed automatically prints each line after processing it.
If you really want to use awk, the equivalent would be:
awk '/^>/ {sub("C$","+",$0)}; {print}'
Use this Perl one-liner:
perl -pe 's{^(>.*)C$}{$1+}' input.fasta > output.fasta
Or, to change the file in-place:
perl -i.bak -pe 's{^(>.*)C$}{$1+}' input.fasta
The Perl one-liner uses these command line flags:
-e : Tells Perl to look for code in-line, instead of in a file.
-p : Loop over the input one line at a time, assigning it to $_ by default. Add print $_ after each loop iteration.
-i.bak : Edit input files in-place (overwrite the input file). Before overwriting, save a backup copy of the original file by appending to its name the extension .bak. If you want to skip writing a backup file, just use -i and skip the extension.
s{^(>.*)C$}{$1+} : Change the line that starts with > (= fasta header) and ends with C to the same line with C changed to +.
^ marks the beginning of the line and $ marks the end of the line. .* means any character repeated 0 or more times. (>.*) captures the pattern inside, which is the entire line minus the C, and stores it in capture variable $1.
SEE ALSO:
perldoc perlrun: how to execute the Perl interpreter: command line switches
perldoc perlre: Perl regular expressions (regexes)
perldoc perlre: Perl regular expressions (regexes): Quantifiers; Character Classes and other Special Escapes; Assertions; Capture groups
perldoc perlrequick: Perl regular expressions quick start
You might harness GNU AWK for this task following way, let file.txt content be
>chr1:35031657-35037706+
GGTGGACTAGCCAGTGAATGTCAACGCGTCCCTA
CCTAAGGCGATATCCGCAGCCGCCCGCGTCCCTA
>chr1:71979382-71985425C
agattaaatgaactattacacataaagtgcttac
ttacacataaagtgcttacgaactattacaggga
then
awk 'BEGIN{FPAT=".";OFS=""}$1==">"{$NF="+"}{print}' file.txt
gives output
>chr1:35031657-35037706+
GGTGGACTAGCCAGTGAATGTCAACGCGTCCCTA
CCTAAGGCGATATCCGCAGCCGCCCGCGTCCCTA
>chr1:71979382-71985425+
agattaaatgaactattacacataaagtgcttac
ttacacataaagtgcttacgaactattacaggga
Explanation: I inform GNU AWK that field is any single character using FPAT and output field separator is empty string using OFS. For each line where 1st field, that is 1st character is > I change value of last field ($NF) to +. Note this is applied also to headers ending with + but this is not problem as it changes + to +. Each line, changed or not, is printed.
(tested in GNU Awk 5.0.1)

Align numbers using only sed

I need to align decimal numbers with the "," symbol using only the sed command. The "," should go in the 5th position. For example:
183,7
2346,7
7,999
Should turn into:
183,7
2346,7
7,999
The maximum amount of numbers before the comma is 4. I have tried using this to remove spaces:
sed 's/ //g' input.txt > nospaces.txt
And then I thought about adding spaces depending on the number of digits before the comma, but I don't know how to do this using only sed.
Any help would be appreciated.
Assuming that there is only one number on each line; that there are at most four digits before the ,, and that there is always a ,:
sed 's/[^0-9,]*\([0-9]\+,[0-9]*\).*/ \1/;s/.*\(.....,.*\)/\1/;'
The first s gets rid of everything other than the (first) number on the line, and puts four spaces before it. The second one deletes everything before the fifth character prior to the ,, leaving just enough spaces to right justify the number.
The second s command might mangle input lines which didn't match the first s command. If it is possible that the input contains such lines, you can add a conditional branch to avoid executing the second substitution if the first one failed. With Gnu sed, this is trivial:
sed 's/[^0-9,]*\([0-9]\+,[0-9]*\).*/ \1/;T;s/.*\(.....,.*\)/\1/;'
T jumps to the end of the commands if the previous s failed. Posix standard sed only has a conditional branch on success, so you need to use this circuitous construction:
sed 's/[^0-9,]*\([0-9]\+,[0-9]*\).*/ \1/;ta;b;:a;s/.*\(.....,.*\)/\1/;'
where ta (conditional branch to a on success) is used to skip over a b (unconditional branch to end). :a is the label referred to by the t command.
if you change your mind, here is an awk solution
$ awk -F, 'NF{printf "%5d,%-d\n", $1,$2} !NF' file
183,7
2346,7
7,999
set the delimiter to comma and handle both parts as separate fields
Try with this:
gawk -F, '{ if($0=="") print ; else printf "%5d,%-d\n", $1, $2 }' input.txt
If you are using GNU sed, you could do as below
sed -r 's/([0-9]+),([0-9]+)/printf "%5s,%d" \1 \2/e' input.txt

Adding a new line to a text file after 5 occurrences of a comma in Bash

I have a text file that is basically one giant excel file on one line in a text file. An example would be like this:
Name,Age,Year,Michael,27,2018,Carl,19,2018
I need to change the third occurance of a comma into a new line so that I get
Name,Age,Year
Michael,27,2018
Carl,19,2018
Please let me know if that is too ambiguous and as always thank you in advance for all the help!
With Gnu sed:
sed -E 's/(([^,]*,){2}[^,]*),/\1\n/g'
To change the number of fields per line, change {2} to one less than the number of fields. For example, to change every fifth comma (as in the title of your question), you would use:
sed -E 's/(([^,]*,){4}[^,]*),/\1\n/g'
In the regular expression, [^,]*, is "zero or more characters other than , followed by a ,; in other words, it is a single comma-delimited field. This won't work if the fields are quoted strings with internal commas or newlines.
Regardless of what Linux's man sed says, the -E flag is an extension to Posix sed, which causes sed to use extended regular expressions (EREs) rather than basic regular expressions (see man 7 regex). -E also works on BSD sed, used by default on Mac OS X. (Thanks to #EdMorton for the note.)
With GNU awk for multi-char RS:
$ awk -v RS='[,\n]' '{ORS=(NR%3 ? "," : "\n")} 1' file
Name,Age,Year
Michael,27,2018
Carl,19,2018
With any awk:
$ awk -v RS=',' '{sub(/\n$/,""); ORS=(NR%3 ? "," : "\n")} 1' file
Name,Age,Year
Michael,27,2018
Carl,19,2018
Try this:
$ cat /tmp/22.txt
Name,Age,Year,Michael,27,2018,Carl,19,2018,Nooka,35,1945,Name1,11,19811
$ echo "Name,Age,Year"; grep -o "[a-zA-Z][a-zA-Z0-9]*,[1-9][0-9]*,[1-9][0-9]\{3\}" /tmp/22.txt
Michael,27,2018
Carl,19,2018
Nooka,35,1945
Name1,11,1981
Or, ,[1-9][0-9]\{3\} if you don't want to put [0-9] 3 more times for the YYYY part.
PS: This solution will give you only YYYY for the year (even if the data for YYYY is 19811 (typo mistakes if any), you'll still get 1981
You are looking for 3 fragments, each without a comma and separated by a comma.
The last fields can give problems (not ending with a comma and mayby only two fields.
The next command looks fine.
grep -Eo "([^,]*[,]{0,1}){0,3}" inputfile
This might work for you (GNU sed):
sed 's/,/\n/3;P;D' file
Replace every third , with a newline, print ,delete the first line and repeat.

sed print more than one matches in a line

I have a file, including some strings and variables, like:
${cat.mouse.dog}
bird://localhost:${xfire.port}/${plfservice.url}
bird://localhost:${xfire.port}/${spkservice.synch.url}
bird://localhost:${xfire.port}/${spkservice.asynch.request.url}
${soabp.protocol}://${hpc.reward113.host}:${hpc.reward113.port}
${configtool.store.folder}/config/hpctemplates.htb
I want to print all the strings between "{}". In some lines there are more than one such string and in this case they should remain in the same line. The output should be:
cat.mouse.dog
xfire.port plfservice.url
xfire.port spkservice.synch.url
xfire.port spkservice.asynch.request.url
soabp.protocol hpc.reward113.host hpc.reward113.port
configtool.store.folder
I tried the following:
sed -n 's/.*{//;s/}.*//p' filename
but it printed only the last occurrence of each line. How can I get all the occurrences, remaining in the same line, as in the original file?
This might work for you (GNU sed):
sed -n 's/${/\n/g;T;s/[^\n]*\n\([^}]*\)}[^\n]*/\1 /g;s/ $//p' file
Replace all ${ by newlines and if there are non then move on as there is nothing to process. If there are newlines then remove non-newline characters to the left and non-newline characters to the right of the next } globally. To finish off remove the extra space introduced in the RHS of the global substitution.
If you're not against awk, you can try the following:
awk -v RS='{|}' -v ORS=' ' '/\n/{printf "\n"} (NR+1)%2' file
The record separator RS is set to either { or }. This splits the wanted pattern from the rest.
The script then displays 1 record out of 2 with the statement (NR+1)%2.
In order to keep the alignment as expected, the output record separator is set to a space ORS=' ' and everytime a newline is encountered this statement /\n/{printf "\n"} inserts one.

Explained shell statement

The following statement will remove line numbers in a txt file:
cat withLineNumbers.txt | sed 's/^.......//' >> withoutLineNumbers.txt
The input file is created with the following statement (this one i understand):
nl -ba input.txt >> withLineNumbers.txt
I know the functionality of cat and i know the output is written to the 'withoutLineNumbers.txt' file. But the part of '| sed 's/^.......//'' is not really clear to me.
Thanks for your time.
That sed regular expression simply removes the first 7 characters from each line. The regular expression ^....... says "Any 7 characters at the beginning of the line." The sed argument s/^.......// substitutes the above regular expression with an empty string.
Refer to the sed(1) man page for more information.
that sed statement says the delete the first 7 characters. a dot "." means any character. There is an even easier way to do this
awk '{print $2}' withLineNumbers.txt
you just have to print out the 2nd column using awk. No need to use regex
if your data has spaces,
awk '{$1="";print substr($0,2)}' withLineNumbers.txt
sed is doing a search and replace. The 's' means search, the next character ('/') is the seperator, the search expression is '^.......', and the replace expression is an empty string (i.e. everything between the last two slashes).
The search is a regular expression. The '^' means match start of line. Each '.' means match any character. So the search expression matches the first 7 characters of each line. This is then replaced with an empty string. So what sed is doing is removing the first 7 characters of each line.
A more simple way to achieve the same think could be:
cut -b8- withLineNumbers.txt > withoutLineNumbers.txt

Resources