I have a folder with the name Translated_cds.
in this folder, there are 52 text files. these are FASTA files that have information about proteins.
>lcl|NZ_JPMI01000003.1_prot_WP_043388330.1_1 [locus_tag=Q664_RS00010] [protein=HAMP domain-containing protein] [protein_id=WP_043388330.1] [location=complement(30..1904)] [gbkey=CDS]
MRIRTRLLLLLIVTAAVPTLAVGLLAWRDAERALSEAVAEQHRRTALAEAEHAATHVLSLATELGGALVHQEPLELGPSE
AQEFLIRVFLRRDRIAQVGLFDARGQLTASVFVDDPEAFARQEPQFRRHDTVAAGEVEDFQRRASELLSQVPEGRAYAIS
APYLTGVRRRPAVVVAARAPGTRTGGLAAELGLEELSQRLAARGVGDERVFLLDGAGRLLLDGEPERERHEDFTGKLPGA
VGARQTGLAAYEEEGRAWLAAYSPVPELGWVAVVARPREAALAPLHALARSTYGVLGLTLLGVLALALMLARALARPIAR
LAEGARALARGNLAHRISLKRRDELGDLARAFNDMGQALEQAHRELLGFNEQLAAQVEERTRELQQTQVQLSRSQRLAAM
GDLAAGMAHEMNNPLAAVLGNVQLMLMDLPKEDPSHRMLGTVHQQAQRIASIVRELQLLSERQQLGRLPLDLHRMLQRVL
ESRCAELSQVGVHVDCRFHPGEVKVLGDTQALGDVLGRLLGNALNAMRDRPERNLVLSTQVVDAEVVRVEMKDTGRGIAR
EHLERIFNPFFTTKQQWTGKGLSLAVCHRVIEDHGGTITLDSVEGVGTTVTLVLPAAPASSGLV
the line starting with > (called the header)is present in all the files. I want to replace the gap ' ' in the headers with _.
till now i have tried this
sed -i 's/ /_/g' Translated_cds*
We can lead with /^>/ to gate the substitution so that it isolates to the pattern we are interested in:
sed -i -e '/^>/ s/ /_/g' Translated_cds*
My test:
echo '>lcl|NZ_JPMI01000003.1_prot_WP_043388330.1_1 [locus_tag=Q664_RS00010] [protein=HAMP domain-containing protein] [protein_id=WP_043388330.1] [location=complement(30..1904)] [gbkey=CDS]
MRIRTRLLLLLIVTAAVPTLAVGLLAWRDAERALSEAVAEQHRRTALAEAEHAATHVLSLATELGGALVHQEPLELGPSE
AQEFLIRVFLRRDRIAQVGLFDARGQLTASVFVDDPEAFARQEPQFRRHDTVAAGEVEDFQRRASELLSQVPEGRAYAIS
APYLTGVRRRPAVVVAARAPGTRTGGLAAELGLEELSQRLAARGVGDERVFLLDGAGRLLLDGEPERERHEDFTGKLPGA
VGARQTGLAAYEEEGRAWLAAYSPVPELGWVAVVARPREAALAPLHALARSTYGVLGLTLLGVLALALMLARALARPIAR
LAEGARALARGNLAHRISLKRRDELGDLARAFNDMGQALEQAHRELLGFNEQLAAQVEERTRELQQTQVQLSRSQRLAAM
GDLAAGMAHEMNNPLAAVLGNVQLMLMDLPKEDPSHRMLGTVHQQAQRIASIVRELQLLSERQQLGRLPLDLHRMLQRVL
ESRCAELSQVGVHVDCRFHPGEVKVLGDTQALGDVLGRLLGNALNAMRDRPERNLVLSTQVVDAEVVRVEMKDTGRGIAR
EHLERIFNPFFTTKQQWTGKGLSLAVCHRVIEDHGGTITLDSVEGVGTTVTLVLPAAPASSGLV' | sed -e '/^>/ s/ /_/g'
My result:
>lcl|NZ_JPMI01000003.1_prot_WP_043388330.1_1_[locus_tag=Q664_RS00010]_[protein=HAMP_domain-containing_protein]_[protein_id=WP_043388330.1]_[location=complement(30..1904)]_[gbkey=CDS]
MRIRTRLLLLLIVTAAVPTLAVGLLAWRDAERALSEAVAEQHRRTALAEAEHAATHVLSLATELGGALVHQEPLELGPSE
AQEFLIRVFLRRDRIAQVGLFDARGQLTASVFVDDPEAFARQEPQFRRHDTVAAGEVEDFQRRASELLSQVPEGRAYAIS
APYLTGVRRRPAVVVAARAPGTRTGGLAAELGLEELSQRLAARGVGDERVFLLDGAGRLLLDGEPERERHEDFTGKLPGA
VGARQTGLAAYEEEGRAWLAAYSPVPELGWVAVVARPREAALAPLHALARSTYGVLGLTLLGVLALALMLARALARPIAR
LAEGARALARGNLAHRISLKRRDELGDLARAFNDMGQALEQAHRELLGFNEQLAAQVEERTRELQQTQVQLSRSQRLAAM
GDLAAGMAHEMNNPLAAVLGNVQLMLMDLPKEDPSHRMLGTVHQQAQRIASIVRELQLLSERQQLGRLPLDLHRMLQRVL
ESRCAELSQVGVHVDCRFHPGEVKVLGDTQALGDVLGRLLGNALNAMRDRPERNLVLSTQVVDAEVVRVEMKDTGRGIAR
EHLERIFNPFFTTKQQWTGKGLSLAVCHRVIEDHGGTITLDSVEGVGTTVTLVLPAAPASSGLV
If we want only the spaces within the keyword/value tags of the header replaced, then:
sed -i -e '/^>/ s/\([A-Za-z0-9]\) \([[A-Za-z0-9]\)/\1_\2/g' Translated_cds*
Or.... We can clarify a bit with more modern regex:
sed -i -E '/^>/ s/([[:alnum:]]) ([[:alnum:]])/\1_\2/g' Translated_cds*
The result will change only inside the header's keyword/value tags:
>lcl|NZ_JPMI01000003.1_prot_WP_043388330.1_1 [locus_tag=Q664_RS00010] [protein=HAMP_domain-containing_protein] [protein_id=WP_043388330.1] [location=complement(30..1904)] [gbkey=CDS]
MRIRTRLLLLLIVTAAVPTLAVGLLAWRDAERALSEAVAEQHRRTALAEAEHAATHVLSLATELGGALVHQEPLELGPSE
AQEFLIRVFLRRDRIAQVGLFDARGQLTASVFVDDPEAFARQEPQFRRHDTVAAGEVEDFQRRASELLSQVPEGRAYAIS
APYLTGVRRRPAVVVAARAPGTRTGGLAAELGLEELSQRLAARGVGDERVFLLDGAGRLLLDGEPERERHEDFTGKLPGA
VGARQTGLAAYEEEGRAWLAAYSPVPELGWVAVVARPREAALAPLHALARSTYGVLGLTLLGVLALALMLARALARPIAR
LAEGARALARGNLAHRISLKRRDELGDLARAFNDMGQALEQAHRELLGFNEQLAAQVEERTRELQQTQVQLSRSQRLAAM
GDLAAGMAHEMNNPLAAVLGNVQLMLMDLPKEDPSHRMLGTVHQQAQRIASIVRELQLLSERQQLGRLPLDLHRMLQRVL
ESRCAELSQVGVHVDCRFHPGEVKVLGDTQALGDVLGRLLGNALNAMRDRPERNLVLSTQVVDAEVVRVEMKDTGRGIAR
EHLERIFNPFFTTKQQWTGKGLSLAVCHRVIEDHGGTITLDSVEGVGTTVTLVLPAAPASSGLV
For a CI / CD pipeline I would like to replace a placeholder in my nginx.conf file with a generated map which looks like the following:
map $cookie_language $lang {
default en
en en
de de
es es
}
In the nginx.conf file there is the placeholder # REPLACE_ME_WITH_LANGUAGE_MAP which I am replacing with the following command:
sed -i -e "s/# REPLACE_ME_WITH_LANGUAGE_MAP/$languagemap/g" ./ci/nginx.conf
Full script looks like the following:
languagemap='map $cookie_language $lang {'
firstlanguage=$(jq -r '.locales[0]' src/assets/locales.json | jq -r '.build')
languagemap="${languagemap}| default $firstlanguage"
for locale in $(jq -r '.locales[] | #base64' src/assets/locales.json); do
lang=$(echo "$locale" | base64 --decode | jq -r '.build')
languagemap="${languagemap}| $lang $lang"
npm run ci-build -- --output-path ${OUTPUT_PATH}/$lang --configuration=${ANGULAR_CONFIGURATION} --i18n-format=xlf --i18n-file=src/locale/messages.$lang.xlf --i18n-locale=$lang
done
languagemap="${languagemap}|}"
sed -i "t" "s/# REPLACE_ME_WITH_LANGUAGE_MAP/$languagemap/g" ./ci/nginx.conf
Running this always brings up:
+ sed -i tmp s/# REPLACE_ME_WITH_LANGUAGE_MAP/map $cookie_language $lang {| default en| en en| de de|}/g ./ci/nginx.conf
sed: can't find label for jump to `mp'
What's the point?
The 10th line of your full script is
sed -i "t" <replace-command-script> <input-file>
and sed correctly interpters the "t" as a "branch to" command, which throws an error.
Replace this line with the variant you have provided above, namely
sed -i -e '<replace command script>' <input-file>
This should fix the problem.
I have a file in Unix, with data sample like the following:
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
The desired output is
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexico
456|Americas|Canada
567|APAC|Japan
I tried with a few sed commands. I could remove the following: '{', '}', ' " ', ':'
There are 2 issues with the output file
All rows from input appear in single line in the output.
Adding the pipe ('|') as delimiter.
Any pointers are highly appreciated.
I recommend the tool jq (http://stedolan.github.io/jq/); jq is a lightweight and flexible command-line JSON processor.
jq -r '"\(.ID)|\(.Region)|\(.Location)"' < infile
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Explanation
-r is --raw-output
Through awk,
awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
Example:
$ cat file
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
$ awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
EXplanation:
-F'"' Sets " as Field Separator value.
OFS="|" Sets | as Output Field Separator value.
Atfirst, awk would execute the function inside the BEGIN block. It helps to print the header section.
This sed one-liner does what you want. It's capturing the field values using parenthesized expressions, and then putting them into the output using \1, \2, and \3.
s/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/
Invoke it like:
$ sed -f one-liner.sed input.txt
Or you can invoke it within a Bash script, producing the header:
echo 'ID|Region|Location'
sed -e 's/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/' $input
It is a JSON file so it is best to use a JSON parser. Here is a perl implementation of it.
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
open my $fh, '<', 'path/to/your/file';
#keys of your structure
my #key = qw(ID Region Location);
print join ("|", #key), "\n";
#iterate over your file, decode it and print in order of your key structure
while (my $json = <$fh>) {
my $text = decode_json($json);
print join ("|", map { $$text{$_} } #key ),"\n";
}
Output:
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Using sed as follows
Command line
echo "my_string" |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
or
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g' my_file
I tried this in a terminal as follows:
echo '{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}' |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
Output
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Many thanks for your response and the pointers/ solutions did help a lot.
For some mysterious reasons, I couldn't get any sed commands work. So, I devised my own solution. Although it's not elegant, it's still worked.
Here is the script I prepared which resolved the issue.
#!/bin/bash
# ource file path.
infile=/home/exfile.txt
# remove if these temp file exist already.
rm ./efile.txt ./xfile.txt ./yfile.txt ./zfile.txt
# removing the curly braces from input file.
cat exfile.txt | cut -d "{" -f2 | cut -d "}" -f1 >> ./efile.txt
# setting input file name to different value.
infile=./efile.txt
# remove double quotes from the file.
while IFS= read -r line
do
echo $line | sed 's/\"//g' >> ./xfile.txt
done < "$infile"
# creating another temp file.
infile2=./xfile.txt
# remove colon from file.
while IFS= read -r line
do
echo $line | sed 's/\:/,/g' >> ./yfile.txt
done < "$infile2"
# set input file path to new temp file.
infile3=yfile.txt
# initialize variables to hold header column values.
t1=0
t3=0
t5=0
# read each of the line to extract header row. Exit loop after reading 1st row.
once=1
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
"$f1 $f2 $f3 $f4 $f5 $f6"
t1=$f1
t3=$f3
t5=$f5
if [ "$once" -eq 1 ]; then
break
fi
done < "$infile3"
# Read each of the line from input file. Write only the value to another output file.
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
echo "$f2|$f4|$f6" >> ./zfile.txt
done < "$infile3"
# insert the header column row into the file generated in the step above.
frstline="$t1|$t3|$t5"
sed -i '1i ID|Region|Location' ./zfile.txt
I'm using a shell script that uses sed
I have this sentence in a variable
kpi="ENFRIADORA CONDENSADA POR AIRE/AIRE-AIRE/AGUA - MENSUAL"
if in a console I put
echo $kpi | sed 's/\//\\\//g'
I see
ENFRIADORA CONDENSADA POR AIRE\/AIRE-AIRE\/AGUA - MENSUAL
That is what I want, but when I put this in a script like this
echo start
echo $kpi2
kpi_2=`echo $kpi2 | sed 's/\//\\\//g'`
echo $kpi_2
echo end
That I see is :
start
ENFRIADORA CONDENSADA POR AIRE/AIRE-AIRE/AGUA - MENSUAL
sed: -e expresion #1, character 9: unknown option to `s'
end
Any suggestion?
$> kpi_2=$( echo $kpi | sed 's/\//\\\//g' )
$> echo $kpi_2
ENFRIADORA CONDENSADA POR AIRE\/AIRE-AIRE\/AGUA - MENSUAL
This works for me:
kpi_2=`echo $kpi2 | sed 's:\/:\\\/:g'`
I would like to rename a linux file to a filename that is legal in windows. It should not be longer than is allowed and should not have characters that are not allowed in windows. Sometimes I copy the title from papers to a filename and they have special characters such as –, ®, or ?
Also there is there are some kind of characters sometimes at the ends of lines generated when copying and pasting a title from a pdf. You can see them when using sed -n 'l':
echo 'Estrogen receptor agonists and estrogen attenuate TNF-α induced
α
apoptosis in VSC4.1 motoneurons.pdf' | sed -n 'l'
Estrogen receptor agonists and estrogen attenuate TNF-\316\261 induce\
d$
\316\261$
apoptosis in VSC4.1 motoneurons.pdf$
or
echo 'A synthetic review of the five molecular Sorlie’s subtypes in
breast cancer' | sed -n 'l'
A synthetic review of the \357\254\201ve molecular Sorlie\342\200\231\
s subtypes in$
breast cancer$
I have started a script but it is not elegant and incomplete. Has someone done something like this already or is there a fast elegant way to do it?
fn2win="$1"
testFn=$(echo "$fn2win" | sed -n 'l')
#SPEC_CHAR="ÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞàáâãäåçèéêëìíîïðñòóôõöøùúûüýþÿ"
#NORM_CHAR="AAAAAACEEEEIIIIDNOOOOOOUUUUYPaaaaaaceeeeiiiionoooooouuuuyby"
#SPEC_LOW_CHAR="aàáâãäåāăąbḃcćçčĉċdḑďḋđeèéěêëēĕęėfḟƒgǵģǧĝğġǥhĥħiìíîĩïīĭįıjĵkḱķǩlĺļľłmṁnńņňñoòóôõöōŏøpṗqrŕŗřsśşšŝṡſtţťṫŧuùúûũüůūŭųvwẁẃŵẅxyỳýŷÿzźžż"
#NORM_LOW_CHAR="aaaaaaaaaabbccccccdddddeeeeeeeeeefffgggggggghhhiiiiiiiiiijjkkkklllllmmnnnnnoooooooooppqrrrrssssssstttttuuuuuuuuuuvwwwwwxyyyyyzzzz"
#SPEC_CAP_CHAR="AÀÁÂÃÄÅĀĂĄBḂCĆÇČĈĊDḐĎḊĐEÈÉĚÊËĒĔĘĖFḞGǴĢǦĜĞĠǤHĤĦIÌÍÎĨÏĪĬĮİJĴKḰĶǨĸLĹĻĽŁMṀNŃŅŇÑOÒÓÔÕÖŌŎØPṖQRŔŖŘSŚŞŠŜṠTŢŤṪŦUÙÚÛŨÜŮŪŬŲVWẀẂŴẄXYỲÝŶŸZŹŽŻ"
#SPEC_CAP_CHAR="AAAAAAAAAABBCCCCCCDDDDDEEEEEEEEEEFFGGGGGGGGHHHIIIIIIIIIIJJKKKKKLLLLLMMNNNNNOOOOOOOOOPPQRRRRSSSSSSTTTTTUUUUUUUUUUVWWWWWXYYYYYZZZZ"
#sed -e "y/'$SPEC_CHAR'/'$NORM_CHAR'/"
if [ "$fn2win" != "$testFn" ]; then
newLinFn=$(echo "$fn2win" | fromdos | tr "\n" " " |\
sed -e "
s/[?()\[\]=+<>:;©®”,*|]/_/g
s/"$'\t'"/ /g
s/–/-/g
s/’/'/g
s/α/alpha/g
s/β/beta/g
s/µ/micro/g
s/Æ/AE/g
s/Ǽ/AE/g
s/æ/ae/g
s/ǽ/ae/g
s/DZ/DZ/g
s/DŽ/DZ/g
s/Dž/Dz/g
s/Dz/Dz/g
s/dz/dz/g
s/dž/dz/g
s/ff/ff/g
s/fi/fi/g
s/fl/fl/g
s/ffi/ffi/g
s/ffl/ffl/g
s/ſt/ft/g
s/IJ/IJ/g
s/ij/ij/g
s/LJ/LJ/g
s/Lj/Lj/g
s/lj/lj/g
s/NJ/NJ/g
s/Nj/Nj/g
s/nj/nj/g
s/Œ/OE/g
s/œ/oe/g
s/ß/SZ/g
s/\"/_/g
s/[[:cntrl:]]/_/g
s/\ $//g
" |\
fold -s -w 251 | head -1 | sed 's/\ $/.pdf/')
if [ "$fn2win" != "$newLinFn" ]; then
mv "$fn2win" "$newLinFn"
fi
fi
winFn=$(echo "z:"$newLinFn | sed 's/\//\\/g' )
This looks like it should do it: http://pwet.fr/man/linux/commandes/konwert