I have a UTF-8 file which has curly quotes ‘Awaara’ like these and in some places curly quotes are used such as don’t and don't' . The issue arises when trying to convert these curly quotes to single quotes. After converting to single quotes, I am unable to extract the single quotes words 'Awaara' without removing all single quotes used as don't , I'm.
GOAL: Convert curly--> single, remove single quotes yet keep apostrophied single quotes.
Here's the code I have written which convert yet fails to remove words within single quotes:
#!/bin/bash
cat $1 | sed -e "s/\’/'/g" -e "s/\‘/'/g" | sed -e "s/^'/ /g" -e "s/'$/ /g" | sed "s/\…/ /g" | tr '>' ' ' | tr '?' ' ' | tr ',' ' ' | tr ';' ' ' | tr '.' ' ' | tr '!' ' ' | tr '′' ' ' | tr ':' ' ' | sed -e "s/\[/ /g" -e "s/\]/ /g" -e 's/(/ /g' -e "s/)/ /g" | tr ' ' '\n' | sort -u | uniq | tr 'a-z' 'A-Z' >our_vocab.txt
The output is:
'AWAARA ---> Should be AWAARA
25
50
70
800
A
AD
AI
AMITABH
AND
ANYWAY
ARE
BACHCHAN
BECAUSE
BUT
C++
CAN
CHECK
COMPUTER
DEVAKI
DIFFICULT
.
.
.
HOON' --> Should be HOON
You can use
sed -E -e "s/([[:alpha:]]['’][[:alpha:]])|['‘’]/\\1/g" \
-e 's/[][()>?,;.!:]|′|…/ /g' "$1" | tr ' ' '\n' | sort -u | \
tr 'a-z' 'A-Z' > our_vocab.txt
See the online demo.
I merged several tr commands into a single (second) sed command, and the ([[:alpha:]]['’][[:alpha:]])|['‘’] regex removes all '‘’ apostrophes other than those in between letters.
I'm trying to format output in a way that inserts newline characters after each 'line', with lines denoted by double quotes (""). The quotes themselves are temporary and to be stripped in a later step.
Input:
"a",1,"aa""b",2,"bb"
Output:
a,1,aa
b,2,bb
I've tried:
sed 's/""/\n/'
sed 's/""/\/g'
tr '""' '\n'
But tr seems to replace every quote character and sed seems to insert \n as text instead of a newline. What can I do to make this work?
echo '"a",1,"aa""b",2,"bb"' |awk -v RS='""' '{$1=$1} {gsub(/"/,"")}1'
a,1,aa
b,2,bb
or using sed:
echo '"a",1,"aa""b",2,"bb"' |sed -e 's/""/\n/' -e 's/"//g' # OR sed -e 's/""/\n/;s/"//g'
a,1,aa
b,2,bb
awk solution: Here the default record separator is changed from new line to "". So awk will consider the EOL when it hits "".
sed solution: Here first "" are converted into new line and second replacement is to remove " from each line.
neech#nicolaw.uk:~ $ cat file.txt
"a",1,"aa""b",2,"bb"
neech#nicolaw.uk:~ $ sed 's/""/\n/' file.txt | tr -d '"'
a,1,aa
b,2,bb
You seem to be dealing with POSIX sed, which does not have support for the \n notation. Insert an actual new-line into the pattern, either:
sed 's/""/\
/'
Or:
sed 's/""/\'$'\n''/'
E.g.:
sed 's/""/\
/' | tr -d \"
Output:
a,1,aa
b,2,bb
As suggested by George Vasiliou if you have perl you could use:
> echo '"a",1,"aa""b",2,"bb"' | perl -pe 's/""/"\n"/g;s/"//g'
This avoids the non portable sed problem.
Or for a crappy hack version.
Replace the "" with another character and then use tr (since tr should work with \n) to replace it with \n instead then remove the single " after.
So you can get the "" replaced with newline like this:
sed 's/""/#/g' | tr '#' '\n'
Then the rest follows:
> echo '"a",1,"aa""b",2,"bb"'| sed 's/""/#/g' | tr '#' '\n' | sed 's/\"//g'
I am running the below line in a shell script
echo "$(tr -s '\n' ' ' < ${data[1]} | grep -oP '<af:popup.*?"${data[2]}".*?>')"
echo "(tr -s '\n' ' ' < ${data[1]} | grep -oP '<af:popup.*?"${data[2]}".*?>')"
The command is supposed to translate all \n from file ${data[1]} and inside this file a pattern something like this:
(af:popup.*?logicalCostingRecordExistsPopup.*?)
Issue is the first line is returning null data. Just to validate my script, I echoed the command to check what is getting replaced and run in directly in a shell.Output came as below
tr -s '\n' ' ' < hello.jsff | grep -oP '<af:popup.*? logicalCostingRecordExistsPopup.*?>'
When I run it directly in shell, it gives me expected output.
Don't know why it is not giving output when running inside in shell script
Instead of using double quotes, use single quotes.
echo "$(tr -s '\n' ' ' < ${data[1]} | grep -oP '<af:popup.*?'${data[2]}.*?>')"
trying to replace the sapces and underscores in this is just a (test)
I do the following:
echo "this is just a (test)" | sed -e 's/ /_/g' | sed -e 's/(//g' | sed -e 's/)//g'
And this gives me:
this_is_just_a_test
Is there a better way? shorter way of writing it in sed?
You can achieve the same thing using tr:
echo "this is just a (test)" | tr \ _ | tr -d \(\)
The first tr replaces spaces with underscores and the second one deletes all parenthesis.
I am new to shell scripting and i am trying to remove new line character from each line using SED. this is what i have done so far :
printf "{new\nto\nlinux}" | sed ':a;N;s/\n/ /g'
removes only Ist new line character.
I somewhere found this command :
printf "{new\nto\nlinux}" | sed ':a;N;$!ba;s/\n/ /g'
but it gives :"ba: Event not found."
if i do:
printf "{new\nto\nlinux}" | sed ':a;N;s/\n/ /g' | sed ':a;N;s/\n/ /g'
then it gives correct output but i am looking for something better as i am not sure how many new character i will get when i run the script.
incoming stream is from echo or printf or some variable in script.
To remove newlines, use tr:
tr -d '\n'
If you want to replace each newline with a single space:
tr '\n' ' '
The error ba: Event not found is coming from csh, and is due to csh trying to match !ba in your history list. You can escape the ! and write the command:
sed ':a;N;$\!ba;s/\n/ /g' # Suitable for csh only!!
but sed is the wrong tool for this, and you would be better off using a shell that handles quoted strings more reasonably. That is, stop using csh and start using bash.
This might work for you:
printf "{new\nto\nlinux}" | paste -sd' '
{new to linux}
or:
printf "{new\nto\nlinux}" | tr '\n' ' '
{new to linux}
or:
printf "{new\nto\nlinux}" |sed -e ':a' -e '$!{' -e 'N' -e 'ba' -e '}' -e 's/\n/ /g'
{new to linux}
Use perl instead of sed. perl is similar to sed:
ubuntu#ubuntu:/$ printf "{new\nto\nlinux}" | sed 's/\n/ /g'; echo ''
{new
to
linux}
ubuntu#ubuntu:/$ printf "{new\nto\nlinux}" | perl -pe 's/\n/ /g'; echo ''
{new to linux}
ubuntu#ubuntu:/$ echo -e "new\nto\nlinux\ntest\n1\n2 3" | perl -pe 's/\n/_ _/g'; echo ''
new_ _to_ _linux_ _test_ _1_ _2 3_ _
ubuntu#ubuntu:/$