Delete double quotes in csv bash - bash

I’m having the following issue:
I have the csv file with data inside looks like this:
“1,””name””,””surname””,””age””,””city”””
“2,””Peter””,””Parker””,””30””,””NY”””
“3,””marry””,””Jane””,””30””,””NY”””
Is it possible using bash to delete the first and the last double quote from each row and then first and last double quote from each field in it?
To get something like this:
1,”name”,”surname”,”age”,”NY”
3,”marry”,”Jane”,”30”,”NY”
I would be grateful for some hints. Thanks

To get you started:
echo '"1,""name"",""surname"",""age"",""city"""' | sed "s/\"\"/\"/g" | sed 's/^\"\(.*\)\"$/\1/'
OUPUT
You can take that, adjust it to run over a file, line by line (instead of the first echo and output into another file

Presuming your input looks like this:
"1,""name"",""surname"",""age"",""city"""
"2,""Peter"",""Parker"",""30"",""NY"""
"3,""marry"",""Jane"",""30"",""NY"""
Note the actual "'s not the ”” in your code:
You can then sed multiple things and chain them together e.g.
sed -e "s/\"\"\"/\"/g" -e "s/\"\"/\"/g" input.txt
This first replaces the triple quotes """, reducing them to double quotes "" and then reduces them further.
Final output:
"1,"name","surname","age","city"
"2,"Peter","Parker","30","NY"
"3,"marry","Jane","30","NY"
If you have special characters then simply replace them in the code e.g.:
$ cat input.txt
“1,””name””,””surname””,””age””,””city”””
“2,””Peter””,””Parker””,””30””,””NY”””
“3,””marry””,””Jane””,””30””,””NY”””
$ sed -e "s/\”\”\”/\”/g" -e "s/\”\”/\”/g" input.txt
“1,”name”,”surname”,”age”,”city”
“2,”Peter”,”Parker”,”30”,”NY”
“3,”marry”,”Jane”,”30”,”NY”
Though I think this input is a transpose error in your question.

Using sed:
sed 's/^"\(.*\)"$/\1/;s/"\+/"/g' file
The first substitution removes the outer double quote on the whole line.
The second substitution replaces the parameter quote to only one double quote.

Related

Changing a line of text with sed with special characters

The name in the title says it all. However, I'm absolutely the worst with the sed command. So I'm trying to edit the following file:
/var/www/html/phpMyAdmin/config.inc.php
I want to edit the line that says
$cfg['Servers'][$i]['AllowRoot'] = false;
into the following
$cfg['Servers'][$i]['AllowRoot'] = true;
It has so many special characters and whatnot and I have no prior knowledge of how sed works. So here's some commands I've tried to specifically edit that one line.
sed -i "/*.AllowRoot.*/\$cfg['Servers'][\$i]['AllowRoot'] = true;/" /var/www/html/phpMyAdmin/config.inc.php
sed -i "/*.AllowRoot.*/$cfg['Servers'][$i]['AllowRoot'] = true;/" /var/www/html/phpMyAdmin/config.inc.php
# this one finds the line successfully and prints it so I know it's got the right string:
sed -n '/AllowRoot/p' /var/www/html/phpMyAdmin/config.inc.php
sed -i "s/'AllowRoot|false'/'AllowRoot|true'/" /var/www/html/phpMyAdmin/config.inc.php
I have absolutely no idea what I'm doing and I'm not learning a whole lot besides the feeling that the last command splits up 'AllowRoot|false' makes sure that both must be present in the sentence to come back as a result. So to my logic, I thought changing the word false into true would make that happen, but nothing. The other commands return... bizarre results at best, one even emptying the file. Or that's one of the commands I had not written down here, I've lost track after 50 attempts. What is the solution here?
The [ and ] need to be escaped to match literal brackets, instead of inadvertently starting a bracket expression. This should work:
$ sed -i "/\$cfg\['Servers'\]\[\$i\]\['AllowRoot'\]/s/false/true/" /var/www/html/phpMyAdmin/config.inc.php
There is not many things to escape in sed. Main problem in your line is / which you have chosen as delimiter (most common, but not required). I suggest you use # and the following will work:
sed -i "s#$cfg['Servers'][$i]['AllowRoot'] = false;<br />#$cfg['Servers'][$i]['AllowRoot'] = true;<br />#g" input.txt
however you need to think about bash interpreter as well. $i and $cfg will be interpreted as variables. My suggestion is that when you want to match a string like this to put the sed expression in a text file like this:
cat allow_root_true.sed
s#['Servers'][]['AllowRoot'] = false;<br />#['Servers'][]['AllowRoot'] = true;<br />#g
and run the command using sed -f like this:
sed -i -f allow_root_true.sed input.txt
Warning -i will change the input file
sed can't do literal string matching which is why you need to escape so many characters (see Is it possible to escape regex metacharacters reliably with sed), but awk can:
$ awk -v str="\$cfg['Servers'][\$i]['AllowRoot']" 'index($0,str){sub(/false/,"true")} 1' file
//some text here
$cfg['Servers'][$i]['AllowRoot'] = true;<br />
//some more text here
Run code snippetHide resultsExpand snippet
In the above we only have to escape the $s to protect them from the shell since the string is enclosed in "s to allow it to include 's.

using variables in regex?

Part of a shell script that I am creating takes a plain text list of files...
11111.jpg
22222.jpg
33333.jpg
...and appends a user-defined prefix that is stored in a variable to create a list of paths that looks like this:
user/defined/prefix/11111.jpg
user/defined/prefix/22222.jpg
user/defined/prefix/33333.jpg
I am attempting to use sed to add the prefix in this manner:
sed -e 's/^/prefix/' oldFile > newFile.new
The variable is getting assigned correctly:
echo $selectedPrefix
user/defined/prefix
Put no combinations of single quotes, double quotes of whatever seem to get sed to use the ACTUAL value of the variable instead of just the variable name.
sed -e 's/^/$selectedPrefix/' oldFile > newFile.new
Yields:
$selectedPrefix11111.jpg
$selectedPrefix22222.jpg
$selectedPrefix33333.jpg
Help! I'm sure the solution is simple but I feel like I've tried everything....
As mentionned by Cyrus, you need to used " (double quote) instead ' (single quote) if you want the variable replacement because single quoted string are interpreted literally so it doesn't see $selectedPrefix as a variable but as the string value of $selectedPrefic hence what you saw.
Since you are working with paths in you sed, you are correct in assuming that you should use a different separator for your sed comment. I usually prefer using | but ~ would also work.
so basically you could have:
sed -e "s~^~$selectedPrefix~" oldFile > newFile.new
This code would solve your problem:
selectedPrefixEscaped="$(echo "$selectedPrefix" | sed 's/\//\\\//g')" && sed -e "s/^/$selectedPrefixEscaped/" oldFile > newFile.new
Just using a different delimiter on sed would leave you open to problems when (if) the path contains the new delimiter (ex.: /folder/folder#5/file.txt would be problematic if using # as sed delimiter).

bash - sed doesn't replace all semicolons

I have three variables I write into a text file. For now it is into a variable, because try and error is faster like this. These three variables are produced by the script before and each variable has a column with values in them. Variable one for example looks like this:
hour
minute
minute
day
I put them together using this code:
New_fileloc=$(paste <(echo "$Grabinterval") <(echo "$Filelocation") <(echo "$Time") --delimiters ';' | sed -e 's/^\|$/"/g' -e 's/\;/";"/')
At the end I need each line in each column in double quotes and separated by a semicolon. At the moment I am doing that using sed in that one liner. And that works mostly fine. My output looks like this:
"hour";"A_path/to/somewhere/;2016-02-10 17:07:00Z"
"minute";"A_path/to/somewhere/;2016-01-29 17:26:20Z"
"minute";"A_path/to/somewhere/;2016-01-29 17:26:20Z"
"day";"A_path/to/somewhere/;2016-01-29 00:07:00Z"
The first semicolon gets replaces with ";", but the second one in each line does not. I have no idea why.
In your last command to sed you are missing a g
s/\;/";"/g
Your original command with 's/\;/";"/' will only make one replacement, the first.

add square brackets in each line of the file using bash

I would like to add square brackets to each and every line of my file.
I was using sed -i "s/\(.*\)/[\1]/" file_name.txt command but I see that it is inserting 2 times the square brackets and my output (just the first line and it's the same for every single line) looks like this
[[1,1,0,0,0,0,1,0,24,3,0,0,0,0,86,149,149,14,0,0,0,0,32,149,46,16,0,0,1,13,3,33,65,66,0,0,0,0,0,2,149,140,6,0,0,2,62,148,88,24,26,2,0,14,116,148,30,15,1,0,0,1,5,30,56,18,0,0,0,0,0,4,149,46,40,14,0,0,1,34,31,46,149,31,0,2,9,12,1,7,8,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,12,2,0,0,0,0,0,0,0,0,0,0,0,0]]
How do I insert just one single square bracket for each line using sed? Please help. Thanks in advance!
Using awk:
awk '{print "["$0"]"}' file_name.txt
your command is fine and should not add 2 [ unless runnning twice. Also group, in this case, in not necessary, taking the whole pattern & is suffisant
sed -i 's/.*/[&]/' file_name.txt
Maybe, try with a single quote your own, double could aways be interpretated by something else from the shell

sed copy substring from fixed position and copy it in front of line

I'm dealing with many csv files and I can't find a way with sed to select a substring at a fixed position (chars 9-16) and copy it at the beginning of the line.
This is what I have:
ABC09638006924340017;SOME_TEXT;SOME_OTHER_TEXT
This is what I need:
00692434;ABC09638006924340017;SOME_TEXT;SOME_OTHER_TEXT
The following code in sed gives the substring I need (00692434) but overwrites the whole line:
sed 's/^.{8}(.{8}).*/\1/')
I'm already using sed to "clean" the linestrings and inserting some variables, called in a bash script that at the end imports data in postgres. This is why I would prefer to remain within sed, but any hint will be greatly appreciated as I'm not a real expert.
You need to escape the curly braces (\{\}) as well as the parentheses (\(\)) and also append the original string (&) in the replacement:
text="ABC09638006924340017;SOME_TEXT;SOME_OTHER_TEXT"
echo $text | sed "s/^.\{8\}\(.\{8\}\).*/\1;&/"
Output:
00692434;ABC09638006924340017;SOME_TEXT;SOME_OTHER_TEXT
Since you want to extract a fixed-length substring at a fixed position, you could also do this with just bash-builtins:
text="ABC09638006924340017;SOME_TEXT;SOME_OTHER_TEXT"
echo "${text:8:8};$text"
This migth work for you (GNU sed):
sed -r 's/^.{8}(.{8})/\1;&/' file

Resources