Parsing column data using sed - bash

Question

My input file:
SMOKE_TEST_FIMS,"['a', 'b', 'c']",2015-08-01 14:00:00+0000,100
4.AIQM-B,,2015-04-16 12:04:21+0000,102
000TEST2,['1.034820'],2015-11-19 05:00:00+0000,130
I want to parse the string in such a way that output file will look like below:
Expected output:
'SMOKE_TEST_FIMS', 'a', '2015-08-01 14:00:00+0000','100'
'SMOKE_TEST_FIMS','b', '2015-08-01 14:00:00+0000','100'
'SMOKE_TEST_FIMS','c', '2015-08-01 14:00:00+0000','100'
'4.AIQM-B','','2015-04-16 12:04:21+0000','102'
'000TEST2','1.034820','2015-11-19 05:00:00+0000','130'
I was able to parse the single column data ['a','b','c'] to
'a'
'b'
'c'
sed -i "s/ *\"/'/g;s/ *[^0-9]*\('[^']*'\)\]*'*/\1/g;s/\(.\)''/\1'\n'/g;" updatebomStatement2.cql

Answer 1

If you are are ok with an gnu awk solution here is such a script:
script.awk
BEGIN { FPAT = "(\"[^\"]+\")|(\\[[^\\]]+\\])|([^,]*)"
OFS = ","
}
{ if ( $2~/\[[^\]]+/ ) {
# sanitize input: strip ", [, ]:
gsub(/[\[\]\"]/, "", $2)
# split at "," into parts: and print them
split($2, parts, ",")
for( ind in parts ) {
# further normalize input
gsub(/^ ?'/, "", parts[ind])
gsub(/'$/, "", parts[ind])
tmp=sprintf("'%s','%s','%s','%s'", $1, parts[ind], $3, $4)
print tmp
}
}
else {
tmp=sprintf("'%s','%s','%s','%s'", $1, $2, $3, $4)
print tmp
}
}
Run it like this: awk -f script.awk yourfile.
Imho gnu awk with its FPAT feature and its control statements is much better suited for your requirements than sed.
The first line with the FPAT describes what mades up a field in your input. It is either
something inside double quotes "
something inside brackets [ ... ]
or something separated by comma
The if statement matches that bracket case which has to be split into several lines.

Answer 2

sed is for simple substitutions on individual lines, that is all. For anything more interesting you should be using awk:
$ cat tst.awk
BEGIN { FS=",?\"?[][]\"?,?"; OFS="," }
{
if (split($2,a,/\047/)) {
for (j=2; j in a; j+=2) {
$2 = a[j]
prt()
}
}
else {
prt()
}
}
function prt( out) {
out = "\047" $0 "\047"
gsub(OFS,"\047,\047",out)
print out
}
$ awk -f tst.awk file
'SMOKE_TEST_FIMS','a','2015-08-01 14:00:00+0000','100'
'SMOKE_TEST_FIMS','b','2015-08-01 14:00:00+0000','100'
'SMOKE_TEST_FIMS','c','2015-08-01 14:00:00+0000','100'
'4.AIQM-B','','2015-04-16 12:04:21+0000','102'
'000TEST2','1.034820','2015-11-19 05:00:00+0000','130'
or building on #karakfa's idea:
$ cat tst.awk
BEGIN { FS="([][ \"\047])*,([][ \"\047])*"; OFS="\047,\047" }
{
for(i=2; i<=(NF-2); i++) {
print "\047" $1, $i, $(NF-1), $NF "\047"
}
}
$ awk -f tst.awk file
'SMOKE_TEST_FIMS','a','2015-08-01 14:00:00+0000','100'
'SMOKE_TEST_FIMS','b','2015-08-01 14:00:00+0000','100'
'SMOKE_TEST_FIMS','c','2015-08-01 14:00:00+0000','100'
'4.AIQM-B','','2015-04-16 12:04:21+0000','102'
'000TEST2','1.034820','2015-11-19 05:00:00+0000','130'

Answer 3

alternative hacky awk
$ awk -F, -v OFS=, -v q="'" '{gsub(/[ "\]\[]/, "");
for(i=2;i <=NF-2; i++)
{$i=$i?$i:q q;
print q $1 q, $i, q $(NF-1) q,q $NF q}}' file
'SMOKE_TEST_FIMS','a','2015-08-0114:00:00+0000','100'
'SMOKE_TEST_FIMS','b','2015-08-0114:00:00+0000','100'
'SMOKE_TEST_FIMS','c','2015-08-0114:00:00+0000','100'
'4.AIQM-B','','2015-04-1612:04:21+0000','102'
'000TEST2','1.034820','2015-11-1905:00:00+0000','130'