Bash script to fill sqlite table not working as expected - bash

Question

I've created a Bash script to look through files within a directory, pull data based on an identifier then use that data to fill an sqlite3 table. It seems some information will fill into the same row as the key, and some won't. The script is as follows:
#!/bin/bash
sqlite3 review.sql "CREATE TABLE Review(Review_ID INTEGER PRIMARY KEY, Author TEXT, Date TEXT);"
path="/home/me/Downloads/test/*"
for i in $path
do
total=$(grep -c '<Author>' $i)
count=1
while [ $count -le $total ]
do
date=$(grep -m$count '<Date>' $i | sed 's#<Date>##' | tail -n1)
author=$(grep -m$count '<Author>' $i | sed 's#<Author>##' | tail -n1)
sqlite3 review.sql "INSERT INTO Review(Author,Date) VALUES('$author','$date');"
((count++))
done
done
The Files I'm look through look like this:
<Author>john
<Date>Jan 6, 2009
<Author>jacob
<Date>Dec 26, 2008
<Author>rachael
<Date>Dec 14, 2008
when I query for the primary key and the date attributes i get this as expected
sqlite> SELECT Review_ID, Date FROM Review;
Review_ID Date
---------- ------------
1 Jan 6, 2009
2 Dec 26, 2008
3 Dec 14, 2008
4 Jan 7, 2009
5 Jan 5, 2009
6 Nov 14, 2008
but when i query for the primary key and author i get this
sqlite> SELECT Review_ID, Author FROM Review;
Review_ID Author
---------- ----------
john
jacob
rachael
Jean
kareem
may
Upon doing some more testing, it definitely seems to have problems with some of the text string. For example I tried adding last names, and get this result:
Review_ID Author
---------- ------------
1 john jacob
2 jacob richa
rae simon
Jean jak
5 kareem jabr
6 may flower
It does better, but still doesn't like a couple of them, i thought maybe something to do with three letters, but then "may" wouldn't be showing up, but indeed if I add a letter to "rae" and a letter to "jak" the 3 and 4 do actually show up in the Review_ID column. I noticed the same thing happens if a column contains a "$", like in "$173" for example. The text, I really can't figure out though, there doesn't seem to be an obvious absolute pattern to what it accepts and what it doesn't. I made up the names in order to simplify this post, but just to give a few more examples I'll include a some more examples from what I'm actually working with to show a few more strings that work and ones that don't.
1 everywhereman2
RW53
Marilyn1949
fallriverma
8 SweetwaterMill
AuntSusie006
13 Traveler34NewJe
madmatriarch
2 Savvytourist2
greatvictory
25 Lightsleeper999
strollaround
30 Lucygoosey1985
lesbriggs
3 miguelluna019
lulubaby
1 myassesdragon
tomu023
BrettOcean
46 A TripAdvisor M
dmills1956
julcarl
49 A TripAdvisor M
TSW42
lass=

Answer 1

After a dry run, remove the echo to take the safety off.
awk -v RS='' -v FS='\n?<[^>]+>' '{print $2 ":" $3}' \
/home/me/Downloads/test/* |while IFS=: read author date; do
echo sqlite review.sql "INSERT INTO Review(Author,Date) VALUES('$author','$date');"
done
This is a brittle awk-to-bash solution based on using the regular expression \n?<[^>]+> as Field Separator and a blank line '' as Record Separator. The FS expression means "optional newline followed by a string enclosed by angles." We then output the fields with a simple separator : and read them into bash.
You can make system calls in awk with system(), but it gets very messy very quickly. Better to export the clean data in this case.

Answer 2

The script with awk and printf is:
#!/bin/bash
sqlite3 review.sql "CREATE TABLE Review(Review_ID INTEGER PRIMARY KEY, Author TEXT, Date TEXT);"
path="/home/drew/Downloads/testcases/*"
for i in $path
do
total=$(grep -c '<Author>' $i)
count=1
while [ $count -le $total ]
do
date=$(grep -m$count '<Date>' $i | sed 's#<Date>##' | tail -n1)
author=$(grep -m$count '<Author>' $i | sed 's#<Author>##' | tail -n1 | awk '{printf "- %s -", $1}')
echo $author
((count++))
done
done
I'm not sure, but I feel like I shouldn't have to add echo to it with printf, but without it nothing prints. With the echo I get the following output:
-Jeanjakey
- kareem -
- may -
- RW53 -
-Marilyn1949
-AuntSusie006
-madmatriarch
-strollaround
-lulubaby
-tomu023
-julcarl
-slass
It seems to somewhat work, but the spacing disappears, any last names disappear, and it seems to do different things to different inputs. With the string concatenation I use the script:
#!/bin/bash
sqlite3 review.sql "CREATE TABLE Review(Review_ID INTEGER PRIMARY KEY, Author TEXT, Date TEXT);"
path="/home/drew/Downloads/testcases/*"
for i in $path
do
total=$(grep -c '<Author>' $i)
count=1
while [ $count -le $total ]
do
date=$(grep -m$count '<Date>' $i | sed 's#<Date>##' | tail -n1)
author2="- "
author2+=$(grep -m$count '<Author>' $i | sed 's#<Author>##' | tail -n1)
author2+=" -"
echo $author
((count++))
done
done
with this I get the output:
-Jeanjakey
-kareem jabron
-may flow she
-RW53
-Marilyn1949
-AuntSusie006
-madmatriarch
-strollaround
-lulubaby
-tomu023
-julcarl
-slass
and a string reassignment:
author2=$(grep -m$count '<Author>' $i | sed 's#<Author>##' | tail -n1)
author2="- $author2 - "
gives the same output.