I'm having trouble storing column values into shell script variables when these include white spaces, since all the results are split on whitespaces instead of actual column values.
For example, this is what I got now:
set -A SQL_RESULTS_ARRAY `sqlplus -s un/pass#database << EOF
SET ECHO OFF
SET FEED OFF
SET HEAD OFF
SET SPACE 0
SELECT EMAIL_SUBJECT, MAIL_TO FROM EMAIL_TABLE;
EOF`
echo "${SQL_RESULTS_ARRAY[0]}"
echo "${SQL_RESULTS_ARRAY[1]}"
This doesn't work because the value of EMAIL_SUBJECT is an entire sentence, ie "Message subject test", so those echos just end up printing
Message
subject
Instead of
Message subject test
email1#email.com email2#email.com
Basically, how do I end up with only two items in the array (one per column), instead of five items (one per word)? Is this at all possible with a single connection? (I'd rather not start a new connection per column)
EDIT: Another thing, another one of my CLOB columns is EMAIL_BODY, which can basically be any text-- thus I'd rather not have a preset separator, since EMAIL_BODY can have all sorts of commas, pipes, new lines, etc...
The key you're missing is to set the shell's IFS (internal field separator) to be the same as your query results. Here's a ksh session:
$ results="Message subject test,email1#email.com email2#email.com"
$ set -A ary $results
$ for i in 0 1 2 3 4; do print "$i. ${ary[$i]}"; done
0. Message
1. subject
2. test,email1#email.com
3. email2#email.com
4.
$ IFS=,
$ set -A ary $results
$ for i in 0 1 2 3 4; do print "$i. ${ary[$i]}"; done
0. Message subject test
1. email1#email.com email2#email.com
2.
3.
4.
You'll probably want to do something like this:
results=`sqlplus ...`
old_IFS="$IFS"
IFS=,
set -A SQL_RESULTS_ARRAY $results
IFS="$old_IFS
print "${SQL_RESULTS_ARRAY[0]}"
print "${SQL_RESULTS_ARRAY[1]}"
You may try to set COLSEP and separate by its value.
Try adding double quotes using string concatenation in the select statement. Array elements that are quoted permit white space (at least in bash).
read up about the bash's "Internal Field Separator" $IFS
it is set to whitespace by default, which may be causing your problem.
Related
quick overview: I got sqlite3 db which contains following structure and data
Id|Name|Value
1|SomeName1|SomeValue1
2|SomeName2|SomeValue2
3|SomeName3|SomeValue3
(continuation of SomeValue3 in here, after ENTER)
Problem is with iteration trough "Value" column, I'm using that code:
records=(`sqlite3 database.db "SELECT Value FROM Values"`)
for record in "${records[#]}"; do
echo $record
done
Problem is there should three values using that iteration, but it is showing four.
As result I received:
1 step of loop - SomeValue1
2 step of loop - SomeValue2
3 step of loop - SomeValue3
4 step of loop - (continuation of SomeValue3 in here, after ENTER)
it should end at third step and just show with line break up something like that:
3 step of loop - SomeValue3
(continuation of SomeValue3 in here, after ENTER)
Any suggestion how I can handle it with bash?
Thank you in advance!
Instead of relying on word splitting to populate an array with the result of a command, it's much more robust to use the readarray builtin, or read a result at a time with a loop. Examples of both follow, using sqlite3's ascii output mode, where rows are separated by the byte 0x1E and columns in the rows by 0x1F. This allows the literal newlines in your data to be easily accepted.
#!/usr/bin/env bash
# The -d argument to readarray and read changes the end-of-line character
# from newline to, in this case, ASCII Record Separator
# Uses the `%q` format specifier to avoid printing the newline
# literally for demonstration purposes.
echo "Example 1"
readarray -d $'\x1E' -t rows < <(sqlite3 -batch -noheader -ascii database.db 'SELECT value FROM "Values"')
for row in "${rows[#]}"; do
printf "Value: %q\n" "$row"
done
echo "Example 2 - multiple columns"
while IFS=$'\x1F' read -d $'\x1E' -ra row; do
printf "Rowid: %d Value: %q\n" "${row[0]}" "${row[1]}"
done < <(sqlite3 -batch -noheader -ascii database.db 'SELECT rowid, value FROM "Values"')
outputs
Example 1
Value: SomeValue1
Value: SomeValue2
Value: $'SomeValue2\nand more'
Example 2 - multiple columns
Rowid: 1 Value: SomeValue1
Rowid: 2 Value: SomeValue2
Rowid: 3 Value: $'SomeValue2\nand more'
See Don't Read Lines With for for more on why your approach is bad.
Since VALUES is a SQL keyword, when using it as a table name (Don't do that!) it has to be escaped by double quotes.
Your problem here is the IFS (internal field seperator) in Bash, which the for -loop counts as a new record.
Your best option is to remove the linefeed in the select statement from sqlite, e.g:
records=(`sqlite3 database.db "SELECT replace(Value, '\n', '') FROM Values"`)
for record in "${records[#]}"; do
echo $record
done
Alternatively, you could change the IFS in Bash - but you are relying on linefeed as a seperator between records.
So I have a variable TRAILER which contains about 50 character. This variable is defined earlier in my shell session. As you can probably tell, it's a trailer to a file we'll be sending. I need to insert the record count of that file into the trailer. This record count is going to be 9 digits long (left padded with zeros if need be) and will start at index 2 of that string TRAILER. I want to retain all other characters in the TRAILER string just insert the RECORD_COUNT variable value into the TRAILER variable starting at index 2 (3rd character)
So the trailer variable is defined like this:
#Trailer details
TRAILER_RECORD_IDENTIFER="T"
LIFE_CYCLE="${LIFE_CYCLE_ENV}"
RECORD_COUNT="" #This will be calculated in the wrapper during the creation step
FILE_NUMBER="1111"
FILE_COUNT="1111"
CONTROL_TOTAL_1=" "
CONTROL_TOTAL_2=" "
CONTROL_TOTAL_3=" "
CONTROL_TOTAL_4=" "
CONTROL_TOTAL_5=" "
TRAILER="${TRAILER_RECORD_IDENTIFER}"\
"${LIFE_CYCLE}"\
"${RECORD_COUNT}"\
"${FILE_NUMBER}"\
"${FILE_COUNT}"\
"${CONTROL_TOTAL_1}"\
"${CONTROL_TOTAL_2}"\
"${CONTROL_TOTAL_3}"\
"${CONTROL_TOTAL_4}"\
"${CONTROL_TOTAL_5}"
Which then prints TRAILER as
TRAILER="TD11111111......" that would be 75 blank spaces for all of the white characters defined by the CONTROL_TOTAL variables.
These variables ALL get defined in the beginning of the shell. REcord count is defined but left blank ebcause we won't know the specific file until later int he shell.
Later in the shell i know the file that i want to use, i get the record coun:
cat ${ADE_DATA_FL_PATH_TMP} | wc -l | read ADE_DATA_FL_PATH_TMP_REC_COUNT >> ${LOG_FILE} 2>&1
Now I want to take ADE_DATA_FL_PATH_TMP_REC_COUNT and write that value into the TRAILER variable starting at the 2nd index, padded with zero's to be 9 characters long. So if my record count is 2700 records the new trailer would look like...
TRAILER="TD00000270011111111......"
You can use printf for padding.
I use TD as fixed first two characters, you can change this the way you want.
printf -v TRAILER "TD%.9d%s" "${ADE_DATA_FL_PATH_TMP_REC_COUNT}" "$(cut -c 12- <<< "${TRAILER}")"
Perhaps this is a good time switching to writing variable names in lowercase.
I'm trying to get back an array from perl to bash.
My perl scrip has an array and then I use return(#arr)
from my bash script I use
VAR = `perl....
when I echo VAR
I get the aray as 1 long string with all the array vars connected with no spaces.
Thanks
In the shell (and in Perl), backticks (``) capture the output of a command. However, Perl's return is normally for returning variables from subroutines - it does not produce output, so you probably want print instead. Also, in bash, array variables are declared with parentheses. So this works for me:
$ ARRAY=(`perl -wMstrict -le 'my #array = qw/foo bar baz/; print "#array"'`); \
echo "<${ARRAY[*]}> 0=${ARRAY[0]} 1=${ARRAY[1]} 2=${ARRAY[2]}"
<foo bar baz> 0=foo 1=bar 2=baz
In Perl, interpolating an array into a string (like "#array") will join the array with the special variable $" in between elements; that variable defaults to a single space. If you simply print #array, then the array elements will be joined by the variable $,, which is undef by default, meaning no space between the elements. This probably explains the behavior you mentioned ("the array vars connected with no spaces").
Note that the above will not work the way you expect if the elements of the array contain whitespace, because bash will split them into separate array elements. If your array does contain whitespace, then please provide an MCVE with sample data so we can perhaps make an alternative suggestion of how to return that back to bash. For example:
( # subshell so IFS is only affected locally
IFS=$'\n'
ARRAY=(`perl -wMstrict -e 'my #array = ("foo","bar","quz baz"); print join "\n", #array'`)
echo "0=<${ARRAY[0]}> 1=<${ARRAY[1]}> 2=<${ARRAY[2]}>"
)
Outputs: 0=<foo> 1=<bar> 2=<quz baz>
Here is one way using Bash word splitting, it will split the string on white space into the new array array:
array_str=$(perl -E '#a = 1..5; say "#a"')
array=( $array_str )
for item in ${array[#]} ; do
echo ": $item"
done
Output:
: 1
: 2
: 3
: 4
: 5
I have a file - abc, which has the below content -
Bob 23
Jack 44
Rahul 36
I also have a shell script that do the addition of all the numbers here.
The specific line that picks up these numbers is -
while read line
do
num=echo ${line#* }
sum=`expr $sum + $num`
count=`expr $count + 1`
done< "$readfile"
I assumed that the code is just picking up the last field from file, but it's not. If i modify the file like
Bob 23 12
Jack 44 23
Rahul 36 34
The same script fails with syntax error.
NOTE: I know there are other ways to pick up the field value, but i would like to know how this works.
The syntax ${line#* } will skip the shortest string from the beginning till it finds a space and returns the rest. It worked fine when you had just 2 columns. But the same will not work when 3 columns are present as it will return you the last 2 column values which when you use it in the sum operator will throw you an error. To explain that, just imagine
str='foo bar'
printf '%s\n' "${str#* }"
bar
but imagine the same for 3 fields
str='foo bar foobar'
printf '%s\n' "${str#* }"
bar foobar
To fix that use the parameter expansion syntax of "${str##* }" to skip the longest sub-string from beginning. To fix your script for the example with 3 columns, I would use a script as below.
This does a simple input redirection on the file and uses the read command with the default IFS value which is a single white space. So I'm getting only the 3rd field on each line (even if it has multiple fields), the _ mark the fields I'm skipping. You could also have some variables as place-holders and use their value in the scripts also.
declare -i sum
while read -r _ _ value _ ; do
((sum+=value)
done < file
printf '%d\n' "$sum"
See Bash - Parameter Expansion (Substring removal) to understand more.
You could also use the PE syntax ${line##* } as below,
while read -r line ; do
((sum+=${line##* }))
done < file
[Not relevant to the current question]
If you just want the sum to be computed and not specifically worried about using bash script for this. You can use a simple Awk command to sum up values in 3rd column as
awk '{sum+=$3}END{print sum}' inputfile
I have a select statement
sqlplus [credentials] select variable from table;
It returns 6 rows and I need to store them as an array in bash array variable.
array=(`sqlplus [credentials] select variable from table;`)
echo ${array[*]}
If your variables contain spaces and you want the array to have an element for each line of output (as opposed to each word of output), you also need to set your IFS. And you may want to use quotes when using the array:
SaveIFS="$IFS"
IFS=$'\n'
array=( $(sqlplus [credentials] select variable from table;) )
echo "${array[*]}"
IFS="$SaveIFS"