I am running a impala query in while loop and for that I have created one separate query file and I am calling it from my shell script.
My question is: Can we pass shell variable matching with impala query in query file?
A="INSERT_SBP_ME_VS_ME_INCOME_LAST_THIRTY_DAYS_Q"${Count}
echo "value of A is $A"
source ${SBP2_MNY_IN_LAST_THIRTY_DAYS_QF}
${IMPALA_CON} -q "${${A}}"
'A' value is like INSERT_SBP_ME_VS_ME_INCOME_LAST_THIRTY_DAYS_Q1 (as count is 1)
I am doing this in this way but getting bad substitution error and I also tried
${IMPALA_CON} -q "${A}"
but not getting a successful result.
You seem to be looking for --var (IMPALA-2179). To substitute from the command line, you can do:
impala-shell -f test.q --var=L=2;
where test.q is:
select * from p_test limit ${VAR:L};
Your query should be :
impala-shell -q "$A"
Refer:
impala-shell Configuration Options
similar post
impala-shell -i node.domain:port -B --var"table=metadata" --var="db=retail" -f "file.sql"
file.sql:
SELECT * FROM ${var:db}.${var:table}"
Related
I'm having hard time executing using -e option with column name is in quotes.
I want to execute some thing like below using unix level. Trying to run from shell script. When i try to put my values in quotes, its taking away the quotes for my column.
select * from keyspace.cf where "columnname"=''
Tried this:
cqlsh hostname -e "select * from keyspace.cf where "columnname"=''"
It is executing as cqlsh hostname -e 'select * from keyspace.cf where columnname='
stdin>:1:InvalidRequest: Error from server: code=2200 [Invalid query] message="Undefined name columnaname in where clause ('columnname= 'value'')"
You don't need to put quotes around columnname, you just need to set it and prefix it with a $.
cqlsh -u cassandra -p cassandra -e "SELECT $COLUMN FROm $KEYSPACE.$TABLE;"
That's an excerpt from a script I wrote called getVersion.sh.
#!/bin/bash
KEYSPACE="system"
TABLE="local"
COLUMN="release_version"
~/local/apache-cassandra-3.10/bin/cqlsh -u cassandra -p cassandra -e "SELECT $COLUMN FROm $KEYSPACE.$TABLE;"
aploetz#dockingBay94:~/scripts$ ./getVersion.sh
release_version
-----------------
3.10
(1 rows)
The same will work if your column names contain quotes. Just be sure to escape them in your variable definition. This is a similar script, but it queries the "columnName" TEXT column:
#!/bin/bash
KEYSPACE="stackoverflow"
TABLE="stuff_with_quotes"
COLUMN="\"columnName\""
~/local/apache-cassandra-3.10/bin/cqlsh -u cassandra -p cassandra -e "SELECT $COLUMN FROm $KEYSPACE.$TABLE;"
I'm trying to script a query from a bash file but the select query is in a file. For compatibility and test, I'd like to keep it this way.
In short I'd like to write something like:
psql -c data_base "\copy (<file.sql>) To './test.csv' With CSV"
Would you know how to do that ?
EDIT
The "file.sql" contains a query :
$cat file.sql
$SELECT country FROM world;
You can use bash substitution as,
psql -c data_base "\copy ($(<file.sql)) To './test.csv' With CSV"
$(<file) expands the contents of the file
There is some similar topics, but this is slightly different.
I have database with names of scripts and parameters a. When I execute:
sqlite3 log/log.db "select name, a from result" | awk -F '|' '{printf("a[%s]=%s;\n",$1,$2);}'
I see:
a[inc.bash]=4.23198234894777e-06;
a[inc.c]=3.53343440279423e-10;
In my bash script I would like to use an associative array.
When I execute this code (coding by hand value of a[inc.bash]):
declare -A a
a[inc.bash]=4.23198234894777e-06;
echo ${a[inc.bash]}
It works correctly and print
4.23198234894777e-06
But I do not know, how to use output of first presented command with awk to assign values of key of associative array a declared in my script.
I want to execute code that is printed by awk inside of my script, but when I use something like $() or ``, it prints a error like this:
code:
declare -A a
$(sqlite3 log/log.db "select name, a from result" | awk -F '|' '{printf("a[%s]=%s;\n",$1,$2);}')
echo ${a[inc.bash]}
output:
a[inc.bash]=4.23198234894777e-06; not found command
To tell Bash to interpret your output as commands, you can use process substitution and the source command:
declare -A a
source <(sqlite3 log/log.db "select name, a from result" |
awk -F '|' '{printf("a[%s]=%s;\n",$1,$2);}')
echo ${a[inc.bash]}
The <() construct (process substitution) can be treated like a file, and source (or the equivalent .) runs the commands in its argument without creating a subshell, making the resulting a array accessible in the current shell.
A simplified example to demonstrate, as I don't have your database:
$ declare -A a
$ source <(echo 'a[inc.bash]=value')
$ echo "${a[inc.bash]}"
value
This all being said, this is about as dangerous as using eval: whatever the output of your sqlite/awk script, it will be executed!
I'm writing a script truncating tables. The problem is that the script goes in infinite loop.
./n_input contains environment variables that are used in script.
#!/bin/ksh
INPUT_FILE=./n_input
if [ ! -e $INPUT_FILE ];
then
echo "Error: Input file require"
exit 1
fi
source $INPUT_FILE
while read Table
do
TABLE="$DSTN_DATABASE.dbo.$Table"
echo "Table : $TABLE"
QUERY="truncate table $TABLE"
echo $QUERY > ./tmp_file
sqlcmd -m 1 -U $DSTN_USER -P $DSTN_PASSWORD -D -S $DSTN_SERVER -m1 -i ./tmp_file
RET_VALUE=$?
if [ $RET_VALUE -ne 0 ]
then
echo "Error $TABLE"
fi
done < $TABLE_LIST
exit 0
How do I break the loop? I have tried to remove sqlcmd from script and verified that it was working. It's working as expected. Observed the same behavior with sqlcmd -Q option.
$TABLE_LIST file contains only one table name.
You don't need to write the query into temporary file. Use -q, or -Q options instead:
q="truncate table ${TABLE};"
sqlcmd -m 1 -U "$DSTN_USER" -P "$DSTN_PASSWORD" -S "$DSTN_SERVER" -q "$q"
Note the ; at the end of the query. Probably, that's the reason why the script "stalls". That may look like an infinitely running loop.
Also note the use of double quotes. You should wrap variables in double quotes to prevent reinterpretation of the special characters.
By the way, you can locate the exact command that is causing the issue by adding set -x at the beginning of the script. set -x turns on debugging mode. With debugging mode on, you see the commands being executed.
It's very unlikely that the content of $TABLE_LIST file is causing such behavior, unless the file is enormously big. The loop construct is correct, and the number of iterations should match the number of lines in the file.
I am querying a database from a bash script using following query:-
Output = echo "$QUERY_STR" | mysql -h $DB_HOST -u $DB_USER -p$DB_PASS $DB_NAME
it gives me the required output which I save in a Variable
However when I echo $output I do not get proper formatted output like in command line of mysql query.
Read one of the post to use -t in the query however for large data set it does not give proper output.
To work around it, I am saving the output in a .csv file.
To maintain all the whitespace that is indeed kept in the variable's value, it is crucial to double-quote the variable:
echo "$output"
Also, you cannot have whitespace around the equal sign in a variable assignment:
output=$(mysql ... <<< "$QUERY_STR")