I was wondering how I could pass * in my variable in a bash for loop. I want to run multiple select queries on a db and I thought I'd use a bash for loop like so:
for query in "select * from filtertype limit 10;" "select * from filter limit 10;"; do echo $query; echo $query | mysql -uroot -ppassword database; done
The trouble is the * keeps on getting expanded like so:
[Bash]$ for query in "select * from filtertype limit 10;" "select * from filter limit 10;"; do echo $query; echo $query | mysql -uroot -ppassword database; done
select acceptance_test bin build-dist.xml build-path-definitions.xml build.properties build.xml classes config core-def database dependencies dev.properties dist docs dummyFile_pdffile.pdf eclipse_classes findbugsExcludeFilter.xml generated getRevision.sh local.build.properties log nohup.out plugins-defs plugins-framework pmc-api-client pmd_rules.xml port_helper.sh raw_data-pigunit-input-overridden.txt raw_event_data-pigunit-input-overridden.txt raw-pigunit-input-overridden.txt rpmbuild src testArea unit_test web from filtertype limit 10;
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'build-dist.xml build-path-definitions.xml build.properties build.xml classes con' at line 1
select acceptance_test bin build-dist.xml build-path-definitions.xml build.properties build.xml classes config core-def database dependencies dev.properties dist docs dummyFile_pdffile.pdf eclipse_classes findbugsExcludeFilter.xml generated getRevision.sh local.build.properties log nohup.out plugins-defs plugins-framework pmc-api-client pmd_rules.xml port_helper.sh raw_data-pigunit-input-overridden.txt raw_event_data-pigunit-input-overridden.txt raw-pigunit-input-overridden.txt rpmbuild src testArea unit_test web from filter limit 10;
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'build-dist.xml build-path-definitions.xml build.properties build.xml classes con' at line 1
[Bash]$
When I tried single quotes around the loop variables I got the same error and when I single and double quoted the stars I got:
select \* from filtertype limit 10;
ERROR at line 1: Unknown command '\*'.
select \* from filter limit 10;
ERROR at line 1: Unknown command '\*'.
So the question is - is it possible to pass a string variable with a '*' in it into a bash for loop?
A
The problem is that your variable references, $query, are unquoted, and thus subject to shell expansions (in this case, * expands to the names of files and directories in the current directory, a process called pathname expansion).
Thus, simply double-quote your variable references to protect them from expansions:
echo "$query"
I found the answer. Hope it's at least useful to somebody else. The solution is to quote the variable when it getting called:
for query in "select * from filtertype limit 10;" "select * from filter limit 10;"; do echo "$query"; echo "$query" |mysql -uroot -ppassword database; done
That works
A
Related
I use an HPC cluster. The compute nodes can't have access to internet, only the frontal.
So I want to wrap all the commands that need to access internet in order to execute them on the frontal.
ex: for wget
#!/bin/bash
ssh frontal /bin/wget "$#"
-> works fine
I have to wrap this bq (google BigQuery) command:
bq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '2016%' AND mgrs_tile == '32ULU' ORDER BY sensing_time ASC LIMIT 1000;"
I managed to requote the command and to launch it successfully on CLI:
ssh frontal '~/downloads_and_builds/builds/google-cloud-sdk/bin/bq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '"'"'2016%'"'"' AND mgrs_tile == '"'"'32ULU'"'"' ORDER BY sensing_time ASC LIMIT 1000;"'
Now I want to write a wrapper named bq able to get the parameters and launch this command through ssh ... here is what i have tried :
#!/bin/bash
set -eu
# all parameters in an array
args=("$#")
# unset globing (there's a * in the SELECT clause)
set -f
# managing inner quotes
arg2=`echo "${args[2]}" | perl -pe 's/'\''/'\''"'\''"'\''/g'`
# put back double quotes (") suppressed by bash
args="${args[0]} ${args[1]} \"${arg2}\""
# build command with parameters
cmd="~/downloads_and_builds/builds/google-cloud-sdk/bin/bq $args"
echo ""
echo "command without external quotes"
echo "$cmd"
echo ""
echo "testing it ..."
ssh hpc-login1 "$cmd"
echo ""
# wrapping command between simple quotes (like on the CLI)
cmd="'"'~/downloads_and_builds/builds/google-cloud-sdk/bin/bq '"$args""'"
echo "commande with external quotes"
echo "$cmd"
echo ""
echo "testing it ..."
ssh hpc-login1 $cmd
echo "done"
Here is the output of this script:
$ bq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '2016%' AND mgrs_tile == '32ULU' ORDER BY sensing_time ASC LIMIT 1000;"
command without external quotes
~/downloads_and_builds/builds/google-cloud-sdk/bin/bq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '"'"'2016%'"'"' AND mgrs_tile == '"'"'32ULU'"'"' ORDER BY sensing_time ASC LIMIT 1000;"
testing it ...
Waiting on bqjob_r102b0c22cdd77c2d_000001629b8391a3_1 ... (0s) Current status: DONE
commande with external quotes
'~/downloads_and_builds/builds/google-cloud-sdk/bin/bq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '"'"'2016%'"'"' AND mgrs_tile == '"'"'32ULU'"'"' ORDER BY sensing_time ASC LIMIT 1000;"'
testing it ...
bash: ~/downloads_and_builds/builds/google-cloud-sdk/bin/bq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '2016%' AND mgrs_tile == '32ULU' ORDER BY sensing_time ASC LIMIT 1000;": Aucun fichier ou dossier de ce type (in english: no file or directory of this kind)
As you can see, I managed to get a correct command string, just like the one which works on CLI, but it doesn't work in my script:
The first attempt succeeded but gives no output (I have tried to redirect it in a file: the file were created but is empty)
In the second attempt (with external simple quotes, just like the CLI command that worked), bash take the quoted arg as a block and don't find the command ...
Has somebody an idea on how to launch a complex command (with quotes, wildcards ...) like this one through ssh using a wrapper script ?
(ie. one wrapper named foo able to replace a foo command and execute it correctly through ssh with the arguments provided)
ssh has the same semantics as eval: all arguments are concatenated with spaces and then evaluated as a shell command.
You can have it work with execve semantics (like sudo) by having a wrapper escape the arguments:
remotebq() {
ssh yourhost "~/downloads_and_builds/builds/google-cloud-sdk/bin/bq $(printf '%q ' "$#")"
}
This quotes thoroughly and consistently, so you no longer have to worry about adding additional escaping. It'll run exactly what you tell it (as long as your remote shell is bash):
remotebq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '2016%' AND mgrs_tile == '32ULU' ORDER BY sensing_time ASC LIMIT 1000;"
However, the downside to running exactly what you tell it is that now you need to know exactly what you want to run.
For example, you can no longer pass '~/foo' as an argument because this is not a valid file: ~ is a shell feature and not a directory name, and when it's correctly escaped it will not be replaced by your home directory.
The basic way to do this, using shell here doc :
#!/bin/bash
ssh -t server<<'EOF'
bq --format=json query "SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '2016%' AND mgrs_tile == '32ULU' ORDER BY sensing_time ASC LIMIT 1000;"
command2
command3
...
EOF
I see you are already using Perl so...
use Net::OpenSSH;
my $query = q(SELECT * FROM [bigquery-public-data:cloud_storage_geo_index.sentinel_2_index] WHERE sensing_time LIKE '2016%' AND mgrs_tile == '32ULU' ORDER BY sensing_time ASC LIMIT 1000;);
my $ssh = Net::OpenSSH->new($host);
$ssh->system('bq', '--format=json', 'query', $query)
or die $ssh->error;
Net::OpenSSH would take care of quoting everything.
I want to pass argument to a BigQuery script in shell, here is the example of script I wrote
#!/bin/bash
bq query --use_legacy_sql=false --destination_table=abc --append 'select * from `xyz.INFORMATION_SCHEMA.VIEWS` union all Select * from `def.VIEWS`) where table_name = "$1"'
when I run this script and pass the argument, I do not get any errors but no row is appended to the table. whereas when i specify the table_name as rty that row is appended to the table. What am I missing here?
When you run the script you'll get a prompt like:
Waiting on <BIGQUERY_JOB_ID> ... (0s) Current status: DONE
You can inspect the job in many ways, including the bqtool:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID>
If you have jq installed (sudo apt install jq) you can get just the translated query with:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID> | jq '.configuration.query.query'
which will get you something similar to:
select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name = \"$1\"
As you can see the variable is not correctly escaped so no table matches the WHERE filter. To avoid this you can enclose the query in double quotes and the variable in single ones like this:
#!/bin/bash
bq query \
--use_legacy_sql=false \
--destination_table=xyz.abc \
--append \
"select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name='$1'"
You can get the INFORMATION_SCHEMA.VIEWS: command not found error if using back-ticks. You can omit or escape them with a backslash:
"select * from \`xyz\`.INFORMATION_SCHEMA.VIEWS where table_name='$1'"
As I'm new to Unix, can someone help why I get this error?
Error: 0403-057 Syntax error at line 1 : `(' is not expected
Unix server used: AIX servname 1 6 00F635064C00
Script used (to send email alert if day before yesterday source files didn't arrive):
#!/usr/bin/ksh
count=$(sqlplus $PROD_DB #select count(*) from file_audit where (file_name like '%abc%' or file_name like '%dce%') and substr(file_name,17,8)=to_char(to_date(sysdate-2,'DD/MM/YY'), 'yyyymmdd') > asa_file_count.log)
daybefore=`TZ=aaa48 date +%d-%m-%Y`
if [[ $count -lt 20 ]]
then
echo "Alert - Source files are yet to be received for date: $daybefore" | mail -s "Alert : Source data files missing" s#g.com
fi
Parentheses are special to the shell. Your SQL script contains parentheses you don't want the shell to process. However, the shell processes all non-quoted parentheses. Therefore, you can use quotes to prevent the parentheses in your SQL from being interpreted by the shell:
count=$(sqlplus $PROD_DB "#select count(*) from file_audit where (file_name like '%abc%' or file_name like '%dce%') and substr(file_name,17,8)=to_char(to_date(sysdate-2,'DD/MM/YY'), 'yyyymmdd')" > asa_file_count.log)
# ^ and similarly, a closing quote at the end, just before ">asa_file..." .
Now, there is a second issue: you have
count=$(sqlplus ... > asa_file_count.log)
However, I think this means count will always be empty, since the count will go into asa_file_count.log and will not be available to be captured with $(). I believe removing the >asa_file_count.log will probably do what you want:
count=$(sqlplus "$PROD_DB" "<your query>")
(I also put double-quotes around $PROD_DB just in case PROD_DB's value contains any spaces.)
i have a delete statement delete * from table_name;. Currently shell expands it to list all existing files in current directory. How can i escape it, so that the string that is passed to sqlplus is indeed "delete * from table_name". I tried \* '*' and \\* and none of them work.
The exact script is
#!/bin/bash
table_name = $1
delete_sql= "delete * from $table_name;"
echo $delete_sql > abc.sql
How about
echo "delete * from table_name" | sqlplus
or
echo "delete * from table_name" > mystatements.txt
sqlplus #mystatements.txt
On a side note, you don't need to specify * in a delete statement - all fields in the matching rows are deleted.
You just need to quote the variable (and fix your spacing around the = sign):
#!/bin/bash
table_name=$1
delete_sql="delete * from $table_name;"
echo "$delete_sql" > abc.sql
I have a DB2 query in a shell script which return an integer value, but I am unable to store it in a variable.
temp1= echo db2 -x "select max(id) from work.work_tb"
I am getting this output when I run it, sh -x test.sh
db2 -x select max(id) from work.work_tb
echo 50
temp1=
50
So for some reason $temp1 is unable to get the value, I think its because the db2 query is returning value prefixed with \n. How do I get rid of the newline char and get the value to temp1?
No, that's not why.
temp1=`db2 -x "select max(id) from work.work_tb"`
emp1=$(echo db2 -x "select max(id) from work.work_tb")
or using backticks
emp1=`echo db2 -x "select max(id) from work.work_tb"`
In general, to remove newlines, you can pass it to tools like tr/sed etc
... | tr -d "\n"