Linux, Bash Script.
I have created a query as below, which works
sqlcmd -S [dbname].database.windows.net -d [database name]-U [username ]-P [password] -Q "SELECT * FROM dbo.microiopenvpn WHERE mode='create'"
How do I then loop over results.
The table has name, email_address & mode fields.
I just want the email_address field (for this query)
Figured it out - one way I guess
while IFS="," read -r d1 d2 d3 d4 d5 d6; do
echo "$d1" "$d2" "$d4"
done < sqloutputNew.csv
Get the column from each row, then iterates, seems to be the best solution I could find.
With the SQL having an output as below
-o "sqloutputNew.csv" -W -w 1024 -s"," -h-1
Related
Below are my example files.
Using AccountInfo(Account) to search the sub-folder existing on FolderStatus(Sub-folder).
AccountInfo
#server,Account,status,homefolder
vmftp01,admin01,enable,/home/admin01
vmftp01,admin02,enable,/home/admin02
vmftp01,admin03,enable,/home/admin03
FolderStatus
#account,sub-folder
admin01,/sftp/inbox
admin02,/ftp/inbox
admin02,/sftp/inbox
admin02,/as2/inbox
admin03,/ftp/inbox
admin03,/sftp/inbox
Desired Output:
/home/admin01/sftp/inbox
/home/admin02/ftp/inbox
/home/admin02/sftp/inbox
/home/admin02/as2/inbox
/home/admin03/ftp/inbox
/home/admin03/sftp/inbox
I tried:
join -t 1.2,2.2 file1 file2
with error:
join: file2:4: is not sorted: admin02,/as2/inbox
First of all, your FolderStatus file is indeed not sorted, but that's only a problem because your join command line doesn't seem to be correct.
try
join -t , -1 2 -o 1.4,2.2 AccountInfo FolderStatus
With AccountInfo = f2 and FolderStatus = f1, it worked for me this way (up to a comma that can be easily eliminated):
ronald#oncilla:~/tmp$ join -t , -1 2 -o 1.4,2.2 f2 f1
/home/admin01,/sftp/inbox
/home/admin02,/ftp/inbox
/home/admin02,/sftp/inbox
/home/admin02,/as2/inbox
/home/admin03,/ftp/inbox
/home/admin03,/sftp/inbox
The "-t ," option specifies the field separator.
The "-1 2" option specifies that the key in file 1 is the second column. And the "-o" flag specifies the output format
I´m trying to acces values in a variable I receive from an sqlcmd in a loop:
while read line ;do AllNumbers=`sqlcmd -h -1 -S mysqlserver.localdomain -U Me -P MyPasswd -d master -Q "SET nocount on; SELECT PHONENUMBER FROM MYTABLE WHERE USER LIKE '"$line"' " < /dev/null`; for i in "$AllNumbers"; do echo "$line $i"; done; done< mytext.txt
The Number of results in the variable "$AllNumbers" can be 1 to 5, and the lenght of the number can also vary.
What I get in Moment is:
John 01234567
9876543
Jack 13579024
Jim 08642135
10293847
56473829
In the end I want to have an output like:
John 01234567
John 98765432
Jack 13579024
Jim 08642135
Jim 10293847
Jim 56473829
I tried with IFS, but I´m unsure what the delimiter in the variable $AllNumber is.
If I do:
OneUser=`sqlcmd -h -1 -S mysqlserver.localdomain -U Me -P MyPasswd -d master -Q "SET nocount on; SELECT PHONENUMBER FROM MYTABLE WHERE USER LIKE '"Jim"' "`; echo "$OneUSer"
I get:
08642135
10293847
56473829
I´m glad for any hint...
Thanks in advance
Lotte
Why not ask the database to do the hard work?
SELECT USER, PHONENUMBER FROM MYTABLE WHERE USER LIKE '"$line"'
Also, make sure you validate $line to prevent SQL injection attacks (see Bobby Tables).
I would like to make an assertion the output of a postgres query using bash. Concretely, I am writing a bash job that counts the number of rows and if the count is not equal to zero, does something to raise alert.
$ psql MY_DATABASE -c "SELECT COUNT(*) WHERE foo=bar"
count
-------
0
(1 row)
In my script, I would like to assert that the output of above query is zero. However I am not sure where to begin because the output is not a number, but a formatted multi line string.
Is there an option in psql that makes it output a single number when counting, or could you think of any other approaches?
I would suggest to use temporary file to redirect the output and use it. Once your work is done, delete the temp file.
psql your_database -c "SELECT COUNT(*) as Count from table_a where c1=something" -t >assert.tmp
line=$(head -n 1 assert.tmp)
if [ $line -ge 0 ]; then
echo "greater then 0 and values is--"$line
fi
rm assert.tmp
Hope it works for you.
I have a bash script as follows
#!/usr/bin/env bash
cd /Users/amar/Documents/ThesisCode/CEP_codes/mqtt-receiver
sqlite3 database <<EOF
SELECT sum(detection_time - generation_time)/count(*) from mobile_cep_data;
SELECT ((max(detection_time)- min(detection_time))*1000)/count(*) from mobile_cep_data;
EOF
it gives me result as
15
12
How can I get the result as
latency = 15
thoughput = 12
#Cyrus's answer is fine and demonstrates a useful technique, and I've therefore upvoted it, but I'd like to point out that you can also do:
sqlite3 database <<EOF
SELECT 'latency = ',
sum(detection_time - generation_time)/count(*) from mobile_cep_data;
SELECT 'thoughput = ',
((max(detection_time)- min(detection_time))*1000)/count(*) from mobile_cep_data;
EOF
Replace
<<EOF
with
<<EOF | sed '1s/^/latency = /;2s/^/thoughput = /'
How can I get row count from all tables using hive? I am interested in the database name, table name and row count
You will need to do a
select count(*) from table
for all tables.
To automate this, you can make a small bash script and some bash commands.
First run
$hive -e 'show tables' | tee tables.txt
This stores all tables in the database in a text file tables.txt
Create a bash file (count_tables.sh) with the following contents.
while read line
do
echo "$line "
eval "hive -e 'select count(*) from $line'"
done
Now run the following commands.
$chmod +x count_tables.sh
$./count_tables.sh < tables.txt > counts.txt
This creates a text file(counts.txt) with the counts of all the tables in the database
A much faster way to get approximate count of all rows in a table is to run explain on the table. In one of the explain clauses, it shows row counts like below:
TableScan [TS_0] (rows=224910 width=78)
The benefit is that you are not actually spending cluster resources to get that information.
The HQL command is explain select * from table_name; but when not optimized not shows rows in the TableScan.
select count(*) from table
I think there is no more efficient way.
You can collect the statistics on the table by using Hive ANALAYZE command. Hive cost based optimizer makes use of these statistics to create optimal execution plan.
Below is the example of computing statistics on Hive tables:
hive> ANALYZE TABLE stud COMPUTE STATISTICS;
Query ID = impadmin_20171115185549_a73662c3-5332-42c9-bb42-d8ccf21b7221
Total jobs = 1
Launching Job 1 out of 1
…
Table training_db.stud stats: [numFiles=5, numRows=5, totalSize=50, rawDataSize=45]
OK
Time taken: 8.202 seconds
Links:
http://dwgeek.com/apache-hive-explain-command-example.html/
You can also set the database in the same command and separate with ;.
hive -e 'use myDatabase;show tables'
try this guys to automate-- put in shell after that run bash filename.sh
hive -e 'select count(distinct fieldid) from table1 where extracttimestamp<'2018-04-26'' > sample.out
hive -e 'select count(distinct fieldid) from table2 where day='26'' > sample.out
lc=cat sample.out | uniq | wc -l
if [ $lc -eq 1 ]; then
echo "PASS"
else
echo "FAIL"
fi
How do I mention the specific database that it needs to refer in below snippet:
while read line
do
echo "$line "
eval "hive -e 'select count(*) from $line'"
done
Here's a solution I wrote that uses python:
import os
dictTabCnt={}
print("=====Finding Tables=====")
tableList = os.popen("hive --outputformat=dsv --showHeader=false -e \"use [YOUR DB HERE]; show tables;\"").read().split('\n')
print("=====Finding Table Counts=====")
for i in tableList:
if i <> '':
strTemp = os.popen("hive --outputformat=dsv --showHeader=false -e \"use [YOUR DB HERE]; SELECT COUNT(*) FROM {}\"".format(i)).read()
dictTabCnt[i] = strTemp
print("=====Table Counts=====")
for table,cnt in dictTabCnt.items():
print("{}: {}".format(table,cnt))
Thanks to #mukul_gupta for providing shell script.
how ever we are encounting below error for the same
"bash syntax error near unexpected token done"
Solution for this at below link
BASH Syntax error near unexpected token 'done'
Also if any one need how to select DB Name
$hive -e 'use databasename;show tables' | tee tables.txt
for passing db name in select statement, give DB name in tableslist file itself.