I use the following script to import data from a csv file:
#!/bin/bash
# show commands being executed, per debug
set -x
# define database connectivity
_db="xxx"
_db_user="xxx"
_db_password="xxx"
_table="movie"
# define directory containing CSV files
_csv_directory="/tmp"
_csv_file='xxxxxx.csv'
_header_columns_string='link,description,duration,thumbnaillink,iframe,tags,category'
# import csv into mysql
mysqlimport --fields-terminated-by=';' --lines-terminated-by="\n" --columns=$_header_columns_string -u $_db_user -p$_db_password $_db $_table $_csv_directory/$_csv_file
exit
When I execute the script as root via bash import.sh I get the following error message:
+ _db=mydatabase
+ _db_user=xxx
+ _db_password=xxx
+ _table=movie
+ _csv_directory=/tmp
+ _csv_file=xxxxxx.csv
+ _header_columns_string=link,description,duration,thumbnaillink,iframe,tags,category
+ mysqlimport --local '--fields-terminated-by=;' '--lines-terminated-by=\n' --columns=link,description,duration,thumbnaillink,iframe,tags,category -u xxx -pxxx mydatabase movie /tmp/xxxxxx.csv
mysqlimport: Error: 13, Can't get stat of '/var/lib/mysql/mydatabase/movie' (Errcode: 2), when using table: movie
+ exit
but the database and the table exist.
The csv file exists and can be read, the table can be selected and I can manually insert data-rows into the db.
What am I doing wrong?
Try doing it from inside SQL:
load data local infile 'FILENAME.CSV' into table TABLENAME
fields terminated by ',' optionally enclosed by '"' ignore 1 lines;
You run this from the command line or from a shell script like this
db='mysql -hX -uX -pX --database=X'
cat MYSCRIPT.SQL | $db
Related
I have following yaml file and I need to take inputs from this yaml file in my bash script
Database: backup
Table: mytable
Partitions: P10,P11,P12
I tried this like below but getting error
#!/bin/bash
Database=yq e '.Database' t_partitions.yaml
Table=yq e '.Table' t_partitions.yaml
Partitions=yq e '.Partitions' t_partitions.yaml
mysql -u root -p -e "
use $Database;
alter table $Table truncate partition $Partitions;
"
The error is
bash m.sh run
m.sh: line 2: e: command not found
m.sh: line 3: e: command not found
m.sh: line 4: e: command not found
Your assignment statement is wrong with Bash's grammar.
You need command substitution, like:
#!/bin/bash
Database="$(yq e '.Database' t_partitions.yaml)"
Table="$(yq e '.Table' t_partitions.yaml)"
Partitions="$(yq e '.Partitions' t_partitions.yaml)"
mysql -u root -p -e "
use $Database;
alter table $Table truncate partition $Partitions;
"
Using $() to get output of a command. Use "" to prevent eventually sentence break inside the output by some special character.
I am new to the shell scripting hence need help. I am trying to execute sql query against Oracle DB. Once sql query result is received, I need to iterate through the result (as it will return multiple rows and columns.).
Goal here is to invoke a REST api using curl for each record retrieved in db result. Input to REST api will be ROOT_PROC_ID column value.
Below is the shell script I have developed so far and sample output of the sql query.
#!/bin/bash
#Update below properties as per enviornment
export ENV=DEV;
export SERVERHOST=localhost
export SERVERPORT=9000
export SERVERUSER=admin
export SERVERPASSWORD=admin123
export DBHOST=localhost
export DBPORT=1537
export DBSID=ORCL
export DBUSER=SCOTT
export DBPASS=TIGER
export LOGDIR=/usr/app/$USER/data/logs/
#-------------------------------------------------------------------------
#----------- DO NOT EDIT AFTER THIS LINE ---------------------------------
#-------------------------------------------------------------------------
#create directory structure if not exists for storing log files
mkdir -p $LOGDIR/process_cancellation
mkdir -p $LOGDIR/process_cancellation/old
mkdir -p $LOGDIR/process_cancellation/halted
export old_proc_cancellation_logfile=$LOGDIR/process_cancellation/old/log_$(date "+%Y%m%d%H%M%S").log;
export halted_proc_cancellation_logfile=$LOGDIR/process_cancellation/halted/log_$(date "+%Y%m%d%H%M%S").log;
#execute sql query to fetch halted process data from database
echo
echo "Enviornment : $ENV"
echo
echo "Connecting to - $DBUSER/$DBPASS#$DBHOST:$DBPORT/$DBSID"
echo
echo "Retrieving halted process data logged before : $(date -d "15 days ago" +%d/%m/%Y) 20:00:00"
echo
sqlplus -s $DBUSER/$DBPASS#$DBHOST:$DBPORT/$DBSID << FIN_SQL > $halted_proc_cancellation_logfile
set head off
set line 1024
set pages 9999
SET SERVEROUTPUT ON;
SELECT ROOT_PROC_ID, PROC_ID, PROC_NAME, START_DATE, STATUS, ORDER_REF
FROM USER.PROC_STATUS
WHERE START_DATE<(SYSDATE - 15) AND (STATUS='proc_halted' OR STATUS='proc_failed')
ORDER BY START_DATE DESC;
SET SERVEROUTPUT OFF;
FIN_SQL
echo "Please check log file for more details : $(readlink -f $halted_proc_cancellation_logfile)"
exit
Sample SQL query output:
ROOT_PROC_ID PROC_ID PROC_NAME START_DATE STATUS ORDER_REF
pvm:0a123akpd pvm:0a123akkh FunctionalErrorProcess 28-NOV-19 01.24.35.115000000 PM pi_halted 2642277
pvm:0a122utrn pvm:0a122uun0 TechnicalErrorProcess 22-NOV-19 02.28.17.217000000 PM pi_halted 2642278
pvm:0a122utl2 pvm:0a122uu1t TechnicalErrorProcess 22-NOV-19 02.27.54.024000000 PM pi_halted 2642279
pvm:0a122utln pvm:0a122uu22 TechnicalErrorProcess 22-NOV-19 02.27.50.287000000 PM pi_halted 2642280
Assuming your sql query output is in output.txt:
awk 'NR!=1' output.txt | while read rootprocid undef
do
callApi $rootprocid
done
NR!=1 is to skip the 1st line which contains the header.
read rootprocid undef reading only the 1st column in rootprocid, rest goes to variable undef since it is not of interest.
callApi $rootprocid callAPI will be replaced with your actual api call.
I want to pass argument to a BigQuery script in shell, here is the example of script I wrote
#!/bin/bash
bq query --use_legacy_sql=false --destination_table=abc --append 'select * from `xyz.INFORMATION_SCHEMA.VIEWS` union all Select * from `def.VIEWS`) where table_name = "$1"'
when I run this script and pass the argument, I do not get any errors but no row is appended to the table. whereas when i specify the table_name as rty that row is appended to the table. What am I missing here?
When you run the script you'll get a prompt like:
Waiting on <BIGQUERY_JOB_ID> ... (0s) Current status: DONE
You can inspect the job in many ways, including the bqtool:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID>
If you have jq installed (sudo apt install jq) you can get just the translated query with:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID> | jq '.configuration.query.query'
which will get you something similar to:
select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name = \"$1\"
As you can see the variable is not correctly escaped so no table matches the WHERE filter. To avoid this you can enclose the query in double quotes and the variable in single ones like this:
#!/bin/bash
bq query \
--use_legacy_sql=false \
--destination_table=xyz.abc \
--append \
"select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name='$1'"
You can get the INFORMATION_SCHEMA.VIEWS: command not found error if using back-ticks. You can omit or escape them with a backslash:
"select * from \`xyz\`.INFORMATION_SCHEMA.VIEWS where table_name='$1'"
I am trying to fetch a row with more than one column value to different shell variables. Infact I found that at a time all the column values can be stored to single shell variable. But how can I put those column values to seperate shell variables. Below is an example I am trying for time being
function sqlQuery {
sqlplus -S shiyas/********* <<'EOF'
set heading OFF termout ON trimout ON feedback OFF
set pagesize 0
SELECT name,open_mode from v$database;
EOF
}
OUTPUT="$( sqlQuery )"
echo $OUTPUT
Here I am getting the output as
ORCL READ WRITE
But my requirement is column values ORCL, READ WRITE should get assigned to different shell variable.
I tried the below of parsing.
echo "$OUTPUT" | while read name open_mode
but it was throwing unexpected end of file error.
-bash-3.2$ sh call_sql_col_val_1.sh
ORCL READ WRITE
call_sql_col_val_1.sh: line 18: syntax error: unexpected end of file
Please let me know what concept I can use to fetch a single row column values to different shell variables.
I do this via eval myself:
oracle#******:/*****> cat test.sh
#!/bin/bash
function sqlQuery {
sqlplus -S / as sysdba <<'EOF'
set heading OFF termout ON trimout ON feedback OFF
set pagesize 0
SELECT name,open_mode from v$database;
EOF
}
eval x=(`sqlQuery`)
NAME=${x[0]}
OPEN_MODE="${x[1]} ${x[2]}"
echo NAME IS $NAME
echo OPEN_MODE IS $OPEN_MODE
So we are running the same function you have above, passing it into x and running it through eval to handle the delimitation. Then you have an array and call call is as such: x[0] for the first item, for example.
Output is:
oracle#******:/******> sh test.sh
NAME IS ******
OPEN_MODE IS READ WRITE
I am trying to export a few columns from a table as encoded integer.
Basically I want to use a bash script to pass the SQL command to mclient as command line argument. My bash script looks like:
#!/bin/bash
dir=`pwd`
for col in occupation native_country martial_status race sex
do
mclient -d adult -s \
"create function encode${col}(s varchar(200)) returns int begin return (select code from ${col}_dict where ${col}=s); end;"
mclient -d adult -s \
"COPY (select encode${col}($col) from adult) INTO '${dir}/${col}.txt' NULL AS '0'"
mclient -d adult -s \
"drop function encode${col}"
done
In each iteration, I want to create a SQL function on the fly. Then use the function to encode an attribute and export it to a text file. And lastly drop the function.
However, the output strangely contains some monster characters,
as if it can't recognize the function name.
If I remove the second mclient command, the other operations are successful.
operation successful
Function 'X�..X�' not defined
operation successful
operation successful
Function '��"X�.X�' not defined
operation successful
operation successful
Function ' X�.PX�' not defined