How to write a Sqoop Job using shell script - sqoop

Question

My use case:
Day by day hourly tables gets created in mysql db. I need to move them everyday to HDFS using Sqoop and process the HDFS data using Impala.
How to write a shell script or job only to move the tables data that are newly created to HDFS(existing file system) periodically?
Say today is 3rd of Jan 2016, when I run my job today then 2nd Jan 2016 data should be moved from mysql to HDFS like wide everyday it should move the data of previous day.
Daily I need to run my Impala queries on this HDFS cluster and generate a report.
How to process this whole data using Impala and generate a report?

Answer 1

Sqoop supports incremental import: http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_incremental_imports
Sqoop-import can use either a last-modified timestamp, or an always-increasing row ID to decide which rows to import. You need to provide a --last-value parameter. You could store the last value in between jobs, or maybe retrieve it from your Impala database before running the job.

Answer 2

The best way would be to get shell script that takes 2 arguments. 1 would be the name of the table and other would be hdfs path because those would be the only 2 factors that change in your scenarios. Below is sample script that you can put into .sh and run in bash.
!/bin/bash
TABLENAME=${^^1}
HDFSPATH=${^^2}
NOW=$(date +"%m-%d-%Y-%H-%M-%S")
sqoop --import --connect jdbc:db2://mystsrem:60000/SCHEMA \
--username username \
--password-file password \
--query "select * from ${TABLENAME} \$CONDITIONS" \
-m 1 \
--delete-target-dir \
--target-dir ${HDFSPATH} \
--fetch-size 30000 \
--class-name ${TABLENAME} \
--fields-terminated-by '\01' \
--lines-terminated-by '\n' \
--escaped-by '\' \
--verbose &> logonly/${TABLENAME}_import_${NOW}.log
OPTIONAL:
if you need to import into hive table
--hive-drop-import-delims \
--hive-import \
--hive-overwrite \
--hive-table HiveSchema.${TABLENAME}