I'm trying to import a Table from Oracle to Hive using Sqoop. I used the following command:
sqoop-import --connect jdbc:<connection> --table test1 --username test --password test --hive-table hive_test --create-hive-table --hive-import -m 1
But this gives me the error
Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory <hdfs path> already exists
So I read online in many forums and it said that I should delete the directory and run the command again.
I did exactly that, but I still keep getting the error.
You need to understand working of Sqoop hive Import.
Import data to HDFS <some-dir>
Create hive table <some-table> IF NOT EXISTS
LOAD data inpath '<some-dir>' into table <some-table>
You are getting the error at step 1.
Output directory <hdfs path> already exists
Delete this <hdfs path> and proceed.
Better way:
No need to delete this manually everytime.
Use --delete-target-dir in the command. It will
Delete the import target directory if it exists
P.S. No need to use --create-hive-table with --hive-import. --hive-import by default create table for you.
Hive store its Table data in hive warehouse on hdfs with table name as directory and usually have below path
/user/hive/warehouse/
need to delete table name directory.
hadoop fs -rmr /user/hive/warehouse/hive_test
Related
So I'm trying to import-all-tables into hive db, ie, user/hive/warehouse/... on hdfs, using the below command:
sqoop import-all-tables --connect "jdbc:sqlserver://<servername>;database=<dbname>" \
--username "<username>" \
--password "<password>" \
--warehouse-dir "/user/hive/warehouse/" \
--hive-import \
-m 1
In the testdatabase I have 3 tables, when mapreduce runs, the output is success,
ie, the mapreduce job is 100% complete but the file is not found on hive db.
It’s basically getting overwritten by the last table, try removing the forward slash at the end of the directory path. For the tests I would suggest not to use the warehouse directory, use something like ‘/tmp/sqoop/allTables’
There is a another way
1. Create a hive database pointing to a location says "targetLocation"
2. Create hcatalog table in your sqoop import using previously created database.
3. Use target-directory import options to point that targetLocation.
you doesn't need need to define warehouse directory.just define hive database it will automatically find out working directory.
sqoop import-all-tables --connect "jdbc:sqlserver://xxx.xxx.x.xxx:xxxx;databaseName=master" --username xxxxxx --password xxxxxxx --hive-import --create-hive-table --hive-database test -m 1
it will just run like rocket.
hope it work for you....
I am new to hadoop, trying to run following sqoop command:
sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --table employees --username root --password-file ${user.home}/.password.txt --target-dir /data/sqoop/eg4/ -m 2
but it gives me an error as
bash: ${user.home}/.password.txt: bad substitution
I tried the way it is given in the docs, but nothing happened. same error every time.
Step by step guide would be appreciated. Thanks
Sqoop expects the password file on HDFS location. Try copying the file to a location on HDFS and specify that path. Also check the read permission of the file. Read permission should be to given to home directory user.
I am new to hadoop and have recently started work on sqoop. While trying to export a table from hadoop to sql server i am getting the following error:
input path does not exist hdfs://sandbox:8020/user/root/
The command i am using is :
sqoop export --connect "jdbc:sqlserver://;username=;password=xxxxx;database=" --table --export-dir /user/root/ -input-fields-terminated-by " "
Could you please guide what i am missing here.
Also could you please let me know the command to navigate to the hadoop directory where the tables are stored.
For a proper sqoop export, Sqoop requires the complete data file location. You cant just specify the root folder.
Try specifying the complete src path
sqoop export --connect jdbc:oracle:thin:<>:1521/<> --username <> --password <> --table <> --export-dir hdfs://<>/user/<>/<> -m 1 --input-fields-terminated-by '|' --input-null-string '\\N' --input-null-non-string '\\N'
Hope this helps
Is it possible to export data from hive to Oracle DB using Sqoop for reporting purpose since i dont want to make any changes in client applications.
Regards,
Bhagwant Bhobe
Use the insert overwrite directory option with Hive for the output of the query to be written to a file and then use the Sqoop export option to insert the data in the file into RDBMs. A work-flow using Oozie or Azkaban (does Azkaban supports Oozie and Hive tasks?) can also be used to automate this.
By using sqoop export command you can export data from hive to oracle DB.
sqoop export --connect jdbc:oracle:thin:#ipaddress:portnumber:DBName --table tableName --export-dir /user/hive/warehouse/emp1 --username uname --password pwd --fields-terminated-by '\001' -m 1
in --export-dir specify the location of hive output directory.
We are using Cloudera CDH 4 and we are able to import tables from our Oracle databases into our HDFS warehouse as expected. The problem is we have 10's of thousands of tables inside our databases and sqoop only supports importing one table at a time.
What options are available for importing multiple tables into HDFS or Hive? For example what would be the best way of importing 200 tables from oracle into HDFS or Hive at a time?
The only solution i have seen so far is to create a sqoop job for each table import and then run them all individually. Since Hadoop is designed to work with large dataset it seems like there should be a better way though.
U can use " import-all-tables " option to load all tables into HDFS at one time .
sqoop import-all-tables --connect jdbc:mysql://localhost/sqoop --username root --password hadoop --target-dir '/Sqoop21/AllTables'
if we want to exclude some tables to load into hdfs we can use " --exclude-tables " option
Ex:
sqoop import-all-tables --connect jdbc:mysql://localhost/sqoop --username root --password hadoop --target-dir '/Sqoop21/AllTables' --exclude-tables <table1>,<tables2>
If we want to store in a specified directory then u can use " --warehouse-dir " option
Ex:
sqoop import-all-tables --connect jdbc:mysql://localhost/sqoop --username root --password hadoop --warehouse-dir '/Sqoop'
Assuming that the sqoop configuration for each table is the same, you can list all the tables you need to import and then iterate over them launching sqoop jobs (ideally launch them asynchronously). You can run the following to fetch the list of tables from Oracle:
SELECT owner, table_name FROM dba_tables reference
Sqoop does offer an option to import all tables. Check this link. There are some limitations though.
Modify sqoop source code and recompile it to your needs. The sqoop codebase is well documented and nicely arranged.
--target-dir is not a valid option when using import-all-tables.
To import all tables in particular directory, Use --warehouse-dir instead of --target-dir.
Example:
$ sqoop import-all-tables --connect jdbc:mysql://localhost/movies --username root --password xxxxx --warehouse-dir '/user/cloudera/sqoop/allMoviesTables' -m 1
The best option is do my shell script
Prepare a inputfile which has list of DBNAME.TABLENAME 2)The shell script will have this file as input, iterate line by line and execute sqoop statement for each line.
while read line;
do
DBNAME=`echo $line | cut -d'.' -f1`
tableName=`echo $line | cut -d'.' -f2`
sqoop import -Dmapreduce.job.queuename=$QUEUE_NAME --connect '$JDBC_URL;databaseName=$DBNAME;username=$USERNAME;password=$PASSWORD' --table $tableName --target-dir $DATA_COLLECTOR/$tableName --fields-terminated-by '\001' -m 1
done<inputFile
You can probably import multiple tables : http://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_literal_sqoop_import_all_tables_literal
You can use Sqoop "import-all-tables" feature to import all the tables in the database. This also has another parameter, --exclude-tables, along with which you can exclude some of the table that you don't want to import in the database.
Note: --exclude-tables only works with import-all-tables command.
importing multiple tables by sqoop if no of tables are very less.
Create sqoop import for each table as below .
sqoop import --connect jdbc:mysql://localhost/XXXX --username XXXX
password=XXXX
--table XXTABLE_1XX*
sqoop import --connect jdbc:mysql://localhost/XXXX --username XXXX
password=XXXX
--table XXTABLE_2XX*
and so on.
But what if no of tables are 100 or 1000 or even more. Below would be ideal solution.
In such scenario, preparing shell script which takes input from text file containing list of table names to be imported, iterate over, run the scoop import job for each table