I am trying to a sqoop action in oozie, but mysql-connector-java.jar is not present in /user/oozie/share/lib/sqoop, because of no permission I am not able to add the jar as of now,
Is there any way or workaround to include mysql-connector-java.jar in workflow.xml
I have placed the jar in sqoop apps / lib directory, but its not working
In general, the Hadoop administrator should keep all the common library in Hadoop distribution to make the usage more efficient, if not, give a try to the following -jarfile option
sqoop import \
-libjars /file/location/path/mysql-connector-java.jar \
--connect jdbc:mysql://localhost:3306:3306/retail_db \
--username root \
--password xyzpwd \
--table order_items \
--target-dir /user/cloudera/landing_zone/sqoop_import/order_items
as per sqoop documentation:
-libjars specify a comma separated jar files to include in the classpath. The -files, -libjars, and -archives arguments are not typically used with Sqoop, but they are included as part of Hadoop’s internal argument-parsing system.
Related
I'm trying to merge the incremented data on hdfs using sqoop, this is the sample code I found on google https://developer.ibm.com/hadoop/2017/02/28/typical-scenario-sqoop-incremental-import-merge/
I do not understand what is that jar-file there and class-name, path to which jar file should I provide and what class name?
Can someone help me understand? Thank you.
sqoop merge --new-data /apps/hive/warehouse/student/part-m-00000
--onto /apps/hive/warehouse/student/part-m-00000_copy_1
--target-dir /tmp/sqoop_merge
--jar-file /tmp/sqoop-ambari-qa/compile/9062c87c959e4090dcec5995a439b514/TIME.jar
--class-name TIME
--merge-key TIME
I used Codegen to create the jar file later could see the class name as well. This is the code I found to create the jar file,
sqoop codegen \
--connect jdbc:sqlserver://localhost/<db> \
--username <username> --password <password> \
--table <tablename from database>
by the end of the execution you will get an output like,
18/01/16 11:44:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-user1/compile/6430d9e2fe24cec8b2cb13f684806ca6/student.jar
after which to check the class-name, I did
$ cd /tmp/sqoop-user1/compile/6430d9e2fe24cec8b2cb13f684806ca6/
:/tmp/sqoop-user1/compile/6430d9e2fe24cec8b2cb13f684806ca6$ ls
That will give you class name,jar etc,
student.class student.jar student.java
Thank you.
Does sqoop import/export create java classes? If it does so, where can I see these generated classes. What is the location of these class files?
Does sqoop import/export create java classes?
Yes
If it does so, where can I see these generated classes. What is the location of these class files?
It automatically generates a java file of same table name in the
current path of local system.
You can use --outdir to provide your own path.
Updated as per comment
You can use codegen command for this:
sqoop codegen \
--connect jdbc:mysql://localhost/databasename\
--username username\
--password password\
--table tablename
After the command is executed successfully there will be a path at the end where you can see the java files.
This is the complete flow of sqoop commands
User---> SQOOP CLI cmd ----> Sqoop Code GEN -----> Sqoop JAR Writer
----> JAR submission ---> ResourceManager ----> MR operation (5phases) ----> HDFS ----> Ack to Sqoop by MR program
**
Sqoop internally uses MapReducev1 or v2 for its execution(Getting data from DB and Storing the same in HDFS in comma delimited values). And it first creates a .java source file for the map-reduce prg and pakages in jar and then submits.
The .java is created in the current local directory with name of table.
sqoop import --connect jdbc:mysql://localhost/hadoop --table employee -m 1
In this case a "employee.java" is created .
I am trying to get some table data imported from PostgreSQL to HDFS using Sqoop. Now due to licensing constraints, Sqoop does not come packaged with JDBC drivers for all JDBC compliant databases. PostgreSQL is one of them. In order to interact with this database, Sqoop needs the relevant JDBC driver to be installed into a preset classpath (typically $SQOOP_HOME/lib).
In my case, the Hadoop administrator does not provide me write access to this predefined classpath. Is there any alternate way to instruct Sqoop client to look into some path (say, my home directory) instead of or in addition to the preset location?
I looked into the official Apache documentation and searched the internet, but could not fetch any answer. Could anyone please help?
Thanks !
I got this working yesterday. Below are the steps to follow.
Download the appropriate JDBC driver from here
Put the jar file under the directory of choice. I chose
the hadoop cluster user's home directory i.e. /home/myuser
export HADOOP_CLASSPATH="/home/myuser/postgresql-9.4.1209.jar"
(replace /home/myuser/postgresql-9.4.1209.jar with your path and jar file name)
To perform Sqoop import you may use the below command.
sqoop import
--connect 'jdbc:postgresql://<postgres_server_url>:<postgres_port>/<db_name>'
--username <db_user_name>
--password <db_user_password>
--table <db_table_name>
--warehouse-dir <existing_empty_hdfs_directory>
To perform Sqoop export you may use the below command.
sqoop export
--connect 'jdbc:postgresql://<postgres_server_url>:<postgres_port>/<db_name>'
--username <db_user_name>
--password <db_user_password>
--table <db_table_name>
--export-dir <existing_hdfs_path_containing_export_data>
As per Sqoop docs,
-libjars <comma separated list of jars>- specify comma separated jar files to include in the classpath.
Make sure you use -libjars as first argument in the command.
EDIT :
According to docs,
The -files, -libjars, and -archives arguments are not typically used with Sqoop, but they are included as part of Hadoop’s internal argument-parsing system.
So, JDBC client jars need to be put at $SQOOP_HOME/lib.
I had recently experienced issue with this -libjars option. It doesn't work perfectly. Probably this issue is propagated from Hadoop jar command line option. Possible option is to specify your extra jars using HADOOP_CLASSPATH environmental variable.
You have to export path to your driver jar file.
export HADOOP_CLASSPATH=<path_to_driver_jar>.jar
After this, it can correctly pick up the jar file you specified. -libjars option doesn't correctly pick the file. I noticed this in sqoop version 1.4.6.
I want to create an Oozie workflow that would use the MapReduceIndexerTool to take my data and index it. I've managed to get it working using a Shell action, which calls my script to execute the following command:
hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar \
org.apache.solr.hadoop.MapReduceIndexerTool \
-D 'mapred.child.java.opts=-Xmx500m' \
--morphline-file morphline.conf \
--output-dir hdfs://cloudera1:8020/user/nicolas/outdir \
--verbose --go-live --zk-host cloudera2:2181/solr \
--collection Test_Collection hdfs:///user/nicolas/indir
It finds all the files and directories it needs, and the workflow will finish successfully. However, I would like to add my custom Morphlines command to modify some of the data. I have been following the kitesdk guide to do just that. I packaged my code into a jar and uploaded it to hdfs://cloudera1:8020/user/nicolas/custom-command.jar through the Hue File Browser. I've also updated my morphline.conf so that I import my package, and use my command. If I just include the file in my workflow, the following error occurs:
Error: org.kitesdk.morphline.api.MorphlineCompilationException: No command builder registered for name: tweakData ...
I'm assuming that the MapReduceIndexerTool is having trouble finding my jar. So, I decided to add the --libjars parameter to my script:
hadoop jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar \
org.apache.solr.hadoop.MapReduceIndexerTool \
--libjars hdfs://cloudera1:8020/user/nicolas/custom-command.jar ...
When I do that, a different error occurs:
WARN org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:yarn (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist:
hdfs://cloudera1:8020/tmp/hadoop-yarn/mapred/staging/yarn1033647717/.staging/job_local1033647717_0001/libjars/custom-command.jar
TD;DR How to I include the jar for my custom Morphlines command so it is found by Oozie / YARN?
You can add the jars to the lib sub-directory of the oozie wf HDFS directory or add the jar to the tag in oozie shell action.
--libjars requires a file on the local file system (rather than an HDFS file).
i am new to hadoop and while practicing sqoop i have got this error message , the command i have used is
i created an import.txt file and in that i used
import --connect jdbc:mysql://localhost/hadoopdb --username hadoop -P and placed this file on HDFS.
while importing i have given this file to the sqoop tool using the --options-file command. so the final command i have given at the command promt is as follows,
sqoop --options-file /user/cloudera/import.txt --table employee
after hiting the enter key i have got the following error message
sqoop --options-file /user/cloudera/import.txt --table employee
13/10/16 13:43:12 ERROR sqoop.Sqoop: Error while expanding arguments
java.lang.Exception: Unable to read options file: /user/cloudera/import.txt
at org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:102)
at com.cloudera.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:33)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:201)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
Caused by: java.io.FileNotFoundException: /user/cloudera/import.txt (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at java.io.FileReader.<init>(FileReader.java:55)
at org.apache.sqoop.util.OptionsFileUtil.expandArguments(OptionsFileUtil.java:70)
... 4 more
Unable to read options file: /user/cloudera/import.txt
can anyone tell me why the error is coming.
Thanks in advance.
--option-file path should be local directory. Don't use HDFS directory.
sqoop --options-file /home/cloudera/import.txt --table employee
I got the same issue. I solved it using the following approach.
In the options file you have to mention tools,commands and their arguments line by line
In your case, Your options file "import.txt" should be created like this
$cat > import.txt
import
--connect
jdbc:mysql://localhost/hadoopdb
--username
hadoop
-P
After you created the options file, you can use this to import the table
sqoop --options-file /user/cloudera/import.txt --table employee
Hope this works. Key is you have to mention tools and arguments line by line.
For more understanding on this refer this link
Sqoop User Guide by Apache.org
Correct me if I am wrong.
If you are Calling Sqoop from Oozie and you are facing the same issue -Unable to read options file.
Then you need place the option-file inside workflow location and specify the file in sqoop action-files and also you need to change the permission for that file to chmod 674(When workflow is running in oozie, it will run with sqoop user so it is mandatory to change permission).
This will resolve the error.
I put option file in local directory, It worked.
Also
Argument and value should in different line.
like
--where
'sal > 5000'
and not like
--where 'sal > 5000'
[cloudera#quickstart sqoop]$ sqoop --options-file
/home/cloudera/Desktop/SqoopOptions.txt --table departments --username root --
password cloudera -m 1 --target-dir jan1301
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
No such sqoop tool: import --connect jdbc:mysql://localhost/retail_db --username root --password cloudera. See 'sqoop help'.
I have received above error when I define the SqoopOptions.txt file data in a single line.
The issue resolved when i define each parameter & value in different line like below.
if you are trying on single node cluster, option file can be placed under local file system.
your option file should be like this.
import
--connect
"jdbc:mysql://localhost:3306/sakila"
--username root
-P
for each parameter there should be a next line space.
once you saved the option file then use below command.
sqoop --options-file "your optionfile location" --table abc
hope this should work as this option is perfectly working for me.
Thanks,
Suresh.