I am importing data from Oracle to Hadoop using Sqoop. In Oracle table I have approximately 2 Millions records with the primary-key which is I am providing as split-by field.
My sqoop job is getting completed and I am getting correct data and job is running for 30 Min till now all good.
When I check the output file I see first file is round 1.4 GB, Second file is around 157.2 MB and last file (20th File) is around 10.4 MB whereas all the other files from 3rd to 19th are 0 bytes.
I am setting -m 20 because I want to run 20 mappers for my job.
here is the sqoop command :
sqoop import --connect "CONNECTION_STRING" --query "SELECT * FROM WHERE AND \$CONDITIONS" --split-by .ID --target-dir /output_data -m 20
Note : My cluster is capable enough to handle 20 mappers and database also capable to handle 20 request at a time.
Any thought?
Dharmesh
From http://sqoop.apache.org/docs/1.4.5/SqoopUserGuide.html#_controlling_parallelism...
If the actual values for the primary key are not uniformly distributed
across its range, then this can result in unbalanced tasks.
The --split-by argument can be used to choose a column with better distribution. Normally, this will vary by data type.
Try using a different --split-by field for better load balancing.
This is because the Primary Key (ID) is not uniformly distributed. Hence, your mappers are not being used appopriately. So you must use some other field for splitting that is uniformly distributed.
Related
One of my Oracle table contains 265 million records, I need to push that table from Oracle database to hdfs but this table doesnt have any primary key/Unique column. Hence, I cant use multiple mappers. If I use multiple mappers, I have to specify Split by column.
Whats the best way to sqoop the table.
Any leads are appreciated.
In order to use multiple mappers, you will need a --split-by parameter. The best column to choose is one that is not null in all 265m rows and evenly distributed. Primary key meets that criteria because it is sequential and in all rows.
Any column that is evenly distributed across the data set could be a good choice for a --split-by choice. The link #yammanuruarun posted includes the --boundary-query argument to help limit the work the RDBMS has to do to return those rows. I suggest using a Fibbonacci sequence for your -m 1,2,3,5,8.
Also, check out:
How to find optimal number of mappers when running Sqoop import and export?
I want to import big table from oracle database to HDFS using Sqoop.
As the table size is huge and it is having primary key sqoop can run multiple mappers parallel.
I have some questions in
1)Due to bad record in oracle database, one mapper got exception and others are running fine. So all the job will get failed or except one mapper data all other mappers will write data in HDFS?
2)Is sqoop is intelligent enough to run parallel mappers if we hive --m option.
If we give --m 4 then sqoop can increase mappers based on tables size or it will run with 4 only?
Is any body came across this kind of scenario??
Based on my knowledge.
If one mapper gets failed, The sqoop process will try to kill other mapper. The process won't delete the data from HDFS. You can see some of the data been created in your HDFS location.
When we specify number of mapper (using -m x option) the program will use at most x mapper.
I am trying to understand the reason behind the default maximum mappers in a sqoop job. Can we set more than four mappers in a sqoop job to achieve higher parallelism.
If you are using integer column in your split-by then the default number of mappers are 4. And it is strongly recomonded that you always use integer column not the string/char/Text column. see the code here for more explaination. https://github.com/apache/sqoop/blob/660f3e8ad07758aabf0a9b6ede3accdfac5fb1be/src/java/org/apache/sqoop/mapreduce/db/TextSplitter.java#L100
Yes you can give increase/decrease the parallelism by specifying -m
from Sqoop Guide
Sqoop imports data in parallel from most database sources. You can specify the number of map tasks (parallel processes) to use to perform the import by using the -m or --num-mappers argument. Each of these arguments takes an integer value which corresponds to the degree of parallelism to employ. By default, four tasks are used. Some databases may see improved performance by increasing this value to 8 or 16. Do not increase the degree of parallelism greater than that available within your MapReduce cluster; tasks will run serially and will likely increase the amount of time required to perform the import. Likewise, do not increase the degree of parallism higher than that which your database can reasonably support. Connecting 100 concurrent clients to your database may increase the load on the database server to a point where performance suffers as a result.
When performing parallel imports, Sqoop needs a criterion by which it can split the workload. Sqoop uses a splitting column to split the workload. By default, Sqoop will identify the primary key column (if present) in a table and use it as the splitting column. The low and high values for the splitting column are retrieved from the database, and the map tasks operate on evenly-sized components of the total range. For example, if you had a table with a primary key column of id whose minimum value was 0 and maximum value was 1000, and Sqoop was directed to use 4 tasks, Sqoop would run four processes which each execute SQL statements of the form SELECT * FROM sometable WHERE id >= lo AND id < hi, with (lo, hi) set to (0, 250), (250, 500), (500, 750), and (750, 1001) in the different tasks.
If the actual values for the primary key are not uniformly distributed across its range, then this can result in unbalanced tasks. You should explicitly choose a different column with the --split-by argument. For example, --split-by employee_id. Sqoop cannot currently split on multi-column indices. If your table has no index column, or has a multi-column key, then you must also manually choose a splitting column.
My guess is that 4 is a default number that works well on practice for most use cases. Use the parameter --num-mappers if you want Sqoop to use a different number of mappers. For example, to use 8 concurrent tasks you would use the following sqoop command:
sqoop import \
--connect jdbc:mysql://mysql.example.com/testdb \
--username abcdef \
--password 123456 \
--table test \
--num-mappers 8
Using more mappers will lead to a higher number of concurrent data transfer tasks, which can result in faster job completion. However, it will also increase the load on the database as Sqoop will execute more concurrent queries. You might want to keep this in mind of you are pulling data from your production environment.
when we don't mention the number of mappers while transferring the data from RDBMS to HDFS file system sqoop will use default number of mapper 4. Sqoop imports data in parallel from most database sources. Sqoop only uses mappers as it does parallel import and export.
If you're not mentioning number of mapper in sqoop it will by default use 4 mapper in parallel to do the sqoop import. If you want to use more then 4 mapper you can use --num-mappers in your sqoop command you can use any number of mapper
Also, if you are not sure you have primary key or not in source table --autoreset-to-one-mapper come handy in that case. If there is primary key it will use the mentioned number of mappers to execute the job or else it will just use one mapper to import the table without primary key
sqoop-import \
-- connect jdbc:mysql://localhost/databasename \
-- username root \
-- password xxxxxxxx \
-- warehouse-dir /directory/path/from/home \
-- autoreset-to-one-mapper \
-- num-mappers 6
Also, it comes handy when you doing sqoop import-all-tables to import multiple tables for all the tables with primary key it will use the mentioned number of mappers and for all the tables without primary key it will reset the number of mappers to 1 without failing the job.
Note: The tables without Primary Key use only one mapper for sqoop import until and unless you're not giving --split-by column
What happens internally when we write --split-by in sqoop?
Example:
sqoop import --connect jdbc:mysql://localhost/test --username root --password training123 --query 'select * from transaction where $CONDITIONS' --split-by Txnid --target-dir input/transaction
Hadoop MAP Reduce is all about divide and conquer .
In order to partition data into multiple independent slices that will be transferred in a parallel manner, Sqoop needs to find the minimum and maximum value of the column specified in the --split-by parameter
When using the split-by option, you should choose a column which contains values that are uniformly distributed.
in the query we are telling data is evenly distributed on base column 'Txnid' and use the column for making splits.
--split-by : It is used to specify the column of the table used to generate splits for imports. This means that it specifies which column will be used to create the split while importing the data into your cluster. It can be used to enhance the import performance by achieving greater parallelism. Sqoop creates splits based on values in a particular column of the table which is specified by --split-by by the user through the import command. If it is not available, the primary key of the input table is used to create the splits.
Reason to use : Sometimes the primary key doesn't have an even distribution of values between the min and max values(which is used to create the splits if --split-by is not available). In such a situation you can specify some other column which has proper distribution of data to create splits for efficient imports.
--split-by <column-name> - Column of the table used to split work units
Reference: Sqoop User Guide
It specifies which column will be used to create the split while importing the data into your cluster. It can be used to enhance the import performance by achieving greater parallelism.
Sqoop creates splits based on values in a particular column of the table which is specified by --split-by by the user through the import command. If it is not available, the primary key of the input table is used to create the splits. We can choose the column through --split-by which can result in the best splitting and thus increasing parallelism and better performance.
split-by in sqoop is used to create input splits for the mapper. It is very useful for parallelism factor as splitting imposes the job to run faster.
Stack : Installed HDP-2.3.2.0-2950 using Ambari 2.1
The source is a MS SQL database of around 1.6TB and around 25 tables
The ultimate objective is to check if the existing queries can run faster on the HDP
There isn't a luxury of time and availability to import the data several times, hence, the import has to be done once and the Hive tables, queries etc. need to be experimented with, for example, first create a normal, partitioned table in ORC. If it doesn't suffice, try indexes and so on. Possibly, we will also evaluate the Parquet format and so on
4.As a solution to 4., I decided to first import the tables onto HDFS in Avro format for example :
sqoop import --connect 'jdbc:sqlserver://server;database=dbname' --username someuser --password somepassword --as-avrodatafile --num-mappers 8 --table tablename --warehouse-dir /dataload/tohdfs/ --verbose
Now I plan to create a Hive table but I have some questions mentioned here.
My question is that given all the points above, what is the safest(in terms of time and NOT messing the HDFS etc.) approach - to first bring onto HDFS, create Hive tables and experiment or directly import in Hive(I dunno if now I delete these tables and wish to start afresh, do I have to re-import the data)
For Loading, you can try these options
1) You can do a mysql import to csv file that will be stored in your Linux file system as backup then do a distcp to HDFS.
2) As mentioned, you can do a Sqoop import and load the data to Hive table (parent_table).
For checking the performance using different formats & Partition table, you can use CTAS (Create Table As Select) queries, where you can create new tables from the base table (parent_table). In CTAS, you can mention the format like parque or avro etc and partition options is also there.
Even if you delete new tables created by CTAS, the base table will be there.
Based on my experience, Parque + partition will give a best performance, but it also depends on your data.
I see that the connection and settings are all correct. But I dint see --fetch-size in the query. By default the --fetch-size is 1000 which would take forever in your case. If the no of columns are less. I would recommend increasing the --fetch-size 10000. I have gone up to 50000 when the no of columns are less than 50. Maybe 20000 if you have 100 columns. I would recommend checking the size of data per row and then decide. If there is one column which has size greater than 1MB data in it. Then I would not recommend anything above 1000.