In sqoop i have 500 tables and i want to import 3 tables only, we use import-all-tables exclude operation, we have to mention 497 tables name, without mentioning 497 table name how we can import only 3 tables.
Use three single import commands
Related
I am trying to import all tables in a specific schema from Oracle database with Sqoop command :
sqoop import-all-tables --connect jdbc:oracle:thin:server:port:database --username x --password y --warehouse-dir warehouse-dir --hive-import --create-hive-table
But this Oracle database has more schemas and I need only to import all tables from one specific schema.
You should use the additional parameter --exclude-tables <tables>, which refers to a comma separated list of tables to exclude from import process.
https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
If you use --import-all-tables the tables to be imported will be those that the user who connects by jdbc has select privilege over. Basically, those tables that the user see in all_tables.
To limit the list, another option is to use a different user, one with limited select over the tables you want to export to.
I tried to import (TABLES, PROCEDURE, FUNCTION etc) from a dump file. I did a mistake by
executing KILL -9 <PROCESS_ID> while import was still going on.
So, I started to import again. Now, I did another mistake by NOT mentioning
TABLE_EXISTS_ACTION=TRUNCATE . So, tables have been imported with duplicate records.
I want to get rid of duplicate data. There are more than 500 tables involved.
I am planning to import again by first truncating the table and then importing data only.
Below is the import command I have come up with. Will this command import ONLY table data(records) by first
truncating the table and then insert only the data?
impdp DIRECTORY=MY_DIRECTORY dumpfile=EXP_MY_DUMP.dmp INCLUDE=TABLE_DATA TABLE_EXISTS_ACTION=TRUNCATE
I could try executing myself and find out if that works. But, I have already tried twice and failed.
Also, I don't want to again import INDEX, SEQUENCES etc. Just table records.
Remove INCLUDE=TABLE_DATA. That will not execute create table.. that should work.
Can we import tables using sqoop having primary and non primary at a time.Eg i am hvaing 200 primary key tables and 200 non priamry key tables in database .How can we import 400 tables at a time?
In addition to Jamie's answer:
You can add --autoreset-to-one-mapper tag in your sqoop import-all-tables... command.
Say you are using 8 mappers (-m 8) in your command. Then using above tag tables with primary keys will split as per the number of mappers and tables without primary keys will be loaded using 1 mappers.
so, overall your efficiency will improve.
Check 1st point of sqoop documentation for details.
Yes, you can add the flag --m 1 to the import command of all your tables (both the 200 with primary key and the 200 without it).
By adding this option, Sqoop will only use one mapper to retrieve all the data from the tables, so your command would look like something like this:
sqoop import-all-tables --connect your-database --username user --password pwd --m 1
how to do a multi table or selective table import in sqoop from MYSQL
I can see either we can import table by table or import-all-tables.
am looking for a way to import only certain tables from MYSQL.
Any help would be appreciated ?
Thanks
There is no way to put multiple tables in sqoop import command.
But you can use import-all-tables with --exclude-tables <tables> to exclude some tables while importing all the tables of a database.
For example
You have 5 tables (tbl1, tbl2, tbl3, tbl4, tbl5) in testdb database and you want to exclude tbl2 and tbl4.
Use the following command:
sqoop import-all-tables --connect jdbc:mysql://localhost:3306/testdb --exclude-tables tbl2,tbl4 .....
I can import SQL tables to Hive, however when I try to SQL view, I am getting errors.
Any ideas ?
From Sqoop documentation:
If your table has no index column, or has a multi-column key, then you must also manually choose a splitting column.
I think it's your case here. Provide additional option like:
--split-by tablex_id
where tablex_id is a field in your view that can be used as a primary key or index.
There is no any special command to import View form RDBMS.
Use simple import command with --split-by that sqoop refers to same as primary key.
you can import the rdbms views data but u have to pass -m 1 or --split-by in your command.