I am trying to use sqoop import-all-tables to get the data from SQL Server into HDFS from a particular database.
After it imports all the expected tables from the DB successfully, it also tries to import system tables in the DB. Is there a way t force sqoop to import only non-system tables?
Thanks.
It looks like a couple of system tables are listed as user tables. Hence the issue.
Related
I have created the necessary storage plugins and the relevant databases in hive show up when issuing the show database command.
When using one of the hive databases though using the use command, I found that I cannot select any tables which are within that database. Looking further, when issuing the show table command, no tables within that database show up via Apache Drill whereas they appear fine in Hive.
Is there anything I am missing by any chance in terms of granting permission via Hive to any user? How exactly does Apache Drill connect to Hive to run the relevant jobs?
Appreciate your responses.
Show tables; will not list hive tables as of now. It's better to create views on top of hive tables. These Views will show up on show tables; command.
I want to import data into hive from greenplum using sqoop.
I am able to import successfully data from default schema of greenplum for user.
But, I am not able to fetch data from table present in other schemas of greenplum.
I tried various option.
Can you please help?
Thanks in advance.
Which sqoop-version do you use?
With v1.4.3 you can set schema-parameter.
With v1.4.2 you can use freeform query (--query) with schema.
I tried and it works fine.
Sqoop itself don't have a notion of "schema". Some specialized connectors (PostgreSQL, Microsoft SQL Server) are exposing user ability to specify a schema, but as Sqoop don't have specialized connector for Greenplum it won't help you here.
You should be able to use query-based import instead of table and specify the schema name in the query, e.g. something like:
sqoop import --query "select * from schema.tablename where $CONDITIONS"
You can take the advantage of custom schema
try with
--schema <<schema_name>>
floks
Here i have a client question. I want to get the tables from sql server(RDBMS) to my hdfs (hadoop cluster). But the servers are in different location.
1)Which is the best way to access the serve,but data is in huge amount.
2)connecting with one sever is okay, we have many servers around the globe we have to get the data from this servers.
3)Can we connect with sqoop remotly to get the data to HDFS.
Your question is a little bit unclear, but yes, you can use sqoop to import the data from your servers into HDFS. You need to specify the connection parameters when importing the data:
sqoop import --connect <JDBC connection string> --table <tablename> --username <username> --password <password>
If you need to do multiple imports from multiple servers, I suggest you try Oozie to automate these imports. You can find a tutorial to achieve that here.
Before writing down the sqoop import, you need to have a user for each of the remote node which is to be identified by your local DB. For Ex:
create user 'username'#'<ip of remote node>' IDENTIFIED BY 'password';
You must also have to ensure about the grant permissions to these users depending on your requirement.
It's then you can frame the sqoop import, one example as below:
$SQOOP_HOME/bin/sqoop import --connect jdbc:mysql://<ip address of remote server node> :port_number/<database_name> --username user --password password --table <table to import>
This question is 5 months old for this answer so I'm hoping the issue would have been resolved but in case someone wanted to go to a step by step procedure for this requirement.
Regards,
Adil
I need to export a subset of an Oracle table and import it in another Oracle instance. The export using expdp works pretty well but when I try to import the data in the other instance using impdp tool it fails because there are dependencies (foreign keys) missing. Is there any option to force expdp tool to export all required dependencies as well?
no.
You should makie sure your dump set is complete.
What you could try is to use impdp to generate the sql file, apply the generated sql to the other database to only create the table[s]. For this you might need to do some edit work on the generated sql until it fits your task.
Next use impdp with contents=data_only to import the rows in the pre-create table[s]
i am newbie to oracle and i like to export database from remote database and import it on local machine. eOn both machines i have oracle 10.2.
I need to know how to export/import schema and data from oracle 10.2 using SQLDeveloper 3.0.0.4.
To export from remote database, i have used export Tool-> Database Export -> export wizard.
and at the end i have got only sql file with DDL and DML statements but somewhere in file it is written
"Cannot render Table DDL for object XXX.TABLE_NAME with DBMS_METADATA attempting internal generator error.
I have ignored previously mentioned message and tried to run those DDL and DML statements but all this ended up with errors.
Is it possible that all this tied with read-only database user? More over, i dont find any table under tables but also tables under other users in SqlDeveloper.
Thanks in advance
As a test, can you select one object in the tree, and navigate to the script panel? SQLDEV also uses DBMS_METADATA to generate those scripts.
Also, as a work-around, try using DataPump to export and import your data. It will be much more efficient for moving around larger schemas.
Your note about not seeing tables under indicates your schema doesn't actually own any tables. You may be working with synonyms that allow you to query objects as if they are in your account. You could be running into a privilege issue, but your error message doesn't indicate that. Error messages often come in bunches, and the very first one is usually the most important.
If you could try using the EXPORT feature say against a very simple schema like SCOTT as a test, this should indicate whether there is a problem with your account settings or with the software.
I'm not sure with SQL Developer 3.0 but with version 3.1 you can follow this:
SQL Developer 3.1 Data Pump Wizards (expdp, impdp)