Import Postgres dump into a table with CockroachDB - cockroachdb

After executing a pg_dump from PostgeSQL, I attempted to import the .sql file into CockroachDB, but received the following errors:
ERROR: unknown variable: "STATEMENT_TIMEOUT"
ERROR: unknown variable: "LOCK_TIMEOUT"
ERROR: unknown variable: "IDLE_IN_TRANSACTION_SESSION_TIMEOUT"
SET
SET
ERROR: unknown variable: "CHECK_FUNCTION_BODIES"
SET
ERROR: unknown variable: "ROW_SECURITY"
SET
ERROR: unknown variable: "DEFAULT_TABLESPACE"
ERROR: unknown variable: "DEFAULT_WITH_OIDS"
CREATE TABLE
ERROR: syntax error at or near "OWNER"
Any guidance?

CockroachDB has special support for using psql, which supports the COPY command (which is faster than batched INSERT statements).
You'll need to do two things:
Clean up the SQL file
Import it into CockroachDB (which sounds like you tried, but I'll include the steps here for anyone else who needs them):
Clean up the SQL File
After generating the .sql file, you need to perform a few editing steps before importing it:
Remove all statements from the file besides the CREATE TABLE and COPY statements.
Manually add the table's PRIMARY KEY constraint to the CREATE TABLE statement.
This has to be done manually because PostgreSQL attempts to add the primary key after creating the table, but CockroachDB requires the primary key be defined upon table creation.
Review any other constraints to ensure they're properly listed on the table.
Remove any unsupported elements, such as arrays.
Import Data
After reformatting the file, you can import it through psql:
$ psql -p [port] -h [node host] -d [database] -U [user] < [file name].sql
For reference, CockroachDB uses these defaults:
[port]: 26257
[user]: root

Related

Can I restore database clickhouse with help binary dump file?

I have docker container with clickhouse database. And I have backups from our production. But I tried many cases with restore. When I using command (in container): clickhouse-client --format Raw --file 21_02_04_db.backup, I get error Bad arguments: unrecognised option '--file'. Okay, I tried without this option: clickhouse-client --format Raw < 21_02_04_db.backup and I get an error:
Code: 62. DB::Exception: Syntax error: failed at position 1 ('') (line 1, col 1): �TimDateTime�#�]�{�]�k�]u �]0&�] 9�]�-�]��]�I�]U��]���]yy�]���]�\�]�\�]��]{B�]�j�]v)�]�d�]J��]�F�]�]�]��]t%�]t%�]���]��]F?�]�B�]��]W3�]��]E��]P��]a�. Unrecognized token: '' I use other backups but in many cases I get similar error. Maybe someone tried restore backup clickhouse with help binary data? I try many cases but I didn't search solution.
it looks like a Native format
it should something like
clickhouse-client -q 'insert into mytable format Native' < 21_02_04_db.backup

AWS DMS - Oracle to PG RDS full load operation error - failed to load data from csv file

I am trying to move data from a oracle instance to postgres RDS using DMS. I am only doing a full load operation and I have disabled all the foreign keys on the target. I also made sure that the datatypes are not mismatched between columns for the same tables. I tried both 'Do Nothing' and 'Truncate' for the Target Table preparation mode and when I run the task, several tables are failing with below error messages:
[TARGET_LOAD ]E: Command failed to load data with exit error code 1, Command output: <truncated> [1020403] (csv_target.c:981)
[TARGET_LOAD ]E: Failed to wait for previous run [1020403] (csv_target.c:1578)
[TARGET_LOAD ]E: Failed to load data from csv file. [1020403] (odbc_endpoint_imp.c:5648)
[TARGET_LOAD ]E: Handling End of table 'public'.'SKEWED_VALUES' loading failed by subtask 6 thread 1 [1020403] (endpointshell.c:2416)
DMS doesn't give the correct error information and I am not able to understand what the above error messages mean.
When I use 'Drop tables on target' for the Target table preparation mode, it works but it creates the datatypes of the columns in a different way which I don't want.
Any help would be appreciated.
To troubleshoot my case, I created a copy of the task that only loaded the one problem table, and upped all the logging severities to "Detailed debug". Then I was able to see this:
[TARGET_LOAD ]D: RetCode: SQL_SUCCESS_WITH_INFO SqlState: 42622 NativeError: -1 Message: NOTICE: identifier "diagnosticinterpretationrequestdata_diagnosticinterpretationcode" will be truncated to "diagnosticinterpretationrequestdata_diagnosticinterpretationcod" (ar_odbc_stmt.c:4720)
In the RDS logs for the target DB I found:
2021-10-11 14:30:36 UTC:...:[19259]:ERROR: invalid input syntax for integer: ""
2021-10-11 14:30:36 UTC:...:[19259]:CONTEXT: COPY diagnosticinterpretationrequest, line 1, column diagnosticinterpretationrequestdata_diagnosticinterpretationcod: ""
2021-10-11 14:30:36 UTC:...:[19259]:STATEMENT: COPY "myschema"."diagnosticinterpretationrequest" FROM STDIN WITH DELIMITER ',' CSV NULL 'attNULL' ESCAPE '\'
I found that if I added a table mapping rule to explicitly rename the column to truncate the name within Postgres's limit for identifier length, then things ran ok.
{
"rule-type": "transformation",
"rule-id": "1",
"rule-name": "1",
"rule-target": "column",
"object-locator": {
"schema-name": "%",
"table-name": "%",
"column-name": "diagnosticinterpretationrequestdata_diagnosticinterpretationcode"
},
"rule-action": "rename",
"value": "diagnosticinterpretationrequestdata_diagnosticinterpretationcod",
"old-value": null
},

greenplum hadoop in greenplum how to configure to generate gphdfs.so

in greenplum how to configure to generate gphdfs.so?
when I
postgres=# CREATE EXTERNAL TABLE csv_hdfs_lineitem (like a) LOCATION ( 'gphdfs://xxxxx/gptest/lineitem.csv' ) FORMAT 'text' (delimiter E'|' null E'\N' escape E'off' fill missing fields) ENCODING 'UTF8' ;
it shows
ERROR: protocol "gphdfs" does not exist
You need to run a SQL Script included in Greenplum Package. You will find the same script in $GPHOME/share/postgresql/cdb_external_extensions.sql
Run this:
psql -f $GPHOME/share/postgresql/cdb_external_extensions.sql
Hope this helps..

How to do BULK INSERT in Oracle Database

I am trying to do a bulk insert into tables from a CSV file using Oracle11. My problem is that the database is on a remote machine which I can sqlpl to using this:
sqlpl username#oracle.machineName
Unfortunately the sqlldr has trouble connecting using the following command:
sqlldr userid=userName/PW#machinename control=BULK_LOAD_CSV_DATA.ctl log=sqlldr.log
Error is:
Message 2100 not found; No message file for product=RDBMS, facility=ULMessage 2100 not found; No message file for product=RDBMS, facility=UL
Now having given up on this approach I tried writing a basic sql script, but I am unsure of the proper Oracle keyword for BULK. I know this works in MySql but I get:
unknown command beginning "BULK INSER..."
When running the script:
BULK INSERT <TABLE_NAME>
FROM 'CSVFILE.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
I don't care which one works! Either one will do, I just need a little help.
Sorry I am a dumb dumb! I forgot to add oracle/bin to my path!
If you have found this post, add the bin directory to your path (linux) using the following commands:
export ORACLE_HOME=/path/to/oracle/client
export PATH=$PATH:$ORACLE_HOME/bin
Sorry if I wasted anyone's time ....

Hive error when creating an external table (state=08S01,code=1)

I'm trying to create an external table in Hive, but keep getting the following error:
create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/hive_test_1375711405.45852.txt";
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
Aborting command set because "force" is false and command failed: "create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/hive_test_1375711405.45852.txt";"
The contents of /tmp/hive_test_1375711405.45852.txt are:
abc\tdef
I'm connecting via the beeline command line interface, which uses Thrift HiveServer2.
System:
Hadoop 2.0.0-cdh4.3.0
Hive 0.10.0-cdh4.3.0
Beeline 0.10.0-cdh4.3.0
Client OS - Red Hat Enterprise Linux Server release 6.4 (Santiago)
The issue was that I was pointing the external table at a file in HDFS instead of a directory. The cryptic Hive error message really threw me off.
The solution is to create a directory and put the data file in there. To fix this for the above example, you'd create a directory under /tmp/foobar and place hive_test_1375711405.45852.txt in it. Then create the table like so:
create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/foobar";
We faced similar problem in our company (Sentry, hive, and kerberos combination). We solved it by removing all privileges from non fully defined hdfs_url. For example, we changed GRANT ALL ON URI '/user/test' TO ROLE test; to GRANT ALL ON URI 'hdfs-ha-name:///user/test' TO ROLE test;.
You can find the privileges for a specific URI in the Hive database (mysql in our case).

Resources