I am trying to move data from a oracle instance to postgres RDS using DMS. I am only doing a full load operation and I have disabled all the foreign keys on the target. I also made sure that the datatypes are not mismatched between columns for the same tables. I tried both 'Do Nothing' and 'Truncate' for the Target Table preparation mode and when I run the task, several tables are failing with below error messages:
[TARGET_LOAD ]E: Command failed to load data with exit error code 1, Command output: <truncated> [1020403] (csv_target.c:981)
[TARGET_LOAD ]E: Failed to wait for previous run [1020403] (csv_target.c:1578)
[TARGET_LOAD ]E: Failed to load data from csv file. [1020403] (odbc_endpoint_imp.c:5648)
[TARGET_LOAD ]E: Handling End of table 'public'.'SKEWED_VALUES' loading failed by subtask 6 thread 1 [1020403] (endpointshell.c:2416)
DMS doesn't give the correct error information and I am not able to understand what the above error messages mean.
When I use 'Drop tables on target' for the Target table preparation mode, it works but it creates the datatypes of the columns in a different way which I don't want.
Any help would be appreciated.
To troubleshoot my case, I created a copy of the task that only loaded the one problem table, and upped all the logging severities to "Detailed debug". Then I was able to see this:
[TARGET_LOAD ]D: RetCode: SQL_SUCCESS_WITH_INFO SqlState: 42622 NativeError: -1 Message: NOTICE: identifier "diagnosticinterpretationrequestdata_diagnosticinterpretationcode" will be truncated to "diagnosticinterpretationrequestdata_diagnosticinterpretationcod" (ar_odbc_stmt.c:4720)
In the RDS logs for the target DB I found:
2021-10-11 14:30:36 UTC:...:[19259]:ERROR: invalid input syntax for integer: ""
2021-10-11 14:30:36 UTC:...:[19259]:CONTEXT: COPY diagnosticinterpretationrequest, line 1, column diagnosticinterpretationrequestdata_diagnosticinterpretationcod: ""
2021-10-11 14:30:36 UTC:...:[19259]:STATEMENT: COPY "myschema"."diagnosticinterpretationrequest" FROM STDIN WITH DELIMITER ',' CSV NULL 'attNULL' ESCAPE '\'
I found that if I added a table mapping rule to explicitly rename the column to truncate the name within Postgres's limit for identifier length, then things ran ok.
{
"rule-type": "transformation",
"rule-id": "1",
"rule-name": "1",
"rule-target": "column",
"object-locator": {
"schema-name": "%",
"table-name": "%",
"column-name": "diagnosticinterpretationrequestdata_diagnosticinterpretationcode"
},
"rule-action": "rename",
"value": "diagnosticinterpretationrequestdata_diagnosticinterpretationcod",
"old-value": null
},
Related
I have docker container with clickhouse database. And I have backups from our production. But I tried many cases with restore. When I using command (in container): clickhouse-client --format Raw --file 21_02_04_db.backup, I get error Bad arguments: unrecognised option '--file'. Okay, I tried without this option: clickhouse-client --format Raw < 21_02_04_db.backup and I get an error:
Code: 62. DB::Exception: Syntax error: failed at position 1 ('') (line 1, col 1): �TimDateTime�#�]�{�]�k�]u �]0&�] 9�]�-�]��]�I�]U��]���]yy�]���]�\�]�\�]��]{B�]�j�]v)�]�d�]J��]�F�]�]�]��]t%�]t%�]���]��]F?�]�B�]��]W3�]��]E��]P��]a�. Unrecognized token: '' I use other backups but in many cases I get similar error. Maybe someone tried restore backup clickhouse with help binary data? I try many cases but I didn't search solution.
it looks like a Native format
it should something like
clickhouse-client -q 'insert into mytable format Native' < 21_02_04_db.backup
When importing a file into Greenplum,one lines fails,and the whole file is not imported successfully.Is there a way can skip the wrong line and import other data into Greenplum successfully?
Here are my SQL execution and error messages:
copy cjh_test from '/gp_wkspace/outputs/base_tables/error_data_test.csv' using delimiters ',';
ERROR: invalid input syntax for integer: "FE00F760B39BD3756BCFF30000000600"
CONTEXT: COPY cjh_test, line 81, column local_city: "FE00F760B39BD3756BCFF30000000600"
Greenplum has an extension to the COPY command that lets you log errors and set up a certain amount of errors that can occur that won't stop the load. Here is an example from the documentation for the COPY command:
COPY sales FROM '/home/usr1/sql/sales_data' LOG ERRORS
SEGMENT REJECT LIMIT 10 ROWS;
That tells COPY that 10 bad rows can be ignored without stopping the load. The reject limit can be # of rows or a percentage of the load file. You can check the full syntax in psql with: \h copy
If you are loading a very large file into Greenplum, I would suggest looking at gpload or gpfdist (which also support the segment reject limit syntax). COPY is single threaded through the master server where gpload/gpfdist load the data in parallel to all segments. COPY will be faster for smaller load files and the others will be faster for millions of rows in a load file(s).
IF EXISTS(select 1 from sys.views where name='release_testcase_count')
DROP VIEW ITCC.release_testcase_count4;
i am able to delete this view but it is saying -
Error starting at line : 1 in command -
IF EXISTS(select 1 from sys.views where name='release_testcase_count')
Error report -
Unknown Command
View ITCC.RELEASE_TESTCASE_COUNT4 dropped.
so in this case 1 line have a error, it is not checking it exist or not.
IF EXISTS command is valid in SQL Server or other databases but not in ORACLE.
Your first line is completely ignored and signaled as unknown command as Oracle does not recognize any command starting with IF EXIST.
Second line is a valid command, so the view is dropped as a result.
After executing a pg_dump from PostgeSQL, I attempted to import the .sql file into CockroachDB, but received the following errors:
ERROR: unknown variable: "STATEMENT_TIMEOUT"
ERROR: unknown variable: "LOCK_TIMEOUT"
ERROR: unknown variable: "IDLE_IN_TRANSACTION_SESSION_TIMEOUT"
SET
SET
ERROR: unknown variable: "CHECK_FUNCTION_BODIES"
SET
ERROR: unknown variable: "ROW_SECURITY"
SET
ERROR: unknown variable: "DEFAULT_TABLESPACE"
ERROR: unknown variable: "DEFAULT_WITH_OIDS"
CREATE TABLE
ERROR: syntax error at or near "OWNER"
Any guidance?
CockroachDB has special support for using psql, which supports the COPY command (which is faster than batched INSERT statements).
You'll need to do two things:
Clean up the SQL file
Import it into CockroachDB (which sounds like you tried, but I'll include the steps here for anyone else who needs them):
Clean up the SQL File
After generating the .sql file, you need to perform a few editing steps before importing it:
Remove all statements from the file besides the CREATE TABLE and COPY statements.
Manually add the table's PRIMARY KEY constraint to the CREATE TABLE statement.
This has to be done manually because PostgreSQL attempts to add the primary key after creating the table, but CockroachDB requires the primary key be defined upon table creation.
Review any other constraints to ensure they're properly listed on the table.
Remove any unsupported elements, such as arrays.
Import Data
After reformatting the file, you can import it through psql:
$ psql -p [port] -h [node host] -d [database] -U [user] < [file name].sql
For reference, CockroachDB uses these defaults:
[port]: 26257
[user]: root
I'm trying to create an external table in Hive, but keep getting the following error:
create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/hive_test_1375711405.45852.txt";
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (state=08S01,code=1)
Aborting command set because "force" is false and command failed: "create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/hive_test_1375711405.45852.txt";"
The contents of /tmp/hive_test_1375711405.45852.txt are:
abc\tdef
I'm connecting via the beeline command line interface, which uses Thrift HiveServer2.
System:
Hadoop 2.0.0-cdh4.3.0
Hive 0.10.0-cdh4.3.0
Beeline 0.10.0-cdh4.3.0
Client OS - Red Hat Enterprise Linux Server release 6.4 (Santiago)
The issue was that I was pointing the external table at a file in HDFS instead of a directory. The cryptic Hive error message really threw me off.
The solution is to create a directory and put the data file in there. To fix this for the above example, you'd create a directory under /tmp/foobar and place hive_test_1375711405.45852.txt in it. Then create the table like so:
create external table foobar (a STRING, b STRING) row format delimited fields terminated by "\t" stored as textfile location "/tmp/foobar";
We faced similar problem in our company (Sentry, hive, and kerberos combination). We solved it by removing all privileges from non fully defined hdfs_url. For example, we changed GRANT ALL ON URI '/user/test' TO ROLE test; to GRANT ALL ON URI 'hdfs-ha-name:///user/test' TO ROLE test;.
You can find the privileges for a specific URI in the Hive database (mysql in our case).