We've had some success with removing lob fields and avoiding row-by-row processing but with hadoop we can't seem to get around this. In some cases, the fields in question are less than 10 characters yet ssis sees them as lobs. is this an issue with hadoop, the odbc driver or ssis? what steps can we take to make a determination? Help me Obi Wan Kenobi. You're our last hope.
Hi if you can identify the columns that you need to convert add a data conversion step to those columns and then point those to the destination columns
and then set Validate External Data to False.
if this step also not working go into the advanced properties of the ODBC source and change the types back to something like DT_WSTR.
The answer you will get!!!
Related
I am currently working on a stupid system where I have given no direct DB access but a weird SQL Workbench which can not do most of the things apart from some basic stuff. So for some reason I need to do a SELECT * on one of the tables which have 174 columns. And whenever I try that it gives me the following error:
"ERROR: Error -27 was encountered whilst running the SQL command. (-3)
Error -3 running SQL : ORACLE Driver Error [-27]: Selected data too
large for SQL Workbench"
Quick googling gave me nothing apart from (in one of the oracle documents):
In the SQL Editor, the maximum length of one row of the formatted
result is 8190 bytes. When this length is exceeded, the ORA connector
generates the above error
Now, I was wondering if anyone could give me a solution that would be a great help. One of the solution I am thinking is to increase the Maximum Length for Ora Connector/Driver. But I am novice in Oracle and do not know anything apart from querying. So haven't been able to change the Maximum Length yet.
So, please if anybody could help me out with this, that would be great.
Thanks a lot guys
Being asked to do database work trough the Uniface SQL Workbench is not a good situation. It is only a very simple thing that you can use in an emergency if nothing else is available.
You could run a couple of queries, each time with the primary key and a bunch of fields and stitch the result together in Excel.
If you have access to the Uniface Development Environment you can use it to convert your Oracle data to, for example, XML. Instructions are in the Uniface helpfile ulibrary.chm, see command line switch /cpy.
You cannot change the maximum record length of the Uniface Oracle Connector.
Ok, tricky question I am trying to figure out where a database schema is storing a particular pointer. I know the pointer value I just don't what table it is in or what column. I know the pointer is 123123123. How do I check all table columns to see if any of them have that value?
Thanks.
In h2 you can use fulltext search, but then you would need to add all tables in the search scope and indexing.
If you need to index only primary keys, then it might be better but you still need to come up with individual FT_CREATE_INDEX() calls for each table. You can automate this with several languages or with ETLs (like scriptella).
If you've enough disk space, you could dump a SQL from your db and use a viewer for big files like glogg.
The advantage of the first solution is no external tools but you need to work out a specific indexing script for SQL for any existing or new table. The 2nd solution is a 1 time fix.
I use SQL Search from RedGate. It's free and it helps you find any text anywhere in the database.
https://www.red-gate.com/products/?gclid=CjwKEAjwiYG9BRCkgK-G45S323oSJABnykKAE7IH_EMhnmq7OdLdXljfIkdGZrDD6OnOrT4VB0agahoCVn3w_wcB
I am migrating data from sybase to oracle using talend. I am using tSybaseInput for input and tOracleOutput for output db. I am mapping them through t_Map in some whereas direct in others.
After running the job, the row order is not maintained i.e. the order in which the data comes from sybase is not same as reflected in oracle. I need the order to be same so that I can validate the data later by outputting the data of both db to csv's and then comparing them(right now I am sorting them(unix sort) ..but it seems wrong).
Please suggest a way to maintain row order of input db in output db.
Also , is my method of validation correct or should I try something else?
The character sets and sort orders between the two vendors may be slightly different, which probably why you are seeing a change in the order. You may want to add a numeric key value to your tables in the Sybase DB, which can then be used to force a particular order once the data is imported into Oracle.
As for validation, if you are already using Unix command line, once you have a key value, you should just be able to use diff to compare the two CSV files without having to involve Excel. Alternatively you can add both Sybase and Oracle as data sources for Excel, and query the data directly into your worksheets, instead of generating CSV.
I'm importing data from a text file using Bulk Insert in the script component in SSIS package.
Package Ran successfully and data imported into SQL
Now how do I validate the completeness of the data?
1. I can get the row count from source and destination and compare.
but my manager wants to know how we can verify all the data has come a cross as it is without any issues.
If we are comparing 2 tables then probably a joining them on all fields and see anything missing out.
I’m not sure how to compare a text file and a sql table.
One way I could is write code to ready the file line by line and query the database for that record and compare each and every field. We have millions of records so this is not going to be a simple task.
Any other way to validate all of the data ??
Any suggestions would be much appreciated
Thanks
Ned
Well you could take the same file and do a look-up to the SQL source and if any of the columns don't match move to a row count.
Here's a generic example of how you can do this.
We have a portal written in php/mysql and an enterprise application based on Java EE and Oracle. Recently we found out that a certain Unicode character (0643 to be precise) is invalid (due to improper data entry by end users) in text columns and must be changed to another character (06A9).
In MySQL I simply changed the export file using a text editor's find and replace tool. But in oracle, the dmp file is a binary file and i have no idea about how to edit the dmp file.
How can I change the invalid character?
Is there an alternative to iterating through all text columns in all tables?
(I have saved that as a last resort!)
Editing an Oracle dump file may be possible but isn't practical; even if you could get in and change something you'd risk corrupting it, and I doubt Oracle support would be impressed. (See this AskTom question for example).
If you're using data pump and you know which column(s) the data is in you might be able to use the REMAP_DATA parameter to change it on the fly, or the QUERY parameter to skip the data, but it doesn't sound like you're in that situation. You could potentially add temporary constraints to the relevant column(s) to block the value, so import would reject (and log) the affected rows, but that's painful and messy.
If you do have to check all columns on all tables, this link may be helpful.