I have an application which is occasionally being passed data which causes it to fail with ORA-12899 error trying to insert the record into a varchar(2000) column on an Oracle 11g database. )In some cases it is trying to insert a field of more than 4000 characters.)
In the longer term, I will request a change to the application to prevent this happening, but meanwhile I need a short-term fix to allow the record to be saved, truncating the data to 2000 characters.
I've experimented with a "before insert" trigger (which works fine when truncating it from say 1500 to 1000 characters), but what I've found is that where the data is more than 2000 characters (i.e. more than the column length) the insert still fails with ORA-12899.
Is there any way round this, other than changing the database column to a CLOB?
Include the case statement similar to the following code in your insert command
insert into test_table
(column1)
values
(
(case when length('ablekkkkkkkkkkkh') > 2000 then substr('ablekkkkkkkkh',1,2000)
else 'ablekkkkkkkkkkkh' end )
);
Related
I work on a project to transfer data from an Oracle database to a PostgreSQL database to create a datawarehouse with bash & SQL scripts. To access to the Oracle database, I use the PostgreSQL extension oracle-fdw.
One of my scripts import data from a massive table (~ 100 000 000 new rows/day). This table is partitioned and each partition contains 1 day of data. The query I use to import data looks like that :
INSERT INTO postgre_target_table (some_fields)
SELECT some_aggregated_fields -- (~150 fields)
FROM oracle_source_table
WHERE partition_id = :v_partition_id AND some_others_filters
GROUP BY primary_key;
On DEV server, the query works fine (there is much less data on this server) but in PREPROD, it returns the error ORA-01406: fetched column value was truncated.
In some posts, people say that the output fields may be too small but if I try to send a simple SELECT query without INSERT or GROUP BY I have the same error.
Another idea I found in another post is to create an Oracle side view but in my query I use multiple parameters that I cannot use in a view.
The last idea I found is to create an Oracle stored procedure that fills a table with aggregated data and then import data from this table but the Oracle database is critical and my customer prefers to avoid adding more data on it.
Now, I'm starting to think there's no solution and it's not good...
PostgreSQL version : 12.4 / Oracle version : 11.2
UPDATE
It seems my problem is more complecated than I thought.
After applying the modification given by Laurenz Albe, the query runs correctly on PGAdmin but the problem still appears when I use psql command.
Moreover, another query seems to have the same problem. This other query does not use the same source table as the first query, it uses 4 joined tables without any partition. The common point between these queries is the structure.
The detail I omit to specify in the original post is that the purpose of both queries is to pivot a table. They look like that :
SELECT osr.id,
MIN(CASE osr.category
WHEN 123 THEN
1
END) AS field1,
MIN(CASE osr.category
WHEN 264 THEN
1
END) AS field2,
MIN(CASE osr.category
WHEN 975 THEN
1
END) AS field3,
...
FROM oracle_source_table osr
WHERE osr.category IN (123, 264, 975, ...)
GROUP BY osr.id;
Now that I have detailed what the queries look like, I can give you some results I had with the second one without changing the value of max_long (this query is lighter than the first one) :
Sometimes it works (~10%), sometimes it failed (~90%) on PGadmin but it never works with psql command
If I delete the WHERE, it always works
I don't understand why deleting the WHERE change something, the field used in this clause is a NUMBER(6, 0) between 0 and 2500 and it is still used in the SELECT clause... Oh and in the 4 Oracle tables used by this query, there is no LONG datatype, only NUMBER datatype is used.
Among 20 queries I have, only these two have a problem, their structure is similar and I don't believe in coincidences.
Don't despair!
Set the max_long option on the foreign table big enough that all your oversized data fit.
The documentation has the details:
max_long (optional, defaults to "32767")
The maximal length of any LONG, LONG RAW and XMLTYPE columns in the Oracle table. Possible values are integers between 1 and 1073741823 (the maximal size of a bytea in PostgreSQL). This amount of memory will be allocated at least twice, so large values will consume a lot of memory.
If max_long is less than the length of the longest value retrieved, you will receive the error message
ORA-01406: fetched column value was truncated
Example:
ALTER FOREIGN TABLE my_tab OPTIONS (ADD max_long '1000000');
In an app that supports MS SQL Server, MySQL, and Oracle, there's a table with the following relevant columns (types shown here are for Oracle):
ShortText VARCHAR2(1700) indexed
LongText CLOB
The app stores values 850 characters or less in ShortText, and longer ones in LongText. I need to create a view that returns that data, whichever column it's in. This works for SQL Server and MySQL:
SELECT
CASE
WHEN ShortText IS NOT NULL THEN ShortText
ELSE LongText
END AS TheValue
FROM MyTable
However, on Oracle, it generates this error:
ORA-00932: inconsistent datatypes: expected CHAR got CLOB
...meaning that Oracle won't implicitly convert the two columns to the same type, so the query has to do it explicitly. Don't want data to get truncated, so the type used has to be able to hold as much data as a CLOB, which as I understand it (not an Oracle expert) means CLOB, only, no other choices are available.
This works on Oracle:
SELECT
CASE
WHEN ShortText IS NOT NULL THEN TO_CLOB(ShortText)
ELSE LongText
END AS TheValue
FROM MyTable
However, performance is amazingly awful. A query that returns LongText directly took 70-80 ms for about 9k rows, but the above construct took between 30 and 60 seconds, unacceptable.
So:
Are there any other Oracle types I could coerce both columns to
that can hold as much data as a CLOB? Ideally something more
text-oriented, like MySQL's LONGTEXT, or SQL Server's NTEXT (or even
better, NVARCHAR(MAX))?
Any other approaches I should be looking at?
Some specifics, in particular ones requested by #Guido Leenders:
Oracle version: Oracle Database 11g 11.2.0.1.0 64bit Production
Not certain if I was the only user, but the relative times are still striking.
Stats for the small table where I saw the performance I posted earlier:
rowcount: 9,237
varchar column total length: 148,516
clob column total length: 227,020
The to_clob is pretty expensive, so try to avoid it. But I think it should perform reasonable well for 9K rows. Following test case based upon one of the applications we develop which has the similar datamodel behaviour:
create table bubs_projecten_sample
( id number
, toelichting varchar2(1700)
, toelichting_l clob
)
begin
for i in 1..10000
loop
insert into bubs_projecten_sample
( id
, toelichting
, toelichting_l
)
values
( i
, case when mod(i, 2) = 0 then 'short' else null end
, case when mod(i, 2) = 0 then rpad('long', i, '*') else null end
)
;
end loop;
commit;
end;
Now make sure everything in cache and dirty blocks written out:
select *
from bubs_projecten_sample
Test performance:
create table bubs_projecten_flat
as
select id
, to_clob(toelichting) toelichting_any
from bubs_projecten_sample
where toelichting is not null
union all
select id
, toelichting_l
from bubs_projecten_sample
where toelichting_l is not null
The create table take less than 1 second on a normal entry level server, including writing out the data, 17K consistent gets, 4K physical reads. Stored on disk (note the rpad) is 25K for toelichting and 16M for toelichting_l.
Can you further elaborate on the problem?
Please check that large CLOBs are not stored inline. Normally large CLOBs are stored in a separate system-maintained table. Storing large CLOBs inside a table can make going through the table with a Full Table Scan expensive.
Also, I can imagine populating both columns always. You still have the benefits of indexing working for the first so many characters. You just need to memorize in the table using an indicator whether the CLOB or the shortText column is leading.
As a side note; I see a difference between 850 and 1700. I would recommend making them equal, but remember to check that you are creating the table using character semantics. That can be done on statement level by using: "varchar2(850 char)". Please note that Oracle will actually create a column that fits 850 * 4 bytes (in AL32UTF8 at least, there the "32" stands for "4 bytes at most per character"). Good luck!
I have the following problem: We have a DB2 Table with a Character Field of the size 4000.
Somehow this field gets interpreted in oracle as varchar2(0 Char) when i view it via Oracle Gateway.
CREATE OR REPLACE FORCE VIEW DB2SCHEMA.TEST_TABLE
(
ID,
TEXT
)
AS
SELECT TRIM ("ID") AS ID,
NULL AS "TEXT
FROM XT.TEST_TABLE#DB2SCHEMA;
Has anybody ever expirienced this issue? For some reason Oracle will treat this field as long. I'm using Oracle 11g. What i want is to show it as normal text field (in DB2 it is a fixed length character field)
Thanks for some inputs and maybe somebody knows how to get this as a normal Varchar2(4000 Char).
DESC XT.TEST_TABLE#DB2SCHEMA;
ORA-00604: error occurred at recursive SQL level 1
ORA-28500: connection from ORACLE to a non-Oracle system returned this message:
[Oracle][ODBC DB2 Wire Protocol driver][UDB DB2 for OS/390 and z/OS]UNAVAILABLE RESOURCE CAUSED FAILED EXEC; 00D70024 TYPE 00000220. XT.DSNDBC.DSNDB06.DSNDLX04.I0001.A001 {HY000,NativeErr = -904}
ORA-02063: preceding 2 lines from QDBC
As before the select * from view did provide a 0 character field we have changed an Oracle Parameter
HS_KEEP_REMOTE_COLUMN_SIZE=LOCAL
This provides the correct result with select * from view. My believe is that this is the solution.
I've never used Oracle Gateway or DB2, but any view you create by selecting "null as some_col" is going to define that column as varchar2(0). You're not even referencing the column you're talking about from the DB2SCHEMA (you're selecting NULL, not the TEXT column.
You could try replacing this line:
NULL AS "TEXT
with:
TEXT AS TEXT
or possibly:
SUBSTR(TEXT,1,2000) || SUBSTR(TEXT,2001,4000) AS TEXT
Oracle's maximum size for CHAR columns is 2000, so I'm not sure if you can just select the column directly.
Edit: As per comments, the OP's issue was fixed via changing the Oracle parameter:
HS_KEEP_REMOTE_COLUMN_SIZE=LOCAL
Get the above error when the execute immediate is called in a loop
Update CustomersPriceGroups set 1AO00=:disc Where cuno=:cuno
Parameters: disc=66 cuno=000974
Update CustomersPriceGroups set 1AP00=:disc Where cuno=:cuno
Parameters: disc=70.5 cuno=000974
Update CustomersPriceGroups set 1AQ00=:disc Where cuno=:cuno
Parameters: disc=66 cuno=000974
Update CustomersPriceGroups set 1ZA00=:disc Where cuno=:cuno
Parameters: disc=60 cuno=000974
What does this mean ?
Here is the code fragment
c:=PriceWorx.frcPriceListCustomers('020','221');
LOOP
fetch c into comno,cuno,nama,cpls;
exit when c%notfound;
dbms_output.put_Line(cuno);
g:=priceWorx.frcPriceListItemGroups('020','221');
d:=priceworx.frcCustomerDiscounts('020','221',cuno);
loop
fetch g into comno,cpgs,n;
fetch d into comno,cpls,cuno,cpgs,stdt,tdat,qanp,disc,src;
--dbms_output.put(chr(9)||cpgs);
sQ:='Update saap.CustomersPriceGroups set "'|| trim(cpgs)||'"=:disc '
|| ' Where cuno=:cuno';
execute immediate sQ using disc,cuno;
commit;
dbms_output.put_line( sQ );
dbms_output.put_line( chr(9)||'Parameters: disc='|| disc||' cuno='||cuno);
exit when g%notfound;
end loop;
close g;
close d;
end loop;
check your query for double comma.
insert into TABLE_NAME (COLUMN1, COLUMN2,,COLUMN3) values(1,2,3);
(there is extra comma after COLUMN2).
Update: recently (some people have special talents) i succeed to get same exception with new approach:
update TABLE_NAME set COLUMN1=7, set COLUMN2=8
(second SET is redundant)
Unquoted identifiers must begin with an alphabetic character (see rule 6 here). You're trying to assign a value to a column with a name starting with a number 1AO00, 1AP00 etc.
Without seeing the table definition for CustomersPriceGroups we don't know if it has columns with those names. If it does then they must have been created as quoted identifiers. If so you'll have to refer to them (everywhere) with quotes, which is not ideal - makes the code a bit harder to read, makes it easy to make a mistake like this, and can be hard to spot what's wrong. Even Oracle say, on the same page:
Note: Oracle does not recommend using quoted identifiers for database
object names. These quoted identifiers are accepted by SQL*Plus, but
they may not be valid when using other tools that manage database
objects.
In you code you appear to be using quotes when you assign sQ, but the output you show doesn't; but it doesn't have the saap. schema identifier either. That may be because you're not running the version of the code you think, but might just have been
lost if you retyped the data instead of pasting it - you're not showing the earlier output of c.cuno either. But it's also possible you have, say, the case of the column name wrong.
If the execute is throwing the error, you won't see the command being executed that time around the loop because the debug comes after it - you're seeing the successful values, not the one that's breaking. You need to check all the values being returned by the functions; I suspect that g is returning a value for cpgs that actually isn't a valid column name.
As #ninesided says, showing more information, particularly the full exception message, will help identify what's wrong.
It means that the Oracle parser thinks that one of your columns is not valid. This might be because you've incorrectly referenced a column, the column name is reserved word, or because you have a syntax error in the UPDATE statement that makes Oracle think that something which is not a column, is a column. It would really help to see the full statement that is being executed, the definition of the CustomersPriceGroups table and the full text of the exception being raised, as it will often tell which column is at fault.
if you add a extra "," at the end of the set statement instead of a syntax error, you will get ORA-01747, which is very very odd from Oracle
e.g
update table1
set col1 = 'Y', --this odd 1
where col2 = 123
and col3 = 456
In addition to reasons cited in other answers here, you may also need to check that none of your table column names have a name which is considered a special/reserved word in oracle database.
In my case I had a table column name uid. uid is a reserved word in oracle and therefore I was getting this error.
Luckly, my table was a new table and I had no data in it. I was a able to use oracle DROP table command to delete the table and create a new one with a modified name for the problem column.
I also had trouble with renaming the problem column as oracle wouldn't let me and kept throwing errors.
You used oracle keyword in your SQL statement
And I was writing query like. I had to remove [ and ]
UPDATE SN.TableName
SET [EXPIRY_DATE] = systimestamp + INTERVAL '12' HOUR,
WHERE [USER_ID] ='12345'
We recently moved from SQL Server to Oracle.
The cause may also be when you group by a different set of columns than in select for example:
select tab.a, tab.b, count(*)
from ...
where...
group by tab.a, tab.c;
ORA-01747: invalid user.table.column, table.column, or column
specification
You will get when you miss the column relation when you compare both column id your is will not be the same check both id in your database
Here is the sample Example which I was facing:
UPDATE TABLE_NAME SET APPROVED_BY='1000',CHECK_CONDITION=ID, WHERE CONSUMER_ID='200'
here Issue you will get when 'CHECK_CONDITION' and 'ID' both column id will no same
If both id will same this time your query will execute fine, Check id Id both Column you compare in your code.
For me, the issue was due to use to column name "CLUSTER" which is a reserved word in Oracle. I was trying to insert into the column. Renaming the column fixed my issue.
insert into table (JOB_NAME, VERSION, CLUSTER, REPO, CREATE_TS) VALUES ('abc', 169, 'abc.war', '1.3', 'test.com', 'test', '26-Aug-19 04.27.09.000000949 PM')
Error at Command Line : 1 Column : 83
Error report -
SQL Error: ORA-01747: invalid user.table.column, table.column, or column specification
In my case, I had some.* in count. like count(dr.*)
I am trying to migrate a DB from Informix to Oracle.Informix had an option like while inserting into a table if the size of the value exceeds the column length then Informix automatically trims the data.But Oracle does not support this and always throws an exception.Is there a way to prevent and allow trim or we have to respect religiously?
There is no automatic trimming of data in Oracle, you have to trim it explicitly yourself e.g.
insert into mytable (id, text) values (123, substr(var,1,4000));
Oracle does support a variety of SQL functions which trim variables. I suspect the one you'll want is 'SUBSTR()'. The problem is that you will need to specify the desired length explicitly. In this example T23.WHATEVER is presumed to be VARCHAR2(30) and T24.TOO_LONG_COLUMN is, er, longer:
insert into t23
(id
, whatever)
select pk_col
, substr(too_long_col, 1, 30)
from t42
/
As well as Tony's suggestion, you can use a CAST
select cast ('1234' as varchar2(3)) a
from dual
If you are doing data migration, look into DML Error Logging
Having all your non-conformant data put into a corresponding table with the failure reason is positively dreamy.