I'm migrating some data from one oracle schema/table to a new schema/table on the same database.
The migration script does the following:
create table newtable as select
...
cast(ACTIVITYDATE as date) as ACTIVITY_DATE,
...
FROM oldtable where ACTIVITYDATE > sysdate - 1000;
If I look at the original data, it looks fine - here's one record:
select
activitydate,
to_char(activitydate, 'MON DD,YYYY'),
to_char(activitydate, 'DD-MON-YYYY HH24:MI:SS'),
dump(activitydate),
length(activitydate)
from orginaltable where oldpk = 1067514
Result:
18-NOV-10 NOV 18,2010 18-NOV-2010 12:59:15 Typ=12 Len=7: 120,110,11,18,13,60,16
The migrated data, showing that the data is corrupt:
select
activity_date,
to_char(activity_date, 'MON DD,YYYY'),
to_char(activity_date, 'DD-MON-YYYY HH24:MI:SS'),
dump(activity_date),
length(activity_date)
from newtable
where id = 1067514
Result:
18-NOV-10 000 00,0000 00-000-0000 00:00:00 Typ=12 Len=7: 120,110,11,18,13,0,16
Around 5000 out of 350k records show this problem.
Can anyone explain how this happened?
UPDATE:
I don't find any published reference to this specific type of DATE corruption on the Oracle support site. (It may be there, my quick searches just didn't turn it up.)
Baddate Script To Check Database For Corrupt dates [ID 95402.1]
Bug 2790435 - Serial INSERT with parallel SELECT and type conversion can insert corrupt data [ID 2790435.8]
The output from the DUMP() function is showing the date value is indeed invalid:
Typ=12 Len=7: 120,110,11,18,13,0,16
We expect that the minutes byte should be a value between one and sixty, not zero.
The 7 bytes of a DATE value represent, in order, century(+100), year(+100), month, day, hour(+1), minutes(+1), seconds(+1).
The only time I have seen invalid DATE values like this when a DATE value was being supplied as a bind variable, from a Pro*C program (where the bind value is supplied in the internal 7 byte representation, entirely bypassing the normal validation routines that catch invalid dates e.g. Feb 30)
There is no reason to expect the behavior you're seeing, given the Oracle syntax you posted.
This is either a spurious anomaly (memory corruption?) or if this is repeatable, then it's a flaw (bug) in the Oracle code. If it's a flaw in the Oracle code, the most likely suspects would be "newish" features in an un-patched release.
(I know CAST is a standard SQL function that's been around for ages in other databases. I guess I'm old school, and have never introduced it into my Oracle-syntax repertoire. I don't know what version of Oracle it was that introduced the CAST, but I would have stayed away from it in the first release it appeared in.)
The big 'red flag' (that another commenter noted) is that CAST( datecol AS DATE).
You would expect the optimizer to treat that as equivalent to date_col ... but past experience shows us that TO_NUMBER( number_col ) is actually interpreted by the optimizer as TO_NUMBER( TO_CHAR ( number_col ) ).
I suspect something similar might be going on with that unneeded CAST.
Based on that one record you showed, I suspect the issue is with values with a "59" value for minutes or seconds, and possibly a "23" value for hours, would be the ones that show the error.
I would try checking for places where the minutes, hour or seconds are stored as 0:
SELECT id, DUMP(activitydate)
FROM newtable
WHERE DUMP(activitydate) LIKE '%,0,%'
OR DUMP(activitydate) LIKE '%,0'
I've seen similar things to spence7593, again with Pro*C.
It is possible to create invalid dates programmatically using a DBMS_STATS package.
Not sure if there is a similar mechanism to reverse that.
create or replace function stats_raw_to_date (p_in raw) return date is
v_date date;
v_char varchar2(25);
begin
dbms_stats.CONVERT_RAW_VALUE(p_in, v_date);
return v_date;
exception
when others then return null;
end;
/
select stats_raw_to_date(utl_raw.cast_to_raw(
chr(120)||chr(110)||chr(11)||chr(18)||chr(13)||chr(0)||chr(16)))
from dual;
Related
I'm trying to run this queries (Oracle 12c):
SELECT trunc(sysdate) - '25-SEP-18' FROM dual;
SELECT 1 FROM dual WHERE trunc(sysdate) = '04-SEP-19';
CREATE TABLE my_table (order_date date);
INSERT INTO my_table (order_date) VALUES ('04-SEP-19');
I expect implicit conversion and everything is good with the 2 last queries, but for the first i get error ORA-01722: invalid number. NLS_DATE_FORMAT = 'DD-MON-RR'. What is the problem?
The question is WHY is does not work? I didn't find any explanations in documentation.
The documentation has a section on Datetime/Interval Arithmetic which explains what is allowed. The table shows that arithmetic is only allowed between dates, timestamp, intervals and numbers. When you do:
SELECT trunc(sysdate) - '25-SEP-18'
you are trying to subtract a string from a date, which isn't possible. Oracle 'helpfully' tries anyway and interprets the string as a number, effectively doing:
SELECT trunc(sysdate) - to_number('25-SEP-18')
which understandably throws the error you see, "ORA-01722: invalid number". As already said, you should explicitly convert your string to a date:
SELECT trunc(sysdate) - to_number('25-SEP-18', 'DD-MON-RR')
or preferably with a four-digit year, and since you're using a month name it's safer to specify the language that is in:
SELECT trunc(sysdate) - to_number('25-SEP-2018', 'DD-MON-YYYY', 'NLS_DATE_LANGUAGE=ENGLISH')
or more simply, if it's a fixed value, with a date literal:
SELECT trunc(sysdate) - DATE '2018-09-25'
I expect implicit conversion
You should not rely on implicit conversion, particularly where that is influenced by session NLS settins. As well as the date language I already mentioned, someone else running your statement could have a different NLS_DATE_FORMAT setting which could lead to errors or more subtle data mismatches or corruption; e.g.
alter session set nls_date_format = 'DD-MON-YYYY';
SELECT trunc(sysdate) - DATE '2018-09-25' FROM dual;
TRUNC(SYSDATE)-DATE'2018-09-25'
-------------------------------
344
SELECT trunc(sysdate) - to_date('25-SEP-18') FROM dual;
TRUNC(SYSDATE)-TO_DATE('25-SEP-18')
-----------------------------------
730831
SELECT 1 FROM dual WHERE trunc(sysdate) = '04-SEP-19';
no rows selected
CREATE TABLE my_table (order_date date);
INSERT INTO my_table (order_date) VALUES ('04-SEP-19');
The second query gets a much bigger value than expected; and the third gets no rows back from dual.
Looking at the implicitly converted date shows you why:
SELECT to_char(order_date, 'SYYYY-MM-DD HH24:MI:SS') FROM my_table;
TO_CHAR(ORDER_DATE,'
--------------------
0019-09-04 00:00:00
With a YYYY mask (and no FX modifier) a 2-digit year value like 19 is converted as 0019, not 2019. That sort of problem could go unnoticed for some time, giving you incorrect results in the meantime.
If the session's format mask had RRRR or - as you have - RR then it would be interpreted as 2019; but the point is that you usually have no control over the settings in another session that runs your code later.
You can also cause performance issues or errors by creating implicit conversions where you didn't expect, or where they behave in a way you didn't expect. Not in this example - "When comparing a character value with a DATE value, Oracle converts the character data to DATE" - but it still comes up. It's better to avoid the possibility.
When dealing with strings with dates in them you should use the to TO_DATE command, otherwise Oracle may not always figure out that the string contains a date.
SELECT trunc(sysdate) - TO_DATE('25-SEP-18') FROM dual;
Even better is to indicate the format of the date within the string
SELECT trunc(sysdate) - TO_DATE('25-SEP-18','DD-MON-RR') FROM dual;
I have a varchar column called begin_time that stores the 24-hour time as a varchar with no time formatting, ie 1330
I need to convert this varchar to a usable timestamp, datetime, etc. where I can easily convert the varchar to a standard time format (1:30 PM)
The end format type doesn't matter as long as I can convert the varchar into a format that I can manipulate to a standard format.
I've tried looking into Cognos-specific format tricks (These functions are for Metric Designer, and I'm using Report Studio) to no avail. The methods I found when looking for oracle-specific tricks seemed to be way too convoluted (using insanely long regex rules) for what I need.
If I need to have a date involved, I can use the column start_date and append the varchar time.
Note: start_date is in the date format
Example
select
to_date('08/27/2018','MM/DD/YYYY') as start_date
, '1300' as begin_time
from dual
What I ultimately need is just to be able to output the time as 1:00 PM
Any help would be appreciated. I'm beating my head against the wall on this... I'm used to using proprietary codes for periods of time and don't have a lot of experience with the true datetime formats.
Updates answering questions
Alex Poole, I make no claims that this system is the best... It's vendor-provided. :)
The BEGIN_TIME is always 4 characters
It does look like I was overthinking it quite a bit... Littlefoot may have nailed it on the head, but I unfortunately won't have a chance to test that until tomorrow.
Thank you all for the fast responses. I might have hair left when this request is over now :)
Final Thought
My lesson learned from this is simple: If you're dealing with time formats, don't throw out the idea of using a Date format function.
Looking for this?
SQL> with test (col) as
2 (select '1330' from dual)
3 select to_char(to_date(col, 'hh24mi'), 'hh:mi am') result
4 from test;
RESULT
--------
01:30 PM
SQL>
What does it do?
TO_DATE converts string you have (such as 1330) into a valid DATE value. By default, it'll be a date value truncated to the first of current month:
SQL> alter session set nls_Date_format = 'dd.mm.yyyy hh24:Mi';
Session altered.
SQL> select to_date('1330', 'hh24mi') res1 from dual;
RES1
----------------
01.04.2019 13:30
SQL>
applying TO_CHAR to it, again with the appropriate format mask, returns the desired result
sorry for asking a question, that has a lot of answers on Stackoverflow and allow me to pose this question in my context, that may differ from the previous questions.
I'm on a Production database, where I CANNOT CHANGE data. The data going into this database is highly dynamic, it changes all the time, so that makes it hard to reproduce that error. I'm accessing Oracle 11g via JDBC (Java).
Ok, for my DELETE I get the
ORA-01841: (full) year must be between -4713 and +9999, and not be 0.
This is my table (simplified):
MY_TABLE
Name Null? Type
---------------------------- -------- ---------------------------
MYTIMESTAMP NOT NULL TIMESTAMP(6) WITH TIME ZONE
From time to time I get the ORA-01841 for this DELETE:
delete from MY_TABLE where MYTIMESTAMP < sysdate - 30
When I look up the data, all seems fine. So where I need an idea:
1) How can I insert an invalid timestamp into MY_TABLE, so that I can reproduce that error? (*)
2) How can I rewrite the DELETE statement, so that I won't fail? Please note, I cannot change the existing data on Oracle.
Thank you
(*) I tried to insert these "invalid" date, but alas, not invalid enough:
insert into MY_TABLE (MY_IMESTAMP) values ('31-DEC-9999 11:00:00 PM +2:00')
insert into MY_TABLE (MY_IMESTAMP) values ('31-DEC-0 11:00:00 PM +2:00')
I see that error a lot when there are implicit type conversions going on. Can you maybe try:
delete from MY_TABLE where MYTIMESTAMP < systimestamp - interval '30' day
My source is from Oracle and the col1 is varchar2(26) but the value looks like YYYY-MM-DD-hh:mi:ss:ff (Sample rec: 2014-08-15-02.03.34.979946).
I have to extract only 6 months records based on COL1. Since there is a hypen between date part and time part - i could not consider as timestamp. Is there any idea how to have this as timestamp to lookup only 6 months data.
If it is possible at all, fix the data first. Storing timestamps in string data type is terrible. How do you know you don't have a time like 25:30:00 in the strings? Or a date like February 30? Besides, you can't really use an index on that column (so queries will be very slow), you will have to write a lot of code whenever referencing that column, etc.
Anyway - to deal with the immediate problem, use TO_TIMESTAMP(), exactly with the format model you show in your post - including the dash between the date part and the time part. Something like this:
select case when to_timestamp('2014-08-15-02.03.34.979946', 'YYYY-MM-DD-HH24:MI:SS.FF')
>= systimestamp - interval '6' month
then 'TRUE' else 'FALSE' end
as result
from dual;
RESULT
------
FALSE
EDIT: As Alex Poole points out (correctly as always) in a Comment below this Answer, interval arithmetic won't work correctly in all cases. It is better, than, to use something like
cast ( timestamp (...., format-model) as date ) <= add_months (sysdate, -6).
Maybe something like this will do:
select *
from your_table
where to_date(substr(col1,1,19),'yyyy-mm-dd-HH24.MI.SS') between add_months(sysdate,-6) and sysdate;
Assuming all the data format in col1 is always the same.
Also note that I used HH24 for hour segment, however could be not your case.
You can include the dash in your format model, as #mathguy showed, to convert your string to a timestamp:
select to_timestamp('2014-08-15-02.03.34.979946', 'YYYY-MM-DD-HH24:MI:SS.FF') from dual;
TO_TIMESTAMP('2014-08-15-02.
----------------------------
15-AUG-14 02.03.34.979946000
although unless you explicitly tell it not to be via the FX modifier, Oracle is flexible enough to allow a dash even if the model has a space (see the text below this table in the documentation:
select to_timestamp('2014-08-15-02.03.34.979946', 'YYYY-MM-DD HH24:MI:SS.FF') from dual;
TO_TIMESTAMP('2014-08-15-02.
----------------------------
15-AUG-14 02.03.34.979946000
However, converting all of the values in your col1 column and then comparing them may be a lot of work, and will prevent any index on that string column being used. Given the format, you can convert your date range to string instead, and use string comparison; e.g. to find everything in the six months up to midnight this morning:
select col1 -- or whichever columns you need
from your_table
where col1 >= to_char(cast(add_months(trunc(sysdate), -6) as timestamp), 'YYYY-MM-DD-HH24:MI:SS.FF6')
and col1 < to_char(cast(trunc(sysdate) as timestamp), 'YYYY-MM-DD-HH24:MI:SS.FF6');
or since the time part can be fixed for that example, you can use character literals instead of casting:
select col1 -- or whichever columns you need
from your_table
where col1 >= to_char(add_months(sysdate, -6), 'YYYY-MM-DD"-00:00:00.000000"')
and col1 < to_char(sysdate, 'YYYY-MM-DD"-00:00:00.000000"');
Of course, storing data in the correct native data type would be a much better solution. Any other solution only works at all if your string data actually contains what you think, and the data is all sane (or as sane as it can be in the wrong data type).
I have an SSIS package set up to pull data from an Oracle database into SQL Server. I recently ran into an issue that was preventing this data pull.
The following code works fine in Oracle SQL Developer (it returns rows, as it should):
SELECT a.MyField ,
a.MyOtherField,
a.FromDate
FROM MyTable a
WHERE a.FromDate BETWEEN CONCAT('01-', TO_CHAR(ADD_MONTHS(SYSDATE, -13), 'MON-YY')) AND TO_CHAR(LAST_DAY(SYSDATE), 'DD-MON-YY')
However, when using this as the SQL command text of an OLE DB Source component in SSIS, it returns no records.
I'm not sure if this is an SSIS issue or a difference in language syntax (I believe this is due to the different systems' date syntax, but I do not have a strong enough grasp of PL/SQL to know how to correct this.).
Any ideas?
Most likely, the problem is that you are comparing a date to a string. This forces Oracle to do an implicit cast using the session's NLS_DATE_FORMAT. Since this is session-specific, it is fragile as different clients will end up doing the conversion differently.
You are almost certainly better off rewriting the query in order to compare a date to other dates, i.e.
SELECT a.MyField ,
a.MyOtherField,
a.FromDate
FROM MyTable a
WHERE a.FromDate BETWEEN trunc( add_months( sysdate, -13 ), 'MM' ) AND
trunc( last_day( sysdate ) )