A client site has supplied the following extract file for us to load into our database.
The problem is, for certain rows (the second row for example) the CREATED_DATE and the LAST_UPDATE_DATE are in "dd Mmm YYYY..." date format when the rest the rows (such as the top) are in the format of "YYYY-MM-DD HH24.MI.SSXFF"
PRIMARY_ID ID VALUE CREATED_DATE LAST_UPDATE_DATE
20166267 20834830491 2012-04-30 08:18:00 2012-04-30 08:18:00
20166536 9112 01 Oct 2010 17:27:04 01 Oct 2010 17:27:04
My questions are:
Q1. To avoid having to request an extract, can we manipulate these “dd Mmm YYYY...” formatted dates at import time in SQL Loader using the .ctl script? CUrrently my .ctl is
My .ctl file is scripted to import using:
IDENTIFIER_START_DATE TIMESTAMP "YYYY-MM-DD HH24.MI.SSXFF",
LAST_UPDATE_DATE TIMESTAMP "YYYY-MM-DD HH24.MI.SSXFF"
Q2. Is simply asking them for a re-extract with all date formats as requested the best practice in situations like this?
Whether to request a re-extract of the data depends on a number of factors.
Is this a one-time process or an ongoing data feed? It may be perfectly reasonable to try to do your best with a one-time load of data where it is easier to eyeball the outliers. If you are going to manage an ongoing data feed, it generally makes much more sense to agree on a strict standard for the file rather than trying to manually inspect problematic rows.
Does the client have an incentive to make your load process simple and repeatable? Or was the client sold on a fixed price to load the data in whatever format they want to provide it? If the client has an incentive to make the load process simple and repeatable, it makes sense for them to invest the time to generate a clean file. If you've sold them a fixed price for whatever work needs to be done to turn the file into coherent data, on the other hand, they probably won't be pleased if you push a lot of that work back on them.
Are there rows where the data is ambiguous? For example "01-02-03" could refer to Jan 2, 2003 or Jan 2, 1903 or Feb 3, 2001 or a number of other dates. If there is ambiguity, it makes sense to request a re-extract.
As to how to load the data, while it is possible to do this in a single step, you generally wouldn't want to. It would generally make more sense to load the data into a staging table (or use an external table) where all the columns are declared as VARCHAR2 and then write some ETL logic that transforms the data into the appropriate data types (and logs errors for the data that cannot be converted). For example, if you loaded the data into a staging table where all the columns were defined as VARCHAR2, you could use something like the my_to_date function in this thread to try a number of different format masks to find one that works (if there are a lot of possible masks, you may want to iterate through a collection rather than hard-coding the two masks as I did in that example).
One additional point... An Oracle DATE stores the time to the second which appears to be the precision of the data you're being given. It would, therefore, seem to make more sense to load the data into a DATE column rather than a TIMESTAMP column.
Use this .ctl script:
load data
append
into table schema_name.table_name
fields terminated by ';' optionally enclosed by '"'
(
PRIMARY_ID,
ID_VALUE,
CREATED_DATE "to_date(:CREATED_DATE, case when regexp_substr(:CREATED_DATE,'\w+',1,2)=regexp_substr(:CREATED_DATE,'\d+',1,2) then 'YYYY-MM-DD HH24:MI:SS' else 'dd Mon YYYY HH24:MI:SS' end)",
LAST_UPDATE_DATE "to_date(:LAST_UPDATE_DATE, case when regexp_substr(:LAST_UPDATE_DATE,'\w+',1,2)=regexp_substr(:LAST_UPDATE_DATE,'\d+',1,2) then 'YYYY-MM-DD HH24:MI:SS' else 'dd Mon YYYY HH24:MI:SS' end)"
)
Related
I'm trying to store date type data from Oracle FORMS with format mask as like DD-MM-YYYY but every time it store as like DD/MON/YY.
I already alter session with NLS_DATE_FORMAT, but result is as same as before.
Oracle internal date format that is written in the table is something you can't change in any way, but, in the same time, it is irrelevant. If you are dealing with DATE type column then you should know that it containes both the date and the time. How, where and when you will show it or use it is on you. Here is a sample of a few formats derived from that original Oracle DATE format...
WITH
t AS
(
Select SYSDATE "MY_DATE_COLUMN" From Dual
)
Select
MY_DATE_COLUMN "DATE_DEFAULT_FORMAT",
To_Char(MY_DATE_COLUMN, 'mm-dd-yyyy') "DATE_1",
To_Char(MY_DATE_COLUMN, 'yyyy/mm/dd') "DATE_2",
To_Char(MY_DATE_COLUMN, 'dd.mm.yyyy') "DATE_3",
To_Char(MY_DATE_COLUMN, 'dd.mm.yyyy hh24:mi:ss') "DATE_4"
From t
DATE_DEFAULT_FORMAT
DATE_1
DATE_2
DATE_3
DATE_4
22-OCT-22
10-22-2022
2022/10/22
22.10.2022
22.10.2022 10:59:44
You can find a lot more about the theme at https://www.oracletutorial.com/oracle-basics/oracle-date/
Regards...
In Oracle, a DATE is a binary data-type consisting of 7-bytes (representing century, year-of-century, month, day, hour, minute and second). It ALWAYS has those 7 components and it is NEVER stored in any particular human-readable format.
every time it store as like DD/MON/YY.
As already mentioned, no, it does not store a date like that; the database stores dates as 7 bytes.
What you are seeing is that the client application, that you are using to connect to the database, is receiving the 7-byte binary date value and is choosing to convert it to something that is more easily comprehensible to you, the user, and is defaulting to converting the date to a string with the format DD/MON/RR.
What you should be doing is changing how the dates are displayed by the client application by either:
Change the settings in the Toad (View > Toad Options > Data Grids > Data and set the Date Format option) and allow Toad to implicitly format the string; or
Use TO_CHAR to explicitly format the date (TO_CHAR(column_name, 'DD-MM-YYYY')).
I'm trying to store data as like DD-MM-YYYY.
If you want to store a date then STORE it as a date (which has no format) and format it when you DISPLAY it.
If you have a valid business case to store it with a format then you will need to store it as a string, rather than as a date, because you can format strings; however, this is generally considered bad practice and should be avoided.
Sadman, to add to what others have posted I suggest you do not write your applications with reliance on the NLS_DATE_FORMAT parameter but rather you screens and application should specify the expected DATE entry format and the code should use the TO_DATE function to store the data into the database. All application SQL should use the TO_CHAR function to format date output for display.
I need to insert NULL into the table whenever the SQL*Loader encounters a bad date (0000-00-00 00:00:00) while reading a .csv file.
The bad date provided in the csv is always 0000-00-00 00:00:00.
I need help figuring out the case for it that handles both a bad date and a correct date.
There are two cases:
Date in the correct format (YYYY/MM/DD hh24:mi:ss)
Date in the bad format (0000-00-00 00:00:00)
Initially I had a simple statement in SQL control file as below, now I need to add the case to handle bad date as well.
START_DATE DATE "YYYY/MM/DD hh24:mi:ss" NULLIF (START_DATE = "NULL").
I need an SQL*Loader statement that handles both the above cases.
All data in a file is essentially a string until validated as a different datatype. So, you can treat it as a string for the bad data before converting it into a date:
START_DATE "TO_DATE(NULLIF(:START_DATE, '0000-00-00 00:00:00'), 'yyyy/mm/dd hh24:mi:ss')"
I'm a little concerned that the format of the "bad" date is different to that of the good date. It indicates something is going on that you're maybe not aware of for instance the data coming from two different systems or this field not being stored as a date in the source system.
If possible, I'd double check how this data is being constructed and what the intended meaning of the bad date is.
I am at a loss as how to insert the current time in a different format than the default. Can somebody help explain?
Here is how my table was created:
CREATE TABLE ACTIVITY_LOG
(
TIME TIMESTAMP NOT NULL
, ACTIVITY VARCHAR2(200) NOT NULL
);
My insert command works:
insert into activity_log
values (localtimestamp,'blah');
But how do i insert the localtimestamp value into my table in a different format using the various MM DD YY HH MM SS tags? I've tried the following, but it gives me the ORA-1830: date format picture ends before converting entire input string error.
insert into activity_log
values (to_timestamp(localtimestamp,'YYYY/MM/DD'),'blah');
You don't insert a timestamp in a particular format. Timestamps (and dates) are stored in the database using an internal representation, which is betwen 7 and 11 bytes depending on the type and precision. There is more about that in this question, among others.
Your client or application decides how to display the value in a human-readable string form.
When you do:
to_timestamp(localtimestamp,'YYYY/MM/DD')
you are implicitly converting the localtimestamp to a string, using your session's NLS settings, and then converting it back to a timestamp. That may incidentally change the value - losing precision - but won't change how the value is stored internally. In your case the mismatch between the NLS setting and the format you are supplying is leading to an ORA-01830 error.
So your first insert is correct (assuming you really want the session time, not the server time). If you want to see the stored values in a particular format then either change your client session's NLS settings, or preferably format it explicitly when you query it, e.g.:
select to_char(time, 'YYYY-MM-DD HH24:MI:SS.FF3') from activity_log
You don't seem to provide any indication of what your 'localtimestamp' is - is that pseudocode? A variable name? A column you haven't shown the definition for?
What data type is 'localtimestamp'? What data does it contain? Pertinent questions as other answers point out, because if it truly is a time stamp then oracle will be converting it to a string for you, before passing that string to to_timestamp() in your final query. Your initial stab at it should just work if the variable is a timestamp, containing a timestamp
Ultimately "date format picture ends" means "you passed me a string looking like '2017-05-17 12:45:59', but claimed it was only 'yyyy-mm-dd'. What was I expected to do with the rest of it?"
Your current final comment on your question "I was hoping to look in the table and see a useful looking time" - that's your query tool's problem. Have a look in the setting of your query tool and change the date format it displays. As has been noted, dates in oracle are stored as a decimal number days since a certain moment in time. If 0 represents 01 Jan 1970, then 1.75 represents 6pm on the 2 Jan 1970. It is up to the end program the user is using, to format the date into something you like.. you cannot "insert a timestamp with a different format" because time stamps don't have a format any more than a number like 1.75 has a format. It is what your query does with it when it gets it out, that gives it the format:
To_char(timestampcol, 'yyyy mm did')
To-char(tomestampcol, 'mon dd yyyy')
These use oracles built in date formatter, that turns that decimal number of the date into a string in the given format; you will see a string.. or you can just write "select * from table" and run it in TOAD and toad will show you the dates according to the format in settings, or you can write a c# program and get a load of date objects out and call my date.ToString("yyyy-MM-dd") on them to format them. The idea I'm trying to get across is that you don't pick the date format on the way in, you pick it on the way out, if you don't like what you're looking at, you have to change it on the way out, not the way in
I have a column in my table which will store the time as hh24miss format, i.e it stores as 091315 which is 09 hrs 13 min 15 sec. I need to convert it into HH24:MI:SS AND concatenate it with the date column which is in YYYYMMDD format.
Simply, the following columns Date: 19940601 and Time: 091315 need to be converted to
01-Jan-94 09:13:15.
You should not store dates as strings, and there is no need to store the time in a separate field. Oracle's DATE data type includes times down the to the second. (You'd need TIMESTAMP for fractions of a second).
If you are really stuck with this schema then you need to convert the two strings into a DATE:
to_date(date_column || time_column, 'YYYYMMDDHH24MISS')
You can then display that in whatever format you want; what you showed would be:
to_char(to_date(date_column || time_column, 'YYYYMMDDHH24MISS'),
'DD-Mon-RR HH24:MI:SS')
Although the data you have is June, not January.
SQL Fiddle demo.
But really, please revisit your schema and use the appropriate data type for this, don't store the values as strings. You have no validation and no easy way to check that the values you have stored actually represent valid dates and times.
We have a situation, where we have a table (say Forms) in Oracle DB which has a column (say edition_date) of type date. It was strictly meant to hold the date information in YYYY-MM-DD (ex: 2012-11-23)format with no timestamp value.
Unfortunately, due to a code problem, lot of rows got created with timestamp values. There are tons of records in this table, and I want to update only those rows which had this bad data. I can do that using the query
UPDATE forms SET edition_date = TRUNC( edition_date )
I want to add a where clause to it, so it updates only the bad data. The problem, I am not sure how to retrieve those rows that has timestamp added to it. Below is a snapshot of the data I have:
FORM_ID EDITION_DATE
5 2012-11-23
6 2012-11-23 11:00:15
..
11 2010-07-11 15:23:22
..
13 2011-12-31
I want to retrieve only the row with form ids 6 and 11. I trioed using the length functions but I think that is good for Strings only. Is there any way to do this. Thanks anyone who can help me.
A date has no format; you're only seeing how it's displayed. However, the answer to your question is, effectively, what you've said:
I want to add a where clause to it, so it updates only the bad data.
So, where the date is not equal to the date without time:
update forms
set edition_date = trunc(edition_date)
where edition_date <> trunc(edition_date)
To ensure that this doesn't happen again you can add a check constraint to your table:
alter table forms
add constraint chk_forms_edition_date
check ( edition_date = trunc(edition_date) );
I would advise against all of the above. Don't destroy potentially useful data. You should simply select trunc(edition_date) where you do not want time. You may want to use the time in the future.
You're correct, do not use LENGTH() etc for dates, it depends on your NLS_DATE_FORMAT settings and so could be different on a session-by-session basis. Always use date functions when dealing with dates.