I have a need to pull in via FTP only files from dates I haven't pulled data. The data pulls happen nightly, but occasionally there is an issue and a night gets skipped, and it will be picked up the following night.
I located a query on StackOverflow which addressed most of my problem. However, I'm left with an uncomfortable solution and a lingering question.
I have a table with data from each file downloaded. The crux of the data I'm using for this example is EXTRACT_DATE. The file format is filename_20160809.csv, as an example. Uses the 4-digit year, 2-digit month and 2-digit day to make unique. The FTP location has thousands of files, and I only want to grab the new, based on this date.
First, I get the latest EXTRACT_DATE from my table as such
SELECT MAX(EXTRACT_DATE) INTO checkDate FROM FILE_DETAILS;
I had attempted to do this in a single query, but couldn't work it, and finally attempted to create a variable, checkDate, and use this variable in a subsequent query to obtain my list. However, having two queries in a single script is not allowed, or I haven't found a way to do it. So my primary issue is to return the value of the latest date. Then call a new query or procedure with the value incorporated in to obtain my list, with this query
SELECT TO_DATE(checkDate, 'MM/DD/YYYY') + rownum AS EXTRACT_DATES
FROM ALL_OBJECTS
WHERE TO_DATE(checkDate, 'MM/DD/YYYY') + rownum <= TO_DATE(SYSDATE, 'DD-MM-YY');
This solution is messy and uncomfortable, I would prefer a single query to get back my results, rather then 2 scripts or a script and a procedure.
The lingering question, the result from the query returns dates in the mm/dd/yyyy format; 8/9/2016. I adjusted the TO_DATE to match.
Initially, it was set to return as YYYY-MM-DD; 2016-08-09.
However, it wouldn't return in this format. It would only come back as 8/9/2016, regardless of the TO_DATE formatting used. I don't understand why the date is coming back in this format.
SELECT TO_DATE('2-AUG-2016', 'DD-MON-RR') + rownum AS EXTRACT_DATES
FROM ALL_OBJECTS
WHERE TO_DATE('2-AUG-2016', 'DD-MON-RR') + rownum <= TO_DATE(SYSDATE, 'DD-MM-YY');
EXTRACT_DATES
8/3/2016
8/4/2016
8/5/2016
8/6/2016
8/7/2016
8/8/2016
8/9/2016
NLS_DATE_FORMAT is set to DD-MON-RR, with American as NLS_DATE_LANGUAGE.
Related
My Datedim function is not returning yesterdays date in webi, any ideas on how to show 13/04/2022, even if it has null values?
Thanks
If you have gaps in your date data the simplest way to fill them in is to create a variable with the TimeDim() function. However, that will not work for you since you do not have a true gap because your missing date is at the end.
You need a data source with all the dates you want to display regardless of if you have data for those dates or not and then merge on your date dimension. I answered a question very similar to this here. I am copying my answer from there below...
The TimeDim() function will fill in the empty periods in your time
data. The problem with that though is if it is the end of your date
range that is missing data those dates will not show up. Let me show
you what I mean. Here is my sample data from 12/01/2021 through
12/26/2021 (note missing dates) in the table on the left. The table on
the right is the my Var Data Date TimeDim variable defined as…
=TimeDim([Data Date]; DayPeriod)
So we have our missing dates in the middle, but not at the end
(12/25/2021 and 12/26/2021). To get those dates you need a query to
return all the dates in your specified range. If you have a universe
based on a
calendar
you could use that. Free-hand SQL based on a calendar table would
suffice as well.
If you have neither of those we can still get it to work using
free-hand SQL with a CTE. This is SQL Server syntax. You will have to
modify this SQL to work for whatever database platform you have if it
isn’t SQL Server.
Here is the SQL…
;with dates ([Date]) as (
Select convert(date,‘2021-12-01’) as [Date] – Put the start date here
union all
Select dateadd(day, 1, [Date])
from dates
where [Date] < ‘2021-12-26’ – Put the end date here
)
select [Date]
from dates
option (maxrecursion 32767) – Don’t forget to use the maxrecursion option!
Source: Generate a Date Table via Common Table Expression (CTE) |
Data and Analytics with Dustin
Ryan
Here is a
demo.
Now that you have a query returning all of the dates in your range
you can merge the date from that query to your Data Date.
You can then put the date object with all of dates or the Merged Date
in table with any measures from your pre-existing query and there you
have it.
If you need to add dimensions from you pre-existing query I think you
will need to create variables for them with Qualification set to
“Detail” and the Associated dimension set to “Merged Date” (or
whatever you called it). And if you do that I believe you will also
need to check “Avoid duplicate row aggregation” check box within the
Format Table properties.
Let us know how it goes.
Hopefully that will get you on the right track.
There are records in table for particular date. But when I query with that value, I am unable to filter the records.
select * from TBL_IPCOLO_BILLING_MST
where LAST_UPDATED_DATE = '03-09-21';
The dates are in dd-mm-yy format.
To the answer by Valeriia Sharak, I would just add a few things since your question is tagged Oracle. I was going to add this as a comment to her answer, but it's too long.
First, it is bad practice to compare dates to strings. Your query, for example, would not even execute for me -- it would end with ORA-01843: not a valid month. That is because Oracle must do an implicit type conversion to convert your string "03-09-21" to a date and it uses the current NLS_DATE_FORMAT setting to do that (which in my system happens to be DD-MON-YYYY).
Second, as was pointed out, your comparison is probably not matching rows due LAST_UPDATED_DATE having hours, minutes, and seconds. But a more performant solution for that might be:
...
WHERE last_update_date >= TO_DATE('03-09-21','DD-MM-YY')
AND last_update_date < TO_DATE('04-09-21','DD-MM-YY')
This makes the comparison without wrapping last_update_date in a TRUNC() function. This could perform better in either of the following circumstances:
If there is an index on last_update_date that would be useful in your query
If the table with last_update_date is large and is being joined to other tables (because it makes it easier for Oracle to estimate the number of rows from your table that are inputs to the join).
Your column might contain hours and seconds, but they can be hidden.
So when you filter on the date, oracle implicitly adds time to the date. So basically you are filtering on '03-09-21 00:00:00'
Try to trunc your column:
select * from TBL_IPCOLO_BILLING_MST
where trunc(LAST_UPDATED_DATE) = '03-09-21';
Hope, I understood your question correctly.
Oracle docs
I have the following SQL which should return all data up to 6.30am tomorrow. It has been working correctly until today (30/6/2016).
SELECT TO_CHAR(TRUK.THEDATE,'DD/MM/YY HH24:MI') DAT
FROM TRUK
WHERE TO_CHAR(TRUK.THEDATE,'DD/MM/YY HH24:MI') <= TO_CHAR(TO_DATE(sysdate + 1) + 6.5/24,'DD/MM/YY HH24:MI')
ORDER BY TRUK.THEDATE
PROBLEM
Today the data returned does not include data for 30th June,which i know exists but only 1st July. When i comment out the where clause, then all data is returned but of course this includes data AFTER 6.30am for the next day which i don't want returned.
I have searched in vain for an answer and would greatly appreciate some assistance with this. How Could the where clause be written differently to prevent this?
My desired result is that all records in the database are returned up to 6.30am the following day.
cheers
You're comparing strings, but you should be comparing dates.
Adjust your where clause to this and it'll work:
WHERE truk.thedate <= trunc(sysdate) + to_dsinterval('1 06:30:00')
I have problem with some thing input multiple date through prompt
my query something like
select something
from
something
where
to_date (bus.TDATE) not in (:didb1)
I want to input like '12-Jun-2016','11-Jun-2016'
i am doing php oracle,
My code following:
select bus.*,con.* from
BusinessPayment bus,
ConsumerPayment con
where bus.TDATE=con.TDATE
and to_date (bus.TDATE) not in (:didbv)
');
$bscode1='11-June-2016','10-June-2016','09-June-2016';
oci_bind_by_name($stid, ':didbv', $bscode1);
oci_execute($stid);
You can't use a single bind variable to represent a list of values in an in() clause. Bind variables don't contain multiple values; the value you bind is seen as a single string value. So you're effectively doing:
where
to_date (bus.TDATE) not in (q'['11-June-2016','10-June-2016','09-June-2016']')
If TDATE is already a date type - which it hopefully is, and the joins suggest it might be - then you should not be passing that through to_date() - you are doing an implicit conversion from date to string, then a semi-explicit conversion back to a date, both using your session's NLS_DATE_FORMAT. That is either doing unnecessary work, or losing resolution, depending on your format model and whether the dates have times. If the intention is to ignore the time part of the value then there are better ways to do that.
Anyway, since that is a date now whatever type it was before, the right had side of that filter will also be converted to a date, so you're doing:
where
to_date (bus.TDATE) not in (to_date(q'['11-June-2016','10-June-2016','09-June-2016']'))
... and it's that which is throwing the exception, whatever your NLS settings are. Even if you passed a single value it would error because of the embedded single quotes. You can replicate that with:
select to_date(q'['11-June-2016','10-June-2016','09-June-2016']') from dual;
which also gets ORA-01858: a non-numeric character was found where a numeric was expected.
Ideally you would pass the individual date strings as elements of an array, but you can also work around this with an XML hack. You can pass a single string containing comma-delimited values to XMLTable(), and it will split it in to rows:
select * from xmltable(q'['11-June-2016','10-June-2016','09-June-2016']');
COLUMN_VALUE
--------------------------------------------------------------------------------
11-June-2016
10-June-2016
09-June-2016
Then you can convert each of those values to a date:
select to_date(column_value, 'DD-Month-YYYY')
from xmltable(q'['11-June-2016','10-June-2016','09-June-2016']')
and use that in a subquery or a join:
select * from dual
where trunc(sysdate) not in (select to_date(column_value, 'DD-Month-YYYY')
from xmltable(q'['11-June-2016','10-June-2016','09-June-2016']'));
And that works with a string bind variable, so your code would be:
select bus.*,con.*
from BusinessPayment bus
join ConsumerPayment con
on bus.TDATE=con.TDATE
where bus.TDATE not in (select to_date(column_value, 'DD-Month-YYYY')
from xmltable(:didbv));
I've modified this to use ANSI join syntax, which isn't relevant but there's rarely a reason to use the ancient syntax you have. If TDATE does include the time you'll need to modify it a bit - truncating is simplest but might affect performance.
Or you could join:
select bus.*,con.*
from xmltable(:didbv) x
join BusinessPayment bus
on bus.TDATE >= to_date(x.column_value, 'DD-Month-YYYY')
and bus.TDATE < to_date(x.column_value, 'DD-Month-YYYY') + 1
join ConsumerPayment con
on bus.TDATE=con.TDATE
In this version TDATE can have a time, and it matches any time on the days specified in the bind variable.
It would be better to supply the dates in a format that doesn't have the month name in it, as that could give you problems when run in different sessions or locales. If you know the PHP side will always be English you could force that to be recognised:
on bus.TDATE >= to_date(x.column_value, 'DD-Month-YYYY', #NLS_DATE_LANGUAGE=ENGLISH')
but if you can change the PHP format to use month numbers, and change the Oracle format model to match, that would be a bit simpler.
this might be a lack of very basic knowledge, but I just can't figure it out. Searching for the answer and trial and error haven't helped much.
Returning all recordsets from a table (SELECT * FROM X) --> no problem.
Returning today's date (SELECT TO_CHAR(SYSDATE, 'DD-MM-YYYY') FROM DUAL) --> no problem.
Returning all recordsets from the same table as well as today's date --> no luck. I have tried subselects, union, joins, with-statements, ... it's driving me nuts.
When I name the columns I want returned (SELECT Columname1, Columnname2, to_char(sysdate....)) it works. This problems seems to only occur when using wildcards.
How do I get Oracle to return "all columns", today's date"?
Thanks!
You have to prefix the wildcard with the table name (or alias, if you've used one):
SELECT X.*, TO_CHAR(SYSDATE, 'DD-MM-YYYY') AS TODAYS_DATE FROM X
Using the wildcard is generally not considered a good idea, as you have no control over the order the columns are listed (if the table was built differently in different environments) and anyone consuming this output may be thrown if the table definition changes in the future, e.g. by adding another column. It's better to list all the columns individually.