I have a TIMESTAMPTZ column in a cockroachDB source, by using clickhouse kafka consumer to read from cockroachDB changefeed i stored the TIMESTAMPTZ fields as DateTime however this resulted with inaccurate data, something of such sort:
1970-01-01 00:00:00
how to map TIMESTAMPTZ to the accurate date type in Clickhosue?
Welcome to the CRDB community!
I'm not very familiar with changefeed nor clickhouse but I'll try my best to help.
I tried to set up a CRDB changefeed on a table with a TIMESTAMPZ column:
create table t (i int primary key, j timestamptz);
insert into t values (1, now());
The output string of this TIMESTAMPZ columns uses ISO 8601 format:
root#localhost:26257/defaultdb> EXPERIMENTAL CHANGEFEED FOR t;
{"key":"[1]","table":"t","value":"{\"after\": {\"i\": 1, \"j\": \"2023-01-16T14:44:01.337341Z\"}}"}
So following #Denny Crane's lead, it does seem like using best_effort will allow Clickhouse to parse input date/time in ISO 8601 format.
Can you try and let us know whether it helps? If not, I can engage my colleague who have more expertise on this matter to help you.
Related
I have an Oracle database and I have to load dat from this database to Azure SQL DWH. This is done once every day. At the beginning of the pipeline I first do a lookup on SQL DWH to look for the latest date. The result for that is something like '2015-10-25'.
I want to use this date to query the Oracle database. But I allready found out, by trying the query on Oracle that the following code does not work:
Select * from Table 1 where day = '2015-10-25'
The date in the day column looks like 25-OCT-15 (DD-MON-YY).
I treid the following where clause:
where day = TO_DATE('2015-10-25','DD-MON-YY')
But then I get the error: "literal does not match format string"
I realy don't know how to make Oracle understand this T-SQL date format.
Your Oracle column is of date datatype. When you connect to an Oracle database and write a query against that date column, you will see its default format DD-MON-YY as per this reference.
You can override this setting by running an ALTER SESSION command, eg
ALTER SESSION SET NLS_DATE_FORMAT = 'YYYY MM DD';
but this is just in that local session. The data is still stored in Oracle in the same manner and it's simply the way you view it that is changing.
In Azure Data Factory (ADF v2) and your example, you are dealing with strings. So you must make sure any parameters you pass in are in the correct format or set to the correct datatype. The Oracle function TO_DATE converts strings to the date datatype. Therefore when passing in a string of format YYYY-MM-DD then that is the format you must use, to let the TO_DATE function know what you are passing in:
TO_DATE('2015-10-25','YYYY-MM-DD')
The function then successfully converts your parameter to a date datetype for correct comparison against the main date column.
You can try this query:
Select * from Table 1 where day = to_char (to_date('2015-10-25','YYYY-MM-DD'), 'DD-Mon-YY')
Reference this blog: how to convert YYYYMMDD to DD-Mon-YYYY in oracle?
Hope this helps.
I have a external table in hive, which has report_dt as column and datatype is string and has value like 2018-09-02 2018-09-03
And i want to create managed table from the query written on this external table with report_dt_1 as column with datatype as date
i have gone through some threads and i was able to query like
create table manag_newtable as select to_date(from_unixtime(unix_timestamp(report_dt,'yyyy-MM-dd'),'yyyy-MM-dd')) as report_dt_1 from exter_table;
so the above query creates a new table, but when i desc manag_newtable ,still that report_dt_1 shows as string datatype
Prior to Hive 2.1.0, TO_DATE returns a string.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions
Perhaps you want to try a CAST.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-date
Otherwise, I suggest leaving the column as epoch time because 1) with your current format you lose timezone information, although you never had it so you are assuming UTC? 2) Hive requires less deserialization time for numbers, making queries faster. If you have to format the epoch, do that at the reporting layer, not the storage layer
I am at a loss as how to insert the current time in a different format than the default. Can somebody help explain?
Here is how my table was created:
CREATE TABLE ACTIVITY_LOG
(
TIME TIMESTAMP NOT NULL
, ACTIVITY VARCHAR2(200) NOT NULL
);
My insert command works:
insert into activity_log
values (localtimestamp,'blah');
But how do i insert the localtimestamp value into my table in a different format using the various MM DD YY HH MM SS tags? I've tried the following, but it gives me the ORA-1830: date format picture ends before converting entire input string error.
insert into activity_log
values (to_timestamp(localtimestamp,'YYYY/MM/DD'),'blah');
You don't insert a timestamp in a particular format. Timestamps (and dates) are stored in the database using an internal representation, which is betwen 7 and 11 bytes depending on the type and precision. There is more about that in this question, among others.
Your client or application decides how to display the value in a human-readable string form.
When you do:
to_timestamp(localtimestamp,'YYYY/MM/DD')
you are implicitly converting the localtimestamp to a string, using your session's NLS settings, and then converting it back to a timestamp. That may incidentally change the value - losing precision - but won't change how the value is stored internally. In your case the mismatch between the NLS setting and the format you are supplying is leading to an ORA-01830 error.
So your first insert is correct (assuming you really want the session time, not the server time). If you want to see the stored values in a particular format then either change your client session's NLS settings, or preferably format it explicitly when you query it, e.g.:
select to_char(time, 'YYYY-MM-DD HH24:MI:SS.FF3') from activity_log
You don't seem to provide any indication of what your 'localtimestamp' is - is that pseudocode? A variable name? A column you haven't shown the definition for?
What data type is 'localtimestamp'? What data does it contain? Pertinent questions as other answers point out, because if it truly is a time stamp then oracle will be converting it to a string for you, before passing that string to to_timestamp() in your final query. Your initial stab at it should just work if the variable is a timestamp, containing a timestamp
Ultimately "date format picture ends" means "you passed me a string looking like '2017-05-17 12:45:59', but claimed it was only 'yyyy-mm-dd'. What was I expected to do with the rest of it?"
Your current final comment on your question "I was hoping to look in the table and see a useful looking time" - that's your query tool's problem. Have a look in the setting of your query tool and change the date format it displays. As has been noted, dates in oracle are stored as a decimal number days since a certain moment in time. If 0 represents 01 Jan 1970, then 1.75 represents 6pm on the 2 Jan 1970. It is up to the end program the user is using, to format the date into something you like.. you cannot "insert a timestamp with a different format" because time stamps don't have a format any more than a number like 1.75 has a format. It is what your query does with it when it gets it out, that gives it the format:
To_char(timestampcol, 'yyyy mm did')
To-char(tomestampcol, 'mon dd yyyy')
These use oracles built in date formatter, that turns that decimal number of the date into a string in the given format; you will see a string.. or you can just write "select * from table" and run it in TOAD and toad will show you the dates according to the format in settings, or you can write a c# program and get a load of date objects out and call my date.ToString("yyyy-MM-dd") on them to format them. The idea I'm trying to get across is that you don't pick the date format on the way in, you pick it on the way out, if you don't like what you're looking at, you have to change it on the way out, not the way in
I'm using Pentaho to datamask some of the information on the oracle DB
I have several transformations of the form:
SELECT -> data mask -> UPDATE rows based on primary key
I have tables where a timestamp is part of the primary key in the update step. Even though I am not masking or updating this field in any way, I get the error ORA-01843: not a valid month when performing the update.
I believe this is because when Pentaho takes in the timestamp from Step 1 it doesn’t actually keep it as a timestamp until I try the update and hence the primary key check. Outputting to excel, I see pentaho giving timestamps in the format
2014-07-30 15:44:31.869033 Europe/London (Pentaho)
But in DB the format is
30-JAN-15 09.21.38.109145000 AM (Oracle - TIMESTAMP(6) WITH LOCAL TIME ZONE)
I have tried to convert the pentaho field to a Timestamp (format: yyyy-MM-dd HH:mm:ss.SSSSSS) before the update step but receive errors if I try and use milliseconds.
2017/03/14 13:19:25 - Select values.0 - AUDIT_CREATE_TS Timestamp : couldn't convert string [2015-01-30 09:21:38.109145 Europe/London] to a timestamp, expecting format [yyyy-mm-dd hh:mm:ss.ffffff]
2017/03/14 13:19:25 - Select values.0 - Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
If I replace my formatting to the one suggested by Pentaho I get "Illegal character 'f'" and then I am stuck in a loop.
Ignoring milliseconds seems to succeed but won’t give me any matches because it isn’t precise enough and returns no results from db..
Any help would be appreciated!
Not sure about Pentaho, but if you're looking for a conversion from this string:
'2015-01-30 09:21:38.109145 Europe/London'
to a timestamp with timezone in Oracle, it would be:
select to_timestamp_tz('2015-01-30 09:21:38.109145 Europe/London', 'YYYY-MM-DD HH24:MI:SS.FF6 TZR') from dual;
See Oracle Datetime Format Models document for more.
I'm a Business Intelligence intern and am trying to write a simple ETL batch job to bring one table into our warehouse using SAP Data Services Designer. The source table has a timestamp column, which halts the job's execution, saying:
You cannot select directly from timestamp column . Using
a timestamp column in SQL transform may cause this error. See
documentation or notify Customer Support.
From the technical manual, this limitation is confirmed in the timestamp section, which reads:
You cannot use timestamp columns in the SQL transform or in an Oracle
stored procedure. To use a timestamp column in the SQL transform,
convert the timestamp column in the select list of the SQL transform
to a character format using the to_char function and convert it back
to timestamp using the to_date function."
I've tried remedying the problem by changing the output schema's column to a datetime type, and converting the timestamp in the SQL transform with
TO_DATE(TO_CHAR(SQL.DATETIME_STAMP, 'YYYY-MON-DD HH24:MI:SS'), 'YYYY-MON-DD HH24:MI:SS')
I'm missing a key concept as it still fails with error 54003 no matter what I try. Thoughts, anyone?