Using Datastage 11.5.0.2, Jobs failed when it try to load the data as "0" into DATE FIELD (DB2)..
In source DB, the column is VARCHAR whereas in target it is DATE field.. the only value in source which failed to load is 0. how to resolve.. any idea pl
Create one stage variable: if (input column value ) = 0 then use StringToDate("0001-01-01", "%dd:%mm:%yyyy")
Else StringToDate(inputcolumn, "%dd:%mm:%yyyy")
You can use the in the transformer stage,
StringToTimestamp(Column,"%yyyy-%mm-%dd")
also you can convert data in Source qualifier by using below in select query
to_date(column,"%yyyy-%mm-%dd")
Can use the to_date function
to_date(column,'YYYY-MM-DD')
TO_DATE(column, 'YYYY-MM-DD HH24:MI:SS')
Related
in some part of my program , I want to run a sql query and have the result which is a date like : %Y/%m/%d %H:%M:%S
SELECT MAX(created_at)
FROM HOT_FILES_LOGS
WHERE FILE_NAME = 'test'
date in created_at column is stored like 04/03/2021 15:45:30 ( it is fulled with SYSDATE)
but when I run this query, I get just 04.03.21
what should I do to fix it?
Apply TO_CHAR with appropriate format mask:
select to_char(max(created_at), 'yyyy.mm.dd hh24:mi:ss') as created_at
from hot_files_logs
where file_name = 'test'
Oracle does not store dates or timestamps in any display format, they are stored in an internal structure, every date in every Oracle database since at least 8i and probably earlier. This structure consists of 7 1-byte integers (timestamps in a similar but larger structure). How the date is displayed or a string converted to a date is controlled the specified date format string in the to_char or to_date function or if no format string given by the NLS_DISPLAY_FORMAT setting. To get a gimps at the internal settings run the following:
create table td( d date);
insert into td(d) values(sysdate);
select d "The Date" , dump(d) "Stored As" from td;
See example. The last used format is not practical but strictly demonstrable. Well I guess you could use it to seed a repeatable random sequence.
I am using SAP BODS and I am trying to fetch data from an ORACLE server using SQL query transormation. Now The table has a column named latest_changed_date which is a datetime column. I only want yesterday and current day data from that table. Now since the column is datetime, I need to convert it to date, but when I am using to_date function I get the following error.
SELECT *
FROM ABC.TEST
WHERE TO_DATE(LATEST_CHANGED_DATE) = TO_DATE(SYSDATE-1)
The database error message is
ORA-01843: not a valid month
I tried giving date format in TO_DATE condition as below:
SELECT *
FROM ABC.TEST
WHERE TO_DATE(LATEST_CHANGED_DATE,'YYYY-MM-DD') >= TO_DATE(SYSDATE-1,'YYYY-MM-DD')
Here I got the error:
date format picture ends before converting entire input string
I used trunc function also and again got either:
not a valid month
or
inconsistent datatypes: expected NUMBER got DATE
Below is a sample data for the column. I just need data for current and day before data from the column.
Update: I think the main issue is that I am not able to determine the proper datatype for the column in the source table and currently I don't have an option to determine that.
Rather than trying to implicitly cast your dates to strings and convert them back using TO_DATE( string_value, format_model ) you can use TRUNC() to truncate SYSDATE to the start of the day:
SELECT *
FROM ABC.TEST
WHERE LATEST_CHANGED_DATE >= TRUNC( SYSDATE-1 )
this will work:
SELECT *
FROM ABC.TEST
where sysdate-LATEST_CHANGED_DATE<=sysdate-(sysdate-2);
for example take this:
ALTER SESSION SET NLS_DATE_FORMAT = ' DD-MON-YYYY HH24:MI:SS';
SELECT * FROM d061_dates ;
03-DEC-2018 17:44:38
25-AUG-2018 17:44:42
30-AUG-2018 17:44:46
01-DEC-2018 17:44:49
02-DEC-2018 17:46:31
SELECT * FROM d061_dates
where sysdate-a<=sysdate-(sysdate-2);
03-DEC-2018 17:44:38
02-DEC-2018 17:46:31
you have to take sysdate minus on both sides to get comparision by a number which is less than equal to 2 to get day and day before yesterday and its giving the correct output.
thank you!!!!!!!!!!!!!
I am getting below error in Spark(1.6) SQL while creating a table with column value default as NULL. Ex: create table test as select column_a, NULL as column_b from test_temp;
The same thing works in Hive and creates the column with data type "void".
I am using empty string instead of NULL to avoid the exception and new column getting string data type.
Is there any better way to insert null values in hive table using spark sql ?
2017-12-26 07:27:59 ERROR StandardImsLogger$:177 - org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.UnsupportedOperationException: Unknown field type: void
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:789)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:746)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$createTable$1.apply$mcV$sp(ClientWrapper.scala:428)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$createTable$1.apply(ClientWrapper.scala:426)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$createTable$1.apply(ClientWrapper.scala:426)
at org.apache.spark.sql.hive.client.ClientWrapper$$anonfun$withHiveState$1.apply(ClientWrapper.scala:293)
at org.apache.spark.sql.hive.client.ClientWrapper.liftedTree1$1(ClientWrapper.scala:239)
at org.apache.spark.sql.hive.client.ClientWrapper.retryLocked(ClientWrapper.scala:238)
at org.apache.spark.sql.hive.client.ClientWrapper.withHiveState(ClientWrapper.scala:281)
at org.apache.spark.sql.hive.client.ClientWrapper.createTable(ClientWrapper.scala:426)
at org.apache.spark.sql.hive.execution.CreateTableAsSelect.metastoreRelation$lzycompute$1(CreateTableAsSelect.scala:72)
at org.apache.spark.sql.hive.execution.CreateTableAsSelect.metastoreRelation$1(CreateTableAsSelect.scala:47)
at org.apache.spark.sql.hive.execution.CreateTableAsSelect.run(CreateTableAsSelect.scala:89)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:56)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:56)
at org.apache.spark.sql.DataFrame.withCallback(DataFrame.scala:153)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:145)
at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:130)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:829)
I couldn't find much information regarding the datatype void but it looks like it is somewhat equivalent to the Any datatype we have in Scala.
The table at the end of this page explains that a void can be cast to any other data type.
Here are some JIRA issues that are kinda similar to the problem you are facing
HIVE-2901
HIVE-747
So, as mentioned in the comment, instead of NULL you can cast it to any of the implicit data types.
select cast(NULL as string) as column_b
I started to get a similar issue. I build the code down to an example
WITH DATA
AS (
SELECT 1 ISSUE_ID,
DATE(NULL) DueDate,
MAKE_DATE(2000,01,01) DDate
UNION ALL
SELECT 1 ISSUE_ID,
MAKE_DATE(2000,01,01),
MAKE_DATE(2000,01,02)
)
SELECT ISNOTNULL(lag(IT.DueDate, 1) OVER (PARTITION by IT.ISSUE_ID ORDER BY IT.DDate ))
AND ISNULL(IT.DueDate)
FROM DATA IT
I have created table with single column having data type -dateTimeOffset value and inserted some values.
create table dto (dto datetimeoffset(7))
insert into dto values (GETDATE()) -- inserts date and time with 0 offset
insert into dto values (SYSDATETIMEOFFSET()) -- current date time and offset
insert into dto values ('20131114 08:54:00 +10:00') -- manual way
In Nifi,i have specified "Select * from dto" query in Execute SQL .
It shows below error..,
java.lang.IllegalArgumentException: createSchema: Unknown SQL type -155 cannot be converted to Avro type
If i change that column into dateTime then ExecuteSQL runs correctly but it doesn't worked in DateTimeOffset column.
Any help appreciated.
Many thanks
datetimeoffset is a MSSQL-specific JDBC type and is not supported by ExecuteSQL (which supports the standard JDBC types). You could try to cast the datetimeoffset field into some other standard type such as datetime, as described here.
I've created a Custom Processor and adapted the JdbcCommon.java class to include SQL Server's DATETIMEOFFSET. It's just one line of code. I'll try to see if I can ask them to merge this on the official repo.
This is a piece of my JdbcCommon.java:
case TIMESTAMP:
case TIMESTAMP_WITH_TIMEZONE:
case -101: // Oracle's TIMESTAMP WITH TIME ZONE
case -102: // Oracle's TIMESTAMP WITH LOCAL TIME ZONE
case -155: // SQL Server's DATETIMEOFFSET <---- added this line
addNullableField(builder, columnName,
u -> options.useLogicalTypes
? u.type(LogicalTypes.timestampMillis().addToSchema(SchemaBuilder.builder().longType()))
: u.stringType());
break;
i have tryed many things but not able to insert data in my timestamp column.
from toad its possible using this
UPDATE SUPPORTSTAFF
SET SUPPSTAFFJOINDATE=to_timestamp('27/02/2002 15:51.12.539880', 'dd/mm/yyyy hh24:mi.ss.ff')
where JDBCUSERID='5700';
its working
but how can i insert data from java class using create statment and execute query its giving me invalid month error
Use a PreparedStatement with a parameter for the timestamp, e.g.
UPDATE SUPPORTSTAFF SET SUPPSTAFFJOINDATE = ? where JDBCUSERID = ?
and then set the parameters:
statement.setTimestamp(1, new Timestamp(System.currentTimeMillis()));
statement.setString(2, "your ID");
(Then execute the statement, obviously.)