I am trying to parse the Unix timestamp in Laravel
/Date(1671847200000-0600)/
where 1671847200000 is the timestamp and this is -0600 offset (-06:00 i.e. -HH::mm)
expected output be like
2022-12-23 18:00:00
I shot a video with a sony camera.
A file with the extension .mp4 was created.
I want to change the shooting date and time to a file name.
The environment is Windows10.
What I tried
I first dragged and dropped the .mp4 video file to exiftool (-k) .exe.
ExifTool Version Number : 11.76
File Name : C0001.MP4
File Modification Date/Time : 2019:10:23 13:10:49+09:00
File Access Date/Time : 2019:11:17 21:50:24+09:00
File Creation Date/Time : 2019:11:17 21:50:21+09:00
MIME Type : video/mp4
Major Brand : Sony XAVC
Create Date : 2019:10:23 04:10:32
Modify Date : 2019:10:23 04:10:32
Track Create Date : 2019:10:23 04:10:32
Track Modify Date : 2019:10:23 04:10:32
Media Create Date : 2019:10:23 04:10:32
Media Modify Date : 2019:10:23 04:10:32
Time Zone : +09:00
Last Update : 2019:10:23 13:10:32+09:00
Creation Date Value : 2019:10:23 13:10:32+09:00
I then typed the following command at a Windows 10 command prompt:
C:\hoge>"exiftool(-k).exe" -r "-FileName<CreateDate" -d "%Y-%m-%d_%H-%M-%S - %%f.%%e" C:\hoge
Warning: [minor] The ExtractEmbedded option may find more tags in the movie data - C:/hoge/C0001.MP4
Warning: Error removing old file - C:/hoge/C0001.MP4
1 directories scanned
1 image files updated
-- press RETURN --
The following file was created.
2019-10-23_04-10-32 - C0001.MP4
How can I output the following file?
I don't know how to add time zone (+9) to CreateDate.
2019_1023_131032.MP4
If the timezone (+09:00) is the local time on the computer you are using, then add -api QuickTimeUTC to the command. This will let exiftool know that the CreateDate, which is supposed to be written as UTC time, was correctly written and to adjust it to the local time. Many older cameras would write the time stamp as local time instead of UTC.
The other option you can use if the time zone isn't the same as the local computer is the GlobalTimeShift option. In your example, you would add -GlobalTimeShift 9 to the command.
I just want to get the system current time using below format in shell
Expected Format:
2019-02-14T08:08:12.300Z
I tried below piece of script but It returned something different.
CURRENTDATE=`date +"%Y-%m-%d %T"`
echo $CURRENTDATE
Output
2019-02-27 01:22:57
I just ran date command in my linux box and got below response back:
-bash-3.2$ date
Wed Feb 27 01:43:26 PST 2019
We are passing the above output as input JSON file. But this output is not accepted by our JSON. So I just want to pass the date and time format in the above specified format (Expected Format).
You may use:
dt=$(date '+%Y-%m-%dT%T.%zZ')
echo "$dt"
2019-02-27T04:35:56.-0500Z
Also note use of --iso-8601=seconds:
date --iso-8601=seconds
2019-02-27T04:37:29-05:00
You can also use this one like
currentDate=$(date +"%Y-%m-%dT%H:%M:%SZ")
echo $currentDate
OutPut :-
2022-07-12T18:05:27Z
I've the following field: 1388481000000 as the number of milliseconds elapsed from the Unix Epoch (1970-01-01 UTC)
How can I convert to Unix TimeStamp?
I'm trying to use:
ToUnixTime(1388481000000,'dd/MM/yyyyHH:mm:ss','GMT')
but it gives me error...
How can I convert into Unix Timestamp? Many thanks!
You are mixing things up.ToUnixTime syntax is ToUnixTime(datetime) where the parameter is a datetime object.What you have is the milliseconds elapsed from the Unix epoch (1970-01-01 UTC).I assume you are trying to convert the milliseconds to a datetime object for which you will have to use ToDate and the syntax is ToDate(milliseconds).
ToDate(1388481000000)
Command used:
sqoop export --connect jdbc:oracle:thin:#//xxx:1521/BDWDEV4 --username xxx --password xxx --table TW5T0 --export-dir '/data/raw/oltp/cogen/oraclexport/TW5T0/2015-08-18' -m 8 --input-fields-terminated-by '\001' --lines-terminated-by '\n' --input-escaped-by '\"' --input-optionally-enclosed-by '\"'
The destination table has columns with datatype date in oracle but as show in error it is parsing simple date as timestamp
Error:
15/09/11 06:07:12 INFO mapreduce.Job: map 0% reduce 0% 15/09/11 06:07:17 INFO mapreduce.Job: Task Id : attempt_1438142065989_99811_m_000000_0, Status : FAILED Error: java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) Caused by: java.lang.RuntimeException: Can't parse input data: '2015-08-15'
at TZ401.__loadFromFields(TZ401.java:792)
at TZ401.parse(TZ401.java:645)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more Caused by: java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
at java.sql.Timestamp.valueOf(Timestamp.java:202)
at TZ401.__loadFromFields(TZ401.java:709)
... 12 more
Instead of changing your data files in Hadoop, you should use the --map-column-java argument in your sqoop export.
If you have for example two DATE columns named DATE_COLUMN_1 and DATE_COLUMN_2 in your Oracle table, then you can add the following argument to your sqoop command:
--map-column-java DATE_COLUMN_1=java.sql.Date,DATE_COLUMN_2=java.sql.Date
As mentioned before, the JDBC format has to be used in your Hadoop text file. But in this case yyyy-mm-dd will work.
From
http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html#_dates_and_times,
Oracle JDBC represents DATE and TIME SQL types as TIMESTAMP values. Any DATE columns in an Oracle database will be imported as a TIMESTAMP in Sqoop, and Sqoop-generated code will store these values in java.sql.Timestamp fields.
When exporting data back to a database, Sqoop parses text fields as
TIMESTAMP types (with the form yyyy-mm-dd HH:MM:SS.ffffffff) even if
you expect these fields to be formatted with the JDBC date escape
format of yyyy-mm-dd. Dates exported to Oracle should be formatted as
full timestamps.
So you would need to format the dates in your files to conform to the format yyyy-mm-dd HH:MM:SS.ffffffff before exporting to Oracle.
EDIT:
Answering the comment,
There around 70 files(tables) in hdfs I need to export..So,in all
files I need to change the date from yyyy-mm-dd to yyyy-mm-dd
HH:MM:SS.ffffffff, any simple way to format it.
Well you could write an awk script to do that for you. Or else you can check if the below idea works:
Create a new temporary table TEMPIMPORT with the same structure as table TW5T0 except changing the column which has the DATE datatype to VARCHAR2
Load using Sqoop into the new temporary table TEMPIMPORT.
Run the DML below to export the data back int TW5T0 (and commit of course):
insert into tw5t0 (select [[all_your_columns_here_except_date_column]],to_date(date_column,'yyyy-mm-dd') from tempimport);
used --connection-param-file ora.porperties in export sqoop
ora.properties contains
oracle.jdbc.mapDateToTimestamp=false
Oracle drivers map oracle.sql.DATE to java.sql.Timestamp, retaining
the time information. If you still want the incorrect but 10g
compatible oracle.sql.DATE to java.sql.Date mapping, then you can get
it by setting the value of mapDateToTimestamp flag to false (default
is true).
https://docs.oracle.com/cd/E11882_01/java.112/e16548/apxref.htm#JJDBC28920
For using with sqoop you need to add option:
--connection-param-file conn-param-file.txt
conn-param-file.txt:
oracle.jdbc.mapDateToTimestamp=false
For using with sqoop you need to add option:
--connection-param-file conn-param-file.txt
conn-param-file.txt:
oracle.jdbc.mapDateToTimestamp=false
If Hive table columns sequence doesn't match with RDBMS table columns sequence order then there is a chance of same error.
I have resolve my issue after rearrange the columns in RDBMS by creating the table again.