How to filter DateTime in QueryRecord (NIFI) - apache-nifi

I have a csv file like the one below. I want to filer records based on DateTime in Nifi
ValidDate,Skycondition,Precip5min,WdSpdAct2m,WdSpdAct90m,Visibility
12/31/2021 11:35:00,928.3,,3,5.6,36791.4
12/31/2022 11:40:00,943.7,,3,5.5,36810.6
12/31/2022 23:45:00,958.8,,3,5.5,36829.8
12/31/2022 23:50:00,973.4,,3,5.5,36848.9
12/31/2022 23:55:00,987.7,,3,5.4,36868.1
01/01/2023 00:00:00,988.9,,2.9,5.4,36887.3
01/01/2023 00:05:00,1002.7,,2.9,5.3,36886.9
01/01/2023 00:10:00,1016.1,,2.8,5.1,36886.5
01/01/2023 00:15:00,1029,,2.7,5,36886
01/01/2023 00:20:00,1041.6,,2.6,4.8,36885.6
01/01/2023 00:25:00,1053.7,,2.6,4.7,36885.2
Query:
select * from Flowfile where "ValidDate" < '01/01/2023 00:15:00'
It is returning only the below records. It is not returning records of the previous year.
ValidDate,Skycondition,Precip5min,WdSpdAct2m,WdSpdAct90m,Visibility
01/01/2023 00:00:00,988.9,,2.9,5.4,36887.3
01/01/2023 00:05:00,1002.7,,2.9,5.3,36886.9
01/01/2023 00:10:00,1016.1,,2.8,5.1,36886.5
Any help would be much appreciated!

Related

Spring MongoRepository filter string date column

Is it possible to filter a MongoRepository in Spring where there is a column with date strings like that:
2019-01-01T00:00:00
2022-01-20T00:00:00
2022-05-01T00:00:00
I try to get all columns where the Date value is after a certain date.
I tried the following statements:
findAllByPersonIdAndDateAfter(int id, Date after);
findAllByDateAfter(Date date)
findAllByDateGreaterThan(Date date);
Invoked it with e.g.:
Date after = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2022-03-31 23:00:00");
repository.findAllByDateAfter(after)
But it always returns 0 rows.

hive pyspark date comparison

I am trying to translate a hiveQL query into pyspark. I am filtering on dates and getting different results and I would like to know how to get the behaviour in pySpark to match that of Hive. The hive query is:
SELECT COUNT(zip_cd) FROM table WHERE dt >= '2012-01-01';
In pySpark I am entering into the interpreter:
import pyspark.sql.functions as psf
import datetime as dt
hc = HiveContext(sc)
table_df = hc.table('table')
DateFrom = dt.datetime.strptime('2012-01-01', '%Y-%m-%d')
table_df.filter(psf.trim(table.dt) >= DateFrom).count()
I am getting similar, but not the same, results in the two counts. Does anyone know what is going on here?

Your code first creates datetime object from date 2012-01-01. Then during filtering the object is replaced with it's string representation (2012-01-01 00:00:00) and dates are compared using lexicographically which filters out 2012-01-01:
>>> '2012-01-01' >= '2012-01-01 00:00:00'
False
>>> '2012-01-02' >= '2012-01-01 00:00:00'
True
To achieve the same result as SQL just remove code with strptime and compare dates using strings.

Elasticsearch sort by date only of datetime

I have a field 'datetime' with value like '2014-07-17T01:20:20', on my query I want to sort it by just date like (2014-07-17). Is it possible to sort this way in elasticsearch?
I want to have something like this in database query:
order by DATE(datetime) DESC, name ASC
Thanks!

Filtering records on date in d3

I am trying to filter records on the basis of date. However, I am unable to do so. Following is the code:
data2 = data1.filter(function(row) { console.log(row["wsrweek"],parseDate("05-30-2014"));
return row["wsrweek"] == parseDate("05-30-2014");})
log output:
Date 2014-05-29T18:30:00.000Z Date 2014-05-29T18:30:00.000Z utilization.js:122
Date 2014-05-22T18:30:00.000Z Date 2014-05-29T18:30:00.000Z utilization.js:122
Date 2014-05-15T18:30:00.000Z Date 2014-05-29T18:30:00.000Z utilization.js:122
Date 2014-05-08T18:30:00.000Z Date 2014-05-29T18:30:00.000Z utilization.js:122
First row is have equal records. However, its not getting returned as data2 is empty. Kindly suggest

linq to entities - how to query dates that are stored as char?

In the db table the dates are stored in a char(8) field in this format yyyyMMdd.
How do I query for a date range?
I tried the following and it does not work:
context.Where(p=> Convert.ToDateTime(p.Date) >= Convert.ToDateTime('20120411');
context.Where(p=> Convert.ToInt32(p.Date) >= Convert.ToInt32('20120411');
context.Where(p=> int.Parse(p.Date) >= int.Parse('20120411');
From what I've read a possible way is to use EntityFunctions class but I'm not sure how to construct the query. Any ideas on how to do this?

Convert the test string to a date:
context.Where(p=> Convert.ToDateTime(p.Date) >= DateTime.Parse("2012/04/11 00:00:00");

context.Where(p=> p.Date.CompareTo("20120411") >= 0);

Resources