I have dataset with past Event Time
01-12-2015 01:10:10
01-12-2015 01:10:20
01-12-2015 01:10:30
01-12-2015 01:10:40
.... (millions of records)
I want to apply timeWindow for this timeWindow(Time.seconds(30))
I can have a TimeExtractor class to get the EventTime in the data. But how do I implement getCurrentWatermark method. It should get the past date and time
In your case it would be best to use one of the provided TimeStampAssigners see here.
So what I would recommend is something like this:
DataStream<MyEvent> stream = ...
DataStream<MyEvent> withTimestampsAndWatermarks =
stream.assignTimestampsAndWatermarks(new AscendingTimestampExtractor<MyEvent>() {
#Override
public long extractAscendingTimestamp(MyEvent element) {
return element.getCreationTime();
}
});
Also remember to set proper TimestampCharacteristic:
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
Related
I am trying to see if there is a way to improve the way data is inserted and updated.
I am using ORACLE DB with JDBC.
The current way i'm doing is to update (e.g.)customer record by using a FOR loop after checking if toUpdate is true . An Example such as the sample code below, followed by calling an existing DAO update() to do so. But this would not allow for the UPSERT of multiple data together.
However, is there a better way to UPSERT multiple data together?
if (toUpdate) {
for (Customer customerRec : customerRecList)
customerRecDAO.update(customerRec);
}
Yes you can use batching:
public <T> int saveInBatch(List<T> records, String sql, Function<T, MapSqlParameterSource> paramFn){
try{
MapSqlParameterSource[] params = records.stream().map(paramFn).toArray(MapSqlParameterSource[]::new);
int rowCount = jdbcTemplate.batchUpdate(sql, params);
return Arrays.stream(rowCount).sum();
}
catch(Exception e){
//exception handling
}
}
paramFn is a lambda of function such that you can map records to their values. example could be
(record)->{
return new MapSqlParameterSource("username" ,username),Integer.class);//just example
}
why we use MapSqlParameterSource
You can call saveInBatch in such a way that you pass smaller batches or customized batches of records. Suppose you have a million records then you may want to update only 200-400 records at a time so you can do something like below:
private <T> int saveRecords(List<T> records, String sql, Function<T, MapSqlParameterSource> paramFn) throws Exception{
return Lists.partition(records, 300).stream().map(batch-> saveInBatch(batch, sql, paramFn)).mapToInt(Integer::intValue).sum();
}
Note: above is not well optimized or streams are not used to their best but this is a working code I tried ages back :).
I red a lot about sorting a CellTable. I also went trough the ColumnSorting with AsyncDataProvider. But my CellTable does not sort.
Here is my code:
public class EventTable extends CellTable<Event> {
public EventTable() {
EventsDataProvider dataProvider = new EventsDataProvider(this);
dataProvider.addDataDisplay(this);
SimplePager.Resources pagerResources = GWT.create(SimplePager.Resources.class);
SimplePager pager = new SimplePager(TextLocation.CENTER, pagerResources, false, 5, true);
pager.setDisplay(this);
[...]
TextColumn<Event> nameCol = new TextColumn<Event>() {
#Override
public String getValue(Event event) {
return event.getName();
}
};
nameCol.setSortable(true);
AsyncHandler columnSortHandler = new AsyncHandler(this);
addColumnSortHandler(columnSortHandler);
addColumn(nameCol, "Name");
getColumnSortList().push(endCol);
}
}
public class EventsDataProvider extends AsyncDataProvider<Event> {
private final EventTable eventTable;
public EventsDataProvider(EventTable eventTable) {
this.eventTable = eventTable;
}
#Override
protected void onRangeChanged(HasData<Event> display) {
int start = display.getVisibleRange().getStart();
int length = display.getVisibleRange().getLength();
// check false values
if (start < 0 || length < 0) return;
// check Cache before making a rpc
if (pageCached(start, length)) return;
// get Events async
getEvents(start, length);
}
}
I do now know, if all the methods are need here. If so, I will add them. But in short:
pageCached calls a method in my PageCache Class which holds a map and a list. Before making a rpc call, the cache is checked if the events where already taken and then displayed.
getEvents just makes an rpc call via asynccallback which updates the rowdata via updateRowData() on success.
My Table is displayed fast with currently around 500 entries (could be more, depends on the customer). No missing data and the paging works fine.
I just cannot get the sorting work. As far as I know, AsyncHandler will fire a setVisibleRangeAndClearData() and then an onRangeChanged(). onRangeChanged is never fired. As for the setVisibleRangeAndClearData() I do not know. But the sortindicator (arrow next to the columnname) does change on every click.
I do not want to let the server sort the list. I have my own Comparators. It is enough, if the current visible page of the table is sorted. I do now want to sort the whole list.
Edit:
I changed following code in the EventTable constructor:
public EventTable() {
[...]
addColumnSortHandler(new ColumnSortEvent.AsyncHandler(this) {
public void onColumnSort(ColumnSortEvent event) {
super.onColumnSort(event);
MyTextColumn<Event> myTextColumn;
if (event.getColumn() instanceof MyTextColumn) {
// Compiler Warning here: Safetytype unchecked cast
myTextColumn = (MyTextColumn<Event>) event.getColumn();
MyLogger.log(this.getClass().getName(), "asc " + event.isSortAscending() + " " + myTextColumn.getName(), Level.INFO);
}
List<Event> list = dataProvider.getCurrentEventList();
if (list == null) return;
if (event.isSortAscending()) Collections.sort(list, EventsComparator.getComparator(EventsComparator.NAME_SORT));
else Collections.sort(list, EventsComparator.descending(EventsComparator.getComparator(EventsComparator.NAME_SORT)));
}
});
addColumn(nameCol, "Name");
getColumnSortList().push(endCol);
}
I had to write my own TextColumn to determine the Name of the column. Otherwise how should I know, which column was clicked? The page gets sorted now but I have to click twice on the column. After then, the sorting is done with every click but in the wrong order.
This solution does need polishing and it seems kinda hacky to me. Any better ideas?
The tutorial, that you linked to, states:
This sorting code is here so the example works. In practice, you would
sort on the server.
Async provider is used to display data that is too big to be loaded in a single call. When a user clicks on any column to sort it, there is simply not enough objects on the client side to display "first 20 evens by name" or whatever sorting was applied. You have to go back to your server and request these first 20 events sorted by name in ascending order. And when a user reverses sorting, you have to go to the server again to get first 20 events sorted by name in a descending order, etc.
If you can load all data in a single call, then you can use regular DataProvider, and all sorting can happen on the client side.
EDIT:
The problem in the posted code was in the constructor of EventsDataProvider. Now it calls onRangeChanged, and the app can load a new sorted list of events from the server.
I'm using a SmartGWT ListGrid with a DataSource. I'm successfully using a CellFormatter to display numeric file size data as mixed text / data (i.e. "10 GB" rather than 10737418240). I have filtering set up.
What I'd like to do is to let the user filter on the CellFormatter output, rather than on the underlying data. IOW, let the user type "GB" into the filter box, and get all the files with sizes in the GB range. The DataSource is cached locally, so I don't have issues about going back to the server to get data.
Edit: the reason why I'm using a CellFormatter is because it want sorting to be correct, IOW when sorting in increasing order I want 200 KB to come before 10 GB, not after (and in a text sort they're reversed). Sorting is more important to me than filtering, so if I have to have both sorting and filtering target the same representation of the data, I'll just give up on filtering working.
Any help would be greatly appreciated. Thank you,
Greg
You have two options to do this. First is to return already modified values from your datasource, so instead of 10737418240 it should return "10 GB" string value.
The second approach seems better for me - you should use SimpleType functionality. There is an example for you:
public class PopulationType extends SimpleType {
public PopulationType() {
super("population", FieldType.TEXT);
// format values in the grid
this.setSimpleTypeValueExtractor(new SimpleTypeValueExtractor() {
#Override
public Object getAtomicValue(Object value) {
if (value instanceof Integer && ((Integer) value) > 1000000) {
return ((Integer) value) / 1000000 + " Mln";
}
return "" + value;
}
});
}
}
public void onModuleLoad() {
final ListGrid countryGrid = new ListGrid();
countryGrid.setWidth100();
countryGrid.setHeight100();
countryGrid.setAutoFetchData(true);
countryGrid.setShowFilterEditor(true);
countryGrid.setShowAllRecords(true);
WorldXmlDS ds = WorldXmlDS.getInstance();
ds.getField("population").setType(new PopulationType());
countryGrid.setDataSource(ds);
countryGrid.draw();
}
You set your SimpleType instance to a field you want to format and set SimpleTypeValueExtractor to override getAtomicValue which is used for showing,filtering,sorting.
There are other methods you could override - e.g. if you need to edit values in your grid you should probably set SimpleTypeValueUpdater as well.
guys!
I have a small issue with LINQ (Im total beginer in this topic). Maybe it is some desing mistake, but let you decide it.
I'm coding a Windows Store App, which is kind a calendar. It has a Day object. Because of the semantic zoom (and some groupping hack), I put this Day into a wrapper class, named as Month.
After loading all data, and after getting the current data, I want to extract from this structure the current Day object.
Here is the important code:
public class Day
{
public int nr { get; set; }
...
}
public class Month
{
public string Title {get;set;}
public List<Day> Days{get;set;}
}
Later I have this:
List<Month> Months;
It is correctly filled with lists of days. Now comes the tricky part:
Day Today = Months.Find( ??? )
I had some idea, but none of them was statisfying...
So, the question is:
How can I select an item from a multiple list hierarchy in LINQ?
(List<List<Day>>, and one condition must met in each list (Day.nr and Month.nr))
Create an Enum for every month:
public enum NamesOfMonths
{
January = 1,
February = 2,
// so on and so forth
}
Now, you can use it to find the correct Month, and eventually the correct Day.
var dayToday = DateTime.Now.Day;
var monthToday = DateTime.Now.Month;
Day Today = Months.Find(m => m.Title.Equals(((NamesOfMonths)monthToday).ToString()))
.Days.Where(d => d.Nr == dayToday).FirstOrDefault();
I think you're looking for SelectMany:
var days = months.SelectMany(m => m.Days); // Gets all the days in those months
var today = days.Where(d => /* some condition goes here */);
I'm using Telerik OpenAccess and SQL Server on a project and I need to be able to search by what someone's age will be on a certain date. The problem that I am running into is that the person's date of birth is stored in one table and the date to compare to is in another table, which prevents me from using a computed column. They are, however, joined together so that I can calculate the age by creating my own non-persistent property in the partial class like so:
public partial class Student
{
[Telerik.OpenAccess.Transient]
private int? _ageUponArrival;
public virtual int? AgeUponArrival
{
get
{
try
{
var dob = DateTime.Parse(this.StudentProfiles.First().Person.YearOfBirth);
var programStart = (DateTime)(this.StudentPrograms.First().ProgramStart);
this._ageUponArrival = programStart.Year - dob.Year;
if (dob > programStart.AddYears(-(int)(this._ageUponArrival)))
{
(this._ageUponArrival)--;
}
}
catch (Exception e)
{
this._ageUponArrival = null;
}
return _ageUponArrival;
}
set { }
}
}
Please ignore how bad the tables are set up, it's something that I inherited and can't change at this point. The problem with this approach is that the property is not available to search on with Linq. I know that I could create a view that would do this for me, but I would much rather not have to maintain a view just for this. Is there any way at all to create a calculated property through Telerik that would be calculated on the db server in such a way as to be searchable?
It appears that this is not possible at this point. http://www.telerik.com/community/forums/orm/linq-questions/dynamic-query-with-extended-field.aspx