facing performance issues with knockout mapping plugin - performance

I have decent large data set of around 1100 records. This data set is mapped to an observable array which is then bound to a view. Since these records are updated frequently, the observable array is updated every time using the ko.mapping.fromJS helper.
This particular command takes around 40s to process all the rows. The user interface just locks for that period of time.
Here is the code -
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
ko.mapping.fromJS(data,transactionList)
Is there a workaround for this? Or should I just opt of web workers to improve performances?

Knockout.viewmodel is a replacement for knockout.mapping that is significantly faster at creating viewmodels for large object arrays like this. You should notice a significant performance increase.
http://coderenaissance.github.com/knockout.viewmodel/

I have also thought of a workaround as follows, this uses less amount of code-
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
// Instead of - ko.mapping.fromJS(data,transactionList)
var i = 0;
//clear the list completely first
transactionList.destroyAll();
//Set an interval of 0 and keep pushing the content to the list one by one.
var interval = setInterval(function () {if (i == data.length - 1 ) {
clearInterval(interval);}
transactionList.push(ko.mapping.fromJS(data[i++]));
}, 0);

I had the same problem with mapping plugin. Knockout team says that mapping plugin is not intended to work with large arrays. If you have to load such big data to the page then likely you have improper design of the system.
The best way to fix this is to use server pagination instead of loading all the data on page load. If you don't want to change design of your application there are some workarounds which maybe help you:
Map your array manually:
var data = storage.transactions();
var mappedData = ko.utils.arrayMap(data , function(item){
return ko.mapping.fromJS(item);
});
var transactionList = ko.observableArray(mappedData);
Map array asynchronously. I have written a function that processes array by portions in another thread and reports progress to the user:
function processArrayAsync(array, itemFunc, afterStepFunc, finishFunc) {
var itemsPerStep = 20;
var processor = new function () {
var self = this;
self.array = array;
self.processedCount = 0;
self.itemFunc = itemFunc;
self.afterStepFunc = afterStepFunc;
self.finishFunc = finishFunc;
self.step = function () {
var tillCount = Math.min(self.processedCount + itemsPerStep, self.array.length);
for (; self.processedCount < tillCount; self.processedCount++) {
self.itemFunc(self.array[self.processedCount], self.processedCount);
}
self.afterStepFunc(self.processedCount);
if (self.processedCount < self.array.length - 1)
setTimeout(self.step, 1);
else
self.finishFunc();
};
};
processor.step();
};
Your code:
var data = storage.transactions();
var transactionList = ko.observableArray([]);
processArrayAsync(data,
function (item) { // Step function
var transaction = ko.mapping.fromJS(item);
transactionList().push(transaction);
},
function (processedCount) {
var percent = Math.ceil(processedCount * 100 / data.length);
// Show progress to the user.
ShowMessage(percent);
},
function () { // Final function
// This function will fire when all data are mapped. Do some work (i.e. Apply bindings).
});
Also you can try alternative mapping library: knockout.wrap. It should be faster than mapping plugin.
I have chosen the second option.

Mapping is not magic. In most of the cases this simple recursive function can be sufficient:
function MyMapJS(a_what, a_path)
{
a_path = a_path || [];
if (a_what != null && a_what.constructor == Object)
{
var result = {};
for (var key in a_what)
result[key] = MyMapJS(a_what[key], a_path.concat(key));
return result;
}
if (a_what != null && a_what.constructor == Array)
{
var result = ko.observableArray();
for (var index in a_what)
result.push(MyMapJS(a_what[index], a_path.concat(index)));
return result;
}
// Write your condition here:
switch (a_path[a_path.length-1])
{
case 'mapThisProperty':
case 'andAlsoThisOne':
result = ko.observable(a_what);
break;
default:
result = a_what;
break;
}
return result;
}
The code above makes observables from the mapThisProperty and andAlsoThisOne properties at any level of the object hierarchy; other properties are left constant. You can express more complex conditions using a_path.length for the level (depth) the value is at, or using more elements of a_path. For example:
if (a_path.length >= 2
&& a_path[a_path.length-1] == 'mapThisProperty'
&& a_path[a_path.length-2] == 'insideThisProperty')
result = ko.observable(a_what);
You can use typeOf a_what in the condition, e.g. to make all strings observable.
You can ignore some properties, and insert new ones at certain levels.
Or, you can even omit a_path. Etc.
The advantages are:
Customizable (more easily than knockout.mapping).
Short enough to copy-paste it and write individual mappings for different objects if needed.
Smaller code, knockout.mapping-latest.js is not included into your page.
Should be faster as it does only what is absolutely necessary.

Related

Suitescript Saved Search Filter using other saved search results

I am trying to use the results of a specific saved search to try and filter another saved search in suitescript.
Basically, there is a button created on a project. Once the button is clicked, I need to go get all the tasks for that specific project and use each task to filter on a transaction saved search using a custom field and get whatever information is on that saved search.
This is what I have so far:
function runScript(context) {
var record = currentRecord.get();
var id = record.id;
var type = record.type;
var i = 0;
console.log(id);
var projectSearch = search.load({id: 'customsearch1532'})
var billableExpenseSearch = search.load({id: 'customsearch1533'})
var projectFilter = search.createFilter({
name:'internalId',
operator: search.Operator.IS,
values: id
});
projectSearch.filters.push(projectFilter);
var projectResults = projectSearch.run().getRange(0,1000);
while(i < projectResults.length){
var task = projectResults[i].getValue(projectSearch.columns[1]);
console.log(task);
var billableExpenseFilter = search.createFilter({
name:'custcol4',
operator: search.Operator.ANYOF,
values: task
});
billableExpenseSearch.filters.push(billableExpenseFilter);
var billableExpenseResults = billableExpenseSearch.run().getRange(0,1000);
console.log(billableExpenseResults.length);
for(var j = 0; j< billableExpenseResults.length; j++){
var testAmount = billableExpenseResults[j].getValue(billableExpenseSearch.columns[3]);
console.log(testAmount);
}
i++;
}
}
The log for the Task is correct. I have 2 tasks on the project I am trying this on but once we get to the second iteration, the billableExpenseSearch length is showing as 0, when it's supposed to be 1.
I am guessing that my logic is incorrect of the createFilter function doesn't accept changes once the filter is created.
Any help is appreciated!
EDIT:
var billableExpenseSearch = search.load({id: 'customsearch1533'});
var billableExpenseFilter = search.createFilter({
name:'custcol4',
operator: search.Operator.ANYOF,
values: task
});
billableExpenseSearch.filters.push(billableExpenseFilter);
var billableExpenseResults = billableExpenseSearch.run().getRange(0,1000);
console.log(billableExpenseResults.length);
for(var j = 0; j< billableExpenseResults.length; j++){
var taskid = billableExpenseResults[j].getValue(billableExpenseSearch.columns[0]);
console.log(taskid);
Thank you
I think your guess is correct your are keep pushing filters
billableExpenseSearch.filters.push(billableExpenseFilter);
After pushing the filter and extract the value you need to remove it before adding a new one, you can do this by pop() the last one:
billableExpenseSearch.filters.pop();
Note: You can fix this by re-loading the search every time before pushing the filter. this will reset your filters, but I do NOT recommend that since loading a search will consume more USAGE and might receive USAGE_LIMIT_EXCEEDED ERROR.
I also recommend the following:
1- Get all task ids before doing the second search, once you do that you only need to search once. Because if you have many records you might encounter USAGE_LIMIT_EXCEEDED ERROR. Since you work with a client or Suitelet script you only have 1000 USAGE.
Edit: Sample might help you.
var ids = [];
var pagedData = projectSearch.runPaged({pageSize : 1000});
// iterate the pages
for( var i=0; i < pagedData.pageRanges.length; i++ ) {
// fetch the current page data
var currentPage = pagedData.fetch(i);
// and forEach() thru all results
currentPage.data.forEach( function(result) {
// you have the result row. use it like this....
var id = result.getValue(projectSearch.columns[1]);
Ids.push(id);
});
}
Note: This search will extract all records not only first 1000.
After that add the array to the Filter
var billableExpenseFilter = search.createFilter({
name:'custcol4',
operator: search.Operator.ANYOF,
values: [ids]
});
2- Don't Use search.load use search.create it will make your script more readable and easier to maintain in the future.

unique Count of values in dimension using crossfilter (dc.js) [duplicate]

I am stuck at a unique problem involving dc.js and crossfilter. I have some data which i need to display using Number Charts powered by dc.js. However i found minimal documentation for the number charts and hence posting my query.
Here is the JSFiddle for what i have conceptualized so far.
I basically want to show the unique project count in box 1 which in this case would be 3, unique place count in box 2 which in this case would be 11 and the screen failure rate which would be 2/15*100 i.e. 15.3%
Currently i have made this working using jquery but thats just a hack. I would like to have these number charts based on cross table aggregation so that i can drill down on the data.
I have come across examples for reductions to calculate counts but they were for bar charts but in the number chart we need to have a value accessor for displaying data.
Can someone help me out please?
PS:
Here is the jquery code that i wrote. Dont know if this would be helpful.
$(document).ready(function() {
var baseURL = window.location.origin;
$.ajax({
url : baseURL + '/api/PlaceTable',
type : 'GET',
data : {},
async : true,
dataType : "json",
success : function(response) {
//Project Count
var projectIdCount = [];
for (i = 0; i < response.length; i++) {
if(response[i].Project != undefined){
if($.inArray(response[i].Project, projectIdCount) === -1){
projectIdCount.push(response[i].Project);
}
}
}
$('#number-box1').text(ProjectIdCount.length);
//Place Count
var placeIdCount = [];
for (i = 0; i < response.length; i++) {
if(response[i].Place != undefined){
if($.inArray(response[i].Place, placeIdCount) === -1){
placeIdCount.push(response[i].Place);
}
}
}
And for displaying a running sum of a column containing binary values i used this code, which worked in the number chart:
numberChart
.valueAccessor(function(x){ return +flag.groupAll().reduceCount().reduceSum(function(d) { return d.Flag; }).value();})
.group(ndx.groupAll());
The failure percentage calculation is a separate problem which I think you've asked elsewhere. To get the unique count, it is pretty easy to make a "fake groupAll" which returns the number of unique keys in its value method.
We'll also need to filter out the empty bins since crossfilter doesn't do that automatically.
function bin_counter(group) {
return {
value: function() {
return group.all().filter(function(kv) {
return kv.value > 0;
}).length;
}
};
}
var projectGroup = project.group();
projectCount
.valueAccessor(function(x){ return x;})
.group(bin_counter(projectGroup));
Updated fiddle here, still ignoring the failure% part:
http://jsfiddle.net/gordonwoodhull/vct0dzou/1/

How to get a loop to load movieclips with eventlisteners

I wanted the scene load 5 different movie clips (named B1-B5). Each movie clip is placed on a specific x and y. Each movie clip grows/shrinks on roll over/roll out....
I got the code working by typing everything out and duplicating each section per time but it's messy and I'd like to clean up the code by getting a loop to do it (if it's possible?).
This is the code that works but I'd have to duplicate it per movie clip (changing the obvious bits)...
var scene1:MovieClip = new B1();
addChild(scene1);
scene1.x = 170.30;
scene1.y = 231.15;
scene1.addEventListener(MouseEvent.MOUSE_OVER, onRollOverEvent1);
scene1.addEventListener(MouseEvent.MOUSE_OUT, onRollOutEvent1);
function onRollOverEvent1(e:MouseEvent) {
scene1.width=25.9;
scene1.height=25;
}
function onRollOutEvent1(e:MouseEvent) {
scene1.width = 20.9;
scene1.height = 20;
}
Below is what I've tried out but have been stuck for a good while...
for (var i:int=1; i<5; i++){
var scene[i]:MovieClip = new "B"+i();
addChild("scene"+i);
//var scene[i]:MovieClip = new B[i]();
scene[i].addEventListener(MouseEvent.MOUSE_OVER, onRollOverEvent);
scene[i].addEventListener(MouseEvent.MOUSE_OUT, onRollOutEvent)
function onRollOverEvent(e:MouseEvent) {
scene[i].width=25.9;
scene[i].height=25;
}
function onRollOutEvent(e:MouseEvent) {
scene[i].width = 20.9;
scene[i].height = 20;
}
}
scene1.x = 170.30;
scene1.y = 231.15;
scene2.x = 284.30;
scene2.y = 250.75;
scene3.x = 377.30;
scene3.y = 280.15;
scene4.x = 444.30;
scene4.y = 321.15;
scene5.x = 196.30;
scene5.y = 172.15;
First, lets go through your mistakes.
new "B"+i();
At the very best that translates to calling a number i as function and adding the result to "B" as a String. But even new "B1"() is not the same as new B1(). There is, in fact, a method getDefinitionByName(..) that allows to address a class via its name, but I don't recommend to use it because it is advanced topic.
var scene[i]:MovieClip
You just cannot define variables scene1, scene2, etc this way. The closest thing you can actually devise is the square bracket notation: this["scene" + i] = ....
addChild("scene"+i);
The argument must be a DisplayObject instance, not a String.
for (...)
{
...
function onRollOverEvent(e:MouseEvent)
...
}
Do not define functions inside other functions or loops.
scene[i].width = 20.9;
scene[i].height = 20;
By the end of your loop i will be equal to 5, so, what do you think such a record will address?
Then, the solution.
When you come to scaling your working solution to multiple instances, you are to go algorithmic. Loops and Arrays are your friends.
// Lets devise a list of classes and (x,y) coordinates.
var Designs:Array = [
null, // the 0-th element
{id:B1, x:170, y:230},
{id:B2, x:285, y:250},
];
for (var i:int = 1; i < Design.length; i++)
{
// Retrieve a record for the future object.
var aDesign:Object = Designs[i];
// Get a reference to the object's class.
var aClass:Class = aDesign.id;
// Create the object. Yes, you CAN omit () with
// the "new" operator if there are no mandatory arguments.
var aThing:Movieclip = new aClass;
// Set coordinates from the design record.
aThing.x = aDesign.x;
aThing.y = aDesign.y;
// Add to the display list.
addChild(aThing);
// Subscribe the event handlers.
aThing.addEventListener(MouseEvent.MOUSE_OVER, onOver);
aThing.addEventListener(MouseEvent.MOUSE_OUT, onOut);
// Save the object's reference for the later use.
// If you'd need to address, say, 3rd object,
// you do it as following:
// Designs[3].instance
aDesign.instance = aThing;
}
function onOver(e:MouseEvent):void
{
// You subscribed all of the objects to this one event handler.
// This is the correct way to learn, which one of the objects
// is under the mouse and is dispatching the said event.
var aThing:MovieClip = e.currentTarget as MovieClip;
// Change the object's size.
aThing.width = 26;
aThing.height = 25;
}
function onOut(e:MouseEvent):void
{
// Get the source of the dispatched event.
var aThing:MovieClip = e.currentTarget as MovieClip;
// Change the object's size.
aThing.width = 21;
aThing.height = 20;
}

Export crossfilter dataset to excel in dc.js

I made a visualization page using crossfilter.js and dc.js . I want to export the filtered dataset to excel. Is any way to do this.?
I think the best way to do this is to create another dimension and then call dimension.top(Infinity) to get all the records (sorted by that dimension's key).
Jacob Rideout created a pull request for a new method to do just this without the overhead, but it was not accepted (doesn't look like it was rejected either ;):
https://github.com/square/crossfilter/pull/95
But I doubt you will notice any performance penalty for creating the extra dimension. (Please comment on that PR if you do!)
function groupArrayAdd(keyfn) {
var bisect = d3.bisector(keyfn);
return function (elements, item) {
var pos = bisect.right(elements, keyfn(item));
elements.splice(pos, 0, item);
return elements;
};
}
function groupArrayRemove(keyfn) {
var bisect = d3.bisector(keyfn);
return function (elements, item) {
var pos = bisect.left(elements, keyfn(item));
if (keyfn(elements[pos]) === keyfn(item))
elements.splice(pos, 1);
return elements;
};
}
function groupArrayInit() {
return [];
}
var facts = crossfilter(data); //pass your mater dataset here.
var filteredRows = facts.groupAll().reduce(
groupArrayAdd(dc.pluck('shift')),
groupArrayRemove(dc.pluck('shift')),
groupArrayInit}
);
filteredRows.value() will give you the crossfilted data. Every time the data is filteded, this function will give automatically five the filted output which you can use to export to excel using any jquery plugin.
Another way to find out filtered data is using below dc function:
dimension.top(Infinity)

Assigning functions to multiple slickgrids

Please help to solve this very annoying problem. I am using a for loop to iterate over an array of data and create multiple grids. It is working well but the filter function is not binding properly (it only binds to the LAST grid created) Here is the code:
// this function iterates over the data to build the grids
function buildTables() {
// "domain" contains the dataset array
for (i = 0; i < domains.length; i++) {
var dataView;
dataView = new Slick.Data.DataView();
var d = domains[i];
grid = new Slick.Grid('#' + d.name, dataView, d.columns, grids.options);
var data = d.data;
// create a column filter collection for each grid - this works fine
var columnFilters = columnFilters[d.name];
// this seems to be working just fine
// Chrome console confirms it is is processed when rendering the filters
grid.onHeaderRowCellRendered.subscribe(function (e, args) {
$(args.node).empty();
$("<input type='text'>")
.data("columnId", args.column.id)
.val(columnFilters[args.column.id])
.appendTo(args.node);
});
// respond to changes in filter inputs
$(grid.getHeaderRow()).delegate(":input", "change keyup", function (e) {
var columnID = $(this).data("columnId");
if (columnID != null) {
// this works fine - when the user enters text into the input - it
// adds the filter term to the filter obj appropriately
// I have tested this extensively and it works appropriately on
// all grids (ie each grid has a distinct columnFilters object
var gridID = $(this).parents('.grid').attr('id');
columnFilters[gridID][columnID] = $.trim($(this).val());
dataView.refresh();
}
});
//##### FAIL #####
// this is where things seem to go wrong
// The row item always provides data from the LAST grid populated!!
// For example, if I have three grids, and I enter a filter term for
// grids 1 or 2 or 3 the row item below always belongs to grid 3!!
function filter(row) {
var gridID = $(this).parents('.grid').attr('id');
for (var columnId in grids.columnFilters[gridID]) {
if (columnId !== undefined && columnFilters[columnId] !== "") {
var header = grid.getColumns()[grid.getColumnIndex(columnId)];
//console.log(header.name);
}
}
return true;
}
grid.init();
dataView.beginUpdate();
dataView.setItems(data);
dataView.setFilter(filter); // does it matter than I only have one dataView instance?
dataView.endUpdate();
grid.invalidate();
grid.render();
In summary, each function seems to be binding appropriately to each grid except for the filter function. When I enter a filter term into ANY grid, it returns the rows from the last grid only.
I have spent several hours trying to find the fault but have to admit defeat. Any help would be most appreciated.
yes, it matters that you have only one instance of dataView. and also sooner or later you will come up to the fact that one variable for all grids is also a bad idea
so add a var dataView to your loop, it should solve the problem

Resources