I made a visualization page using crossfilter.js and dc.js . I want to export the filtered dataset to excel. Is any way to do this.?
I think the best way to do this is to create another dimension and then call dimension.top(Infinity) to get all the records (sorted by that dimension's key).
Jacob Rideout created a pull request for a new method to do just this without the overhead, but it was not accepted (doesn't look like it was rejected either ;):
https://github.com/square/crossfilter/pull/95
But I doubt you will notice any performance penalty for creating the extra dimension. (Please comment on that PR if you do!)
function groupArrayAdd(keyfn) {
var bisect = d3.bisector(keyfn);
return function (elements, item) {
var pos = bisect.right(elements, keyfn(item));
elements.splice(pos, 0, item);
return elements;
};
}
function groupArrayRemove(keyfn) {
var bisect = d3.bisector(keyfn);
return function (elements, item) {
var pos = bisect.left(elements, keyfn(item));
if (keyfn(elements[pos]) === keyfn(item))
elements.splice(pos, 1);
return elements;
};
}
function groupArrayInit() {
return [];
}
var facts = crossfilter(data); //pass your mater dataset here.
var filteredRows = facts.groupAll().reduce(
groupArrayAdd(dc.pluck('shift')),
groupArrayRemove(dc.pluck('shift')),
groupArrayInit}
);
filteredRows.value() will give you the crossfilted data. Every time the data is filteded, this function will give automatically five the filted output which you can use to export to excel using any jquery plugin.
Another way to find out filtered data is using below dc function:
dimension.top(Infinity)
Related
I'm trying to create a bubble chart with dc.js that will have a bubble for each data row and will be filtered by other charts on the same page. The initial bubble chart is created correctly, but when items are filtered from another chart and added or removed from the group it looks like they are being applied to the wrong group. I'm not sure what I'm messing up on the grouping or dimensions. I've created an example fiddle here
There's simple pie chart to filter on filterColumn, a bubble chart that uses identifer1, a unique field, as the dimension and xVal, yVal, and rVal to display the data, and a dataTable to display the current records.
I've tried other custom groups functions, but switched to the example from the FAQ and still had problems.
var
filterPieChart=dc.pieChart("#filterPieChart"),
bubbleChart = dc.bubbleChart('#bubbleChart'),
dataTable = dc.dataTable('#data-table');
var
bubbleChartDim=ndx.dimension(dc.pluck("identifier1")),
filterPieChartDim=ndx.dimension(dc.pluck("filterColumn")),
allDim = ndx.dimension(function(d) {return d;});
var filterPieChartGroup=filterPieChartDim.group().reduceCount();
function reduceFieldsAdd(fields) {
return function(p, v) {
fields.forEach(function(f) {
p[f] += 1*v[f];
});
return p;
};
}
function reduceFieldsRemove(fields) {
return function(p, v) {
fields.forEach(function(f) {
p[f] -= 1*v[f];
});
return p;
};
}
function reduceFieldsInitial(fields) {
return function() {
var ret = {};
fields.forEach(function(f) {
ret[f] = 0;
});
return ret;
};
}
var fieldsToReduce=['xVal', 'yVal', 'rVal'];
var bubbleChartGroup = bubbleChartDim.group().reduce(
reduceFieldsAdd(fieldsToReduce),
reduceFieldsRemove(fieldsToReduce),
reduceFieldsInitial(fieldsToReduce)
);
filterPieChart
.dimension(filterPieChartDim)
.group(filterPieChartGroup)
...
;
bubbleChart
.dimension(bubbleChartDim)
.group(bubbleChartGroup)
.keyAccessor(function (p) { return p.value.xVal; })
.valueAccessor(function (p) { return p.value.yVal; })
.radiusValueAccessor(function (p) { return p.value.rVal; })
...
;
This was a frustrating one to debug. Your groups and reductions are fine, and that's the best way to plot one bubble for each row, using a unique identifier like that.
[It's annoying that you have to specify a complicated reduction, when the values will be either the original value or 0, but the alternatives aren't much better.]
The reductions are going crazy. Definitely not just original values and zero, some are going to other values, bigger or negative, and sometimes clicking a pie slice twice does not even return to the original state.
I put breakpoints in the reduce functions and noticed, as you did, that the values were being removed from the wrong groups. How could this be? Finally, by logging bubbleChartGroup.all() in a filtered handler for the pie chart, I noticed that the groups were out of order after the first rendering!
Your code is fine. But you've unearthed a new bug in dc.js, which I filed here.
In order to implement the sortBubbleSize feature, we sort the bubbles. Unfortunately we are also sorting crossfilter's internal array of groups, which it trusted us with. (group.all() returns an internal data structure which must never be modified.)
The fix will be easy; we just need to copy the array before sorting it. You can test it out in your code by commenting out sortBubbleSize and instead supplying the data function, which is what it does internally:
bubbleChart.data(function (group) {
var data = group.all().slice(0);
if (true) { // (_sortBubbleSize) {
// sort descending so smaller bubbles are on top
var radiusAccessor = bubbleChart.radiusValueAccessor();
data.sort(function (a, b) { return d3.descending(radiusAccessor(a), radiusAccessor(b)); });
}
return data;
});
Notice the .slice(0) at the top.
Hope to fix this in the next release, but this workaround is pretty solid in case it takes longer.
Here is a fiddle demonstrating the workaround.
I am trying to filter my row chart by selecting values in a dropdown.
I do this by creating a dimension and applying a filter to that dimension based on selected values in the dropdown.
The row chart has to display average values, to do this I created ReduceAdd/Remove/Initial functions.
The average values are working as intended.
However, when I filter my dimension used for the dropdown, it seems to add duplicate values to my row chart.
Example code: http://jsfiddle.net/sy9xA/4/
var dropDownFilterDimension;
function ReduceAdd(p, v) {
var value = v.value;
++p.countAvg;
p.totalAvg += value;
return p;
}
function ReduceRemove(p, v) {
var value = v.value
--p.countAvg;
p.totalAvg -= value;
return p;
}
function ReduceInitial() {
return {countAvg: 0, totalAvg: 0};
}
function filterDropdown(dropDownID){
dropDown = document.getElementById(dropDownID);
dropDownFilterDimension.filterAll();
values = $(dropDown).val();
if( values != null ){
dropDownFilterDimension.filter(function(d) { if (values.indexOf(d) > -1) {return d;} });
}
dc.redrawAll();
}
Here for example, when "banana" is selected for filtering. ReduceRemove is called for all non selected values, but in addition ReduceAdd is called for banana (even though it is already in it).
So now when after selecting banana, apple is selected. Banana still has some value.
Can someone explain why this is happening? Or how i can avoid this from happening?
Thanks in advance!
Robert
I don't have fully understand what you are doing also because the jsfddle doesn't show nothing more than the js.
I can tell how I do:
you have a dimension associated to a dc chart (ex chartDim)
you have a group that calculate the average associated to a dc chart (ex averageGroup)
you have a dropdown with a set of values to be use as filter (#myDropDown)
d3.select('#myDropDown')
.on('change', function(){
chartDim.filter(this.value)
dc.redrawAll();
})
the call of dc.redrawAll() will update not only the chart but also the averageGroup and everything should go fine. If the dropDown has is own dimension just filter this one instead of the chartDim
Please help to solve this very annoying problem. I am using a for loop to iterate over an array of data and create multiple grids. It is working well but the filter function is not binding properly (it only binds to the LAST grid created) Here is the code:
// this function iterates over the data to build the grids
function buildTables() {
// "domain" contains the dataset array
for (i = 0; i < domains.length; i++) {
var dataView;
dataView = new Slick.Data.DataView();
var d = domains[i];
grid = new Slick.Grid('#' + d.name, dataView, d.columns, grids.options);
var data = d.data;
// create a column filter collection for each grid - this works fine
var columnFilters = columnFilters[d.name];
// this seems to be working just fine
// Chrome console confirms it is is processed when rendering the filters
grid.onHeaderRowCellRendered.subscribe(function (e, args) {
$(args.node).empty();
$("<input type='text'>")
.data("columnId", args.column.id)
.val(columnFilters[args.column.id])
.appendTo(args.node);
});
// respond to changes in filter inputs
$(grid.getHeaderRow()).delegate(":input", "change keyup", function (e) {
var columnID = $(this).data("columnId");
if (columnID != null) {
// this works fine - when the user enters text into the input - it
// adds the filter term to the filter obj appropriately
// I have tested this extensively and it works appropriately on
// all grids (ie each grid has a distinct columnFilters object
var gridID = $(this).parents('.grid').attr('id');
columnFilters[gridID][columnID] = $.trim($(this).val());
dataView.refresh();
}
});
//##### FAIL #####
// this is where things seem to go wrong
// The row item always provides data from the LAST grid populated!!
// For example, if I have three grids, and I enter a filter term for
// grids 1 or 2 or 3 the row item below always belongs to grid 3!!
function filter(row) {
var gridID = $(this).parents('.grid').attr('id');
for (var columnId in grids.columnFilters[gridID]) {
if (columnId !== undefined && columnFilters[columnId] !== "") {
var header = grid.getColumns()[grid.getColumnIndex(columnId)];
//console.log(header.name);
}
}
return true;
}
grid.init();
dataView.beginUpdate();
dataView.setItems(data);
dataView.setFilter(filter); // does it matter than I only have one dataView instance?
dataView.endUpdate();
grid.invalidate();
grid.render();
In summary, each function seems to be binding appropriately to each grid except for the filter function. When I enter a filter term into ANY grid, it returns the rows from the last grid only.
I have spent several hours trying to find the fault but have to admit defeat. Any help would be most appreciated.
yes, it matters that you have only one instance of dataView. and also sooner or later you will come up to the fact that one variable for all grids is also a bad idea
so add a var dataView to your loop, it should solve the problem
edit See here for the non-working example of what I'm trying to do: http://bl.ocks.org/elsherbini/5814788
I am using dc.js to plot data collected from bee hives at my university. I am pushing new data to the graphs on every database change (using the magic of Meteor). When the database is over 5000 records or so, rerendering the lines gets really slow. So I want to use simplify.js to preprocess the lines before rendering. To see what I'm talking about, go to http://datacomb.meteor.com/. The page freezes after a couple of seconds, so be warned.
I have started to extend dc.js with a simpleLineChart, which would inherit from the existing dc.lineChart object/function. Here is what I have so far:
dc.simpleLineChart = function(parent, chartGroup) {
var _chart = dc.lineChart(),
_tolerance = 1,
_highQuality = false,
_helperDataArray;
_chart.tolerance = function (_) {
if (!arguments.length) return _tolerance;
_tolerance = _;
return _chart;
};
_chart.highQuality = function (_) {
if (!arguments.length) return _highQuality;
_highQuality = _;
return _chart;
};
return _chart.anchor(parent, chartGroup);
}
simplify.js takes in an array of data, a tolerance, and a boolean highQuality, and returns a new array with fewer elements based on it's simplification algorithm.
dc.js uses crossfilter.js. dc.js charts are associated with a particular crossfilter dimension and group. Eventually, it uses the data from someGroup().all() as the data to pass to a d3.svg.line(). I can't find where this is happening in the dc.js source, but this is where I need to intervene. I want to find this method, and override it in the dc.simpleLineChart object that I am making.
I was thinking something like
_chart.theMethodINeedToOverride = function(){
var helperDataArray = theChartGroup().all().map(function(d) { return {
x: _chart.keyAccessor()(d),
y: _chart.valueAccessor()(d)};})
var simplifiedData = simplify(helperDataArray, _tolerance, _highQuality)
g.datum(simplifiedData); // I know I'm binding some data at some point
// I'm just not sure to what or when
}
Can anyone help me either identify which method I need to override, or even better, show me how to do so?
dc.js source: https://github.com/NickQiZhu/dc.js/blob/master/dc.js
edit:
I think I may have found the function I need to override. The original function is
function createGrouping(stackedCssClass, group) {
var g = _chart.chartBodyG().select("g." + stackedCssClass);
if (g.empty())
g = _chart.chartBodyG().append("g").attr("class", stackedCssClass);
g.datum(group.all());
return g;
}
And I have tried to override it like so
function createGrouping(stackedCssClass, group) {
var g = _chart.chartBodyG().select("g." + stackedCssClass);
if (g.empty())
g = _chart.chartBodyG().append("g").attr("class", stackedCssClass);
var helperDataArray = group().all().map(function(d) { return {
x: _chart.keyAccessor()(d),
y: _chart.valueAccessor()(d)};})
var simplifiedData = simplify(helperDataArray, _tolerance, _highQuality)
g.datum(simplifiedData);
return g;
}
However, when I make a simpleLineChart, it is just a linechart with a tolerance() and highQuality() method. See here: http://bl.ocks.org/elsherbini/5814788
Well, I pretty much did what I set out to do.
http://bl.ocks.org/elsherbini/5814788
The key was to not only modify the createGrouping function, but also the lineY function in the code. (lineY gets set to tell the d3.svg.line() instance how to set the y value of a given point d)
I changed it to
var lineY = function(d, dataIndex, groupIndex) {
return _chart.y()(_chart.valueAccessor()(d));
};
The way lineY was written before, it was looking up the y value in an array, rather than using the data bound to the group element. This array had it's data set before i made my changes, so it was still using the old, pre-simplification data.
I have decent large data set of around 1100 records. This data set is mapped to an observable array which is then bound to a view. Since these records are updated frequently, the observable array is updated every time using the ko.mapping.fromJS helper.
This particular command takes around 40s to process all the rows. The user interface just locks for that period of time.
Here is the code -
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
ko.mapping.fromJS(data,transactionList)
Is there a workaround for this? Or should I just opt of web workers to improve performances?
Knockout.viewmodel is a replacement for knockout.mapping that is significantly faster at creating viewmodels for large object arrays like this. You should notice a significant performance increase.
http://coderenaissance.github.com/knockout.viewmodel/
I have also thought of a workaround as follows, this uses less amount of code-
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
// Instead of - ko.mapping.fromJS(data,transactionList)
var i = 0;
//clear the list completely first
transactionList.destroyAll();
//Set an interval of 0 and keep pushing the content to the list one by one.
var interval = setInterval(function () {if (i == data.length - 1 ) {
clearInterval(interval);}
transactionList.push(ko.mapping.fromJS(data[i++]));
}, 0);
I had the same problem with mapping plugin. Knockout team says that mapping plugin is not intended to work with large arrays. If you have to load such big data to the page then likely you have improper design of the system.
The best way to fix this is to use server pagination instead of loading all the data on page load. If you don't want to change design of your application there are some workarounds which maybe help you:
Map your array manually:
var data = storage.transactions();
var mappedData = ko.utils.arrayMap(data , function(item){
return ko.mapping.fromJS(item);
});
var transactionList = ko.observableArray(mappedData);
Map array asynchronously. I have written a function that processes array by portions in another thread and reports progress to the user:
function processArrayAsync(array, itemFunc, afterStepFunc, finishFunc) {
var itemsPerStep = 20;
var processor = new function () {
var self = this;
self.array = array;
self.processedCount = 0;
self.itemFunc = itemFunc;
self.afterStepFunc = afterStepFunc;
self.finishFunc = finishFunc;
self.step = function () {
var tillCount = Math.min(self.processedCount + itemsPerStep, self.array.length);
for (; self.processedCount < tillCount; self.processedCount++) {
self.itemFunc(self.array[self.processedCount], self.processedCount);
}
self.afterStepFunc(self.processedCount);
if (self.processedCount < self.array.length - 1)
setTimeout(self.step, 1);
else
self.finishFunc();
};
};
processor.step();
};
Your code:
var data = storage.transactions();
var transactionList = ko.observableArray([]);
processArrayAsync(data,
function (item) { // Step function
var transaction = ko.mapping.fromJS(item);
transactionList().push(transaction);
},
function (processedCount) {
var percent = Math.ceil(processedCount * 100 / data.length);
// Show progress to the user.
ShowMessage(percent);
},
function () { // Final function
// This function will fire when all data are mapped. Do some work (i.e. Apply bindings).
});
Also you can try alternative mapping library: knockout.wrap. It should be faster than mapping plugin.
I have chosen the second option.
Mapping is not magic. In most of the cases this simple recursive function can be sufficient:
function MyMapJS(a_what, a_path)
{
a_path = a_path || [];
if (a_what != null && a_what.constructor == Object)
{
var result = {};
for (var key in a_what)
result[key] = MyMapJS(a_what[key], a_path.concat(key));
return result;
}
if (a_what != null && a_what.constructor == Array)
{
var result = ko.observableArray();
for (var index in a_what)
result.push(MyMapJS(a_what[index], a_path.concat(index)));
return result;
}
// Write your condition here:
switch (a_path[a_path.length-1])
{
case 'mapThisProperty':
case 'andAlsoThisOne':
result = ko.observable(a_what);
break;
default:
result = a_what;
break;
}
return result;
}
The code above makes observables from the mapThisProperty and andAlsoThisOne properties at any level of the object hierarchy; other properties are left constant. You can express more complex conditions using a_path.length for the level (depth) the value is at, or using more elements of a_path. For example:
if (a_path.length >= 2
&& a_path[a_path.length-1] == 'mapThisProperty'
&& a_path[a_path.length-2] == 'insideThisProperty')
result = ko.observable(a_what);
You can use typeOf a_what in the condition, e.g. to make all strings observable.
You can ignore some properties, and insert new ones at certain levels.
Or, you can even omit a_path. Etc.
The advantages are:
Customizable (more easily than knockout.mapping).
Short enough to copy-paste it and write individual mappings for different objects if needed.
Smaller code, knockout.mapping-latest.js is not included into your page.
Should be faster as it does only what is absolutely necessary.