Extending dc.js to add a "simpleLineChart" chart - d3.js

edit See here for the non-working example of what I'm trying to do: http://bl.ocks.org/elsherbini/5814788
I am using dc.js to plot data collected from bee hives at my university. I am pushing new data to the graphs on every database change (using the magic of Meteor). When the database is over 5000 records or so, rerendering the lines gets really slow. So I want to use simplify.js to preprocess the lines before rendering. To see what I'm talking about, go to http://datacomb.meteor.com/. The page freezes after a couple of seconds, so be warned.
I have started to extend dc.js with a simpleLineChart, which would inherit from the existing dc.lineChart object/function. Here is what I have so far:
dc.simpleLineChart = function(parent, chartGroup) {
var _chart = dc.lineChart(),
_tolerance = 1,
_highQuality = false,
_helperDataArray;
_chart.tolerance = function (_) {
if (!arguments.length) return _tolerance;
_tolerance = _;
return _chart;
};
_chart.highQuality = function (_) {
if (!arguments.length) return _highQuality;
_highQuality = _;
return _chart;
};
return _chart.anchor(parent, chartGroup);
}
simplify.js takes in an array of data, a tolerance, and a boolean highQuality, and returns a new array with fewer elements based on it's simplification algorithm.
dc.js uses crossfilter.js. dc.js charts are associated with a particular crossfilter dimension and group. Eventually, it uses the data from someGroup().all() as the data to pass to a d3.svg.line(). I can't find where this is happening in the dc.js source, but this is where I need to intervene. I want to find this method, and override it in the dc.simpleLineChart object that I am making.
I was thinking something like
_chart.theMethodINeedToOverride = function(){
var helperDataArray = theChartGroup().all().map(function(d) { return {
x: _chart.keyAccessor()(d),
y: _chart.valueAccessor()(d)};})
var simplifiedData = simplify(helperDataArray, _tolerance, _highQuality)
g.datum(simplifiedData); // I know I'm binding some data at some point
// I'm just not sure to what or when
}
Can anyone help me either identify which method I need to override, or even better, show me how to do so?
dc.js source: https://github.com/NickQiZhu/dc.js/blob/master/dc.js
edit:
I think I may have found the function I need to override. The original function is
function createGrouping(stackedCssClass, group) {
var g = _chart.chartBodyG().select("g." + stackedCssClass);
if (g.empty())
g = _chart.chartBodyG().append("g").attr("class", stackedCssClass);
g.datum(group.all());
return g;
}
And I have tried to override it like so
function createGrouping(stackedCssClass, group) {
var g = _chart.chartBodyG().select("g." + stackedCssClass);
if (g.empty())
g = _chart.chartBodyG().append("g").attr("class", stackedCssClass);
var helperDataArray = group().all().map(function(d) { return {
x: _chart.keyAccessor()(d),
y: _chart.valueAccessor()(d)};})
var simplifiedData = simplify(helperDataArray, _tolerance, _highQuality)
g.datum(simplifiedData);
return g;
}
However, when I make a simpleLineChart, it is just a linechart with a tolerance() and highQuality() method. See here: http://bl.ocks.org/elsherbini/5814788

Well, I pretty much did what I set out to do.
http://bl.ocks.org/elsherbini/5814788
The key was to not only modify the createGrouping function, but also the lineY function in the code. (lineY gets set to tell the d3.svg.line() instance how to set the y value of a given point d)
I changed it to
var lineY = function(d, dataIndex, groupIndex) {
return _chart.y()(_chart.valueAccessor()(d));
};
The way lineY was written before, it was looking up the y value in an array, rather than using the data bound to the group element. This array had it's data set before i made my changes, so it was still using the old, pre-simplification data.

Related

AmCharts AmMap - Set starting location for zoom actions

I would like to use the "zoomToMapObject" method based on a selection on a dropdown menu.
For some reason the start zoom location is the middle of the map and not the set the geoPoint.
(The zooming works but the start location make it look a bit weird.)
My current approach looks like this:
const duration = this.chart.zoomToMapObject(selectedPoloygon, this.countryZoom, true).duration;
setTimeout(() => {
this.chart.homeGeoPoint = geoPoint;
this.chart.homeZoomLevel = this.countryZoom;
}, duration);
this.handleCountrySelection(selectedPoloygon);
Somehow even setting the homeGeoPoint / homeZoomLevel doesn't affect next zoom actions.
**UPDATE: Workaround heavy cost (from 1300 nodes to over 9000) **
I examined the problem a step further. It seems the middle point gets set when I push a new mapImageSeries into the map.
My workarround currently is to draw all points on the map and hide them.
Then after I select a country I change the state to visible.
However this approach is very costly. The DOM-Nodes rises from 1300 to ~ 9100.
My other approach with creating them after a country has been selected AND the zoom animation finished was much more
effective. But due to the map starting every time for a center location it is not viable? Or did I do s.th. wrong?
Here is my current code which is not performant:
// map.ts
export class MapComponent implements AfterViewInit, OnDestroy {
imageSeriesMap = {};
// ... standard map initialization ( not in zone of course )
// creating the "MapImages" which is very costly
this.dataService.getCountries().forEach(country => {
const imageSeriesKey = country.id;
const imageSeriesVal = chart.series.push(new am4maps.MapImageSeries()); // takes arround 1-2 ms -> 300 x 2 ~ 500 ms.
const addressForCountry = this.dataService.filterAddressToCountry(country.id); // returns "DE" or "FR" for example.
const imageSeriesTemplate = imageSeriesVal.mapImages.template;
const circle = imageSeriesTemplate.createChild(am4core.Circle);
circle.radius = 4;
circle.fill = am4core.color(this.colorRed);
circle.stroke = am4core.color('#FFFFFF');
circle.strokeWidth = 2;
circle.nonScaling = true;
circle.tooltipText = '{title}';
imageSeriesTemplate.propertyFields.latitude = 'latitude';
imageSeriesTemplate.propertyFields.longitude = 'longitude';
imageSeriesVal.data = addressForCountry.map(address => {
return {
latitude: Number.parseFloat(address.lat),
longitude: Number.parseFloat(address.long),
title: address.company
};
});
imageSeriesVal.visible = false;
this.imageSeriesMap[imageSeriesKey] = imageSeriesVal;
});
// clicking on the map
onSelect(country) {
this.imageSeriesMap[country].visible = true;
setTimeout( () => {
const chartPolygons = <any>this.chart.series.values[0];
const polygon = chartPolygons.getPolygonById(country);
const anim = this.chart.zoomToMapObject(polygon, 1, true, 1000);
anim.events.on('animationended', () => {});
this.handleCountrySelection(polygon);
}, 100);
});
}
handleCountrySelection(polygon: am4maps.MapPolygon) {
if (this.selectedPolygon && this.selectedPolygon !== polygon) {
this.selectedPolygon.isActive = false;
}
polygon.isActive = true;
const geoPoint: IGeoPoint = {
latitude: polygon.latitude,
longitude: polygon.longitude
};
this.chart.homeGeoPoint = geoPoint;
this.chart.homeZoomLevel = this.countryZoom;
this.selectedPolygon = polygon;
}
}
Thanks to your thorough followup I was able to replicate the issue. The problem you were having is triggered by any one of these steps:
dynamically pushing a MapImageSeries to the chart
dynamically creating a MapImage via data (also please note in the pastebind you provided, data expects an array, I had to change that while testing)
In either step, the chart will fully zoom out as if resetting itself. I'm going to look into why this is happening and if it can be changed, so in the meantime let's see if the workaround below will work for you.
If we only use a single MapImageSeries set in advance (I don't particularly see a reason to have multiple MapImageSeries, would one not do?), that eliminates problem 1 from occurring. Asides from data, we can create() MapImages manually via mapImageSeries.mapImages.create(); then assign their latitude and longitude properties manually, too. With that, problem 2 does not occur either, and we seem to be good.
Here's a demo with a modified version of the pastebin:
https://codepen.io/team/amcharts/pen/c460241b0efe9c8f6ab1746f44d666af
The changes are that the MapImageSeries code is taken out of the createMarkers function so it only happens once:
const mapImageSeries = chart.series.push(new am4maps.MapImageSeries());
const imageSeriesTemplate = mapImageSeries.mapImages.template;
const circle = imageSeriesTemplate.createChild(am4core.Circle);
circle.radius = 10;
circle.fill = am4core.color('#ff0000');
circle.stroke = am4core.color('#FFFFFF');
circle.strokeWidth = 2;
circle.nonScaling = true;
circle.tooltipText = 'hi';
In this case, there's no need to pass chart to createMarkers and return it, so I've passed polygon instead just to demo dynamic latitude/longitudes, I also assign our new MapImage to the polygon's data (dataItem.dataContext) so we can refer to it later. Here's the new body of createMarkers:
function createMarkers(polygon) {
console.log('calling createMarkers');
if ( !polygon.dataItem.dataContext.redDot) {
const dataItem = polygon.dataItem;
// Object notation for making a MapImage
const redDot = mapImageSeries.mapImages.create();
// Note the lat/long are direct properties
redDot.id = `reddot-${dataItem.dataContext.id}`;
// attempt to make a marker in the middle of the country (note how this is inaccurate for US since we're getting the center for a rectangle, but it's not a rectangle)
redDot.latitude = dataItem.north - (dataItem.north - dataItem.south)/2;
redDot.longitude = dataItem.west - (dataItem.west - dataItem.east)/2;;
dataItem.dataContext.redDot = redDot;
}
}
There's no need for the animationended event or anything, it just works since there is no longer anything interfering with your code. You should also have your performance back.
Will this work for you?
Original answer prior to question's edits below:
I am unable to replicate the behavior you mentioned. Also, I don't know what this.countryZoom is.
Just using the following in a button handler...
chart.zoomToMapObject(polygon);
...seems to zoom just fine to the country, regardless of the current map position/zoomLevel.
If you need to time something after the zoom animation has ended, the zoomToMapObject returns an Animation, you can use its 'animationended' event, e.g.
const animation = this.chart.zoomToMapObject(selectedPoloygon, this.countryZoom, true);
animation.events.on("animationended", () => {
// ...
});
Here's an example with all that with 2 external <button>s, one for zooming to USA and the other Brazil:
https://codepen.io/team/amcharts/pen/c1d1151803799c3d8f51afed0c6eb61d
Does this help? If not, could you possibly provide a minimal example so we can replicate the issue you're having?

dc.js Grouping for Bubble Chart Removing from wrong groups

I'm trying to create a bubble chart with dc.js that will have a bubble for each data row and will be filtered by other charts on the same page. The initial bubble chart is created correctly, but when items are filtered from another chart and added or removed from the group it looks like they are being applied to the wrong group. I'm not sure what I'm messing up on the grouping or dimensions. I've created an example fiddle here
There's simple pie chart to filter on filterColumn, a bubble chart that uses identifer1, a unique field, as the dimension and xVal, yVal, and rVal to display the data, and a dataTable to display the current records.
I've tried other custom groups functions, but switched to the example from the FAQ and still had problems.
var
filterPieChart=dc.pieChart("#filterPieChart"),
bubbleChart = dc.bubbleChart('#bubbleChart'),
dataTable = dc.dataTable('#data-table');
var
bubbleChartDim=ndx.dimension(dc.pluck("identifier1")),
filterPieChartDim=ndx.dimension(dc.pluck("filterColumn")),
allDim = ndx.dimension(function(d) {return d;});
var filterPieChartGroup=filterPieChartDim.group().reduceCount();
function reduceFieldsAdd(fields) {
return function(p, v) {
fields.forEach(function(f) {
p[f] += 1*v[f];
});
return p;
};
}
function reduceFieldsRemove(fields) {
return function(p, v) {
fields.forEach(function(f) {
p[f] -= 1*v[f];
});
return p;
};
}
function reduceFieldsInitial(fields) {
return function() {
var ret = {};
fields.forEach(function(f) {
ret[f] = 0;
});
return ret;
};
}
var fieldsToReduce=['xVal', 'yVal', 'rVal'];
var bubbleChartGroup = bubbleChartDim.group().reduce(
reduceFieldsAdd(fieldsToReduce),
reduceFieldsRemove(fieldsToReduce),
reduceFieldsInitial(fieldsToReduce)
);
filterPieChart
.dimension(filterPieChartDim)
.group(filterPieChartGroup)
...
;
bubbleChart
.dimension(bubbleChartDim)
.group(bubbleChartGroup)
.keyAccessor(function (p) { return p.value.xVal; })
.valueAccessor(function (p) { return p.value.yVal; })
.radiusValueAccessor(function (p) { return p.value.rVal; })
...
;
This was a frustrating one to debug. Your groups and reductions are fine, and that's the best way to plot one bubble for each row, using a unique identifier like that.
[It's annoying that you have to specify a complicated reduction, when the values will be either the original value or 0, but the alternatives aren't much better.]
The reductions are going crazy. Definitely not just original values and zero, some are going to other values, bigger or negative, and sometimes clicking a pie slice twice does not even return to the original state.
I put breakpoints in the reduce functions and noticed, as you did, that the values were being removed from the wrong groups. How could this be? Finally, by logging bubbleChartGroup.all() in a filtered handler for the pie chart, I noticed that the groups were out of order after the first rendering!
Your code is fine. But you've unearthed a new bug in dc.js, which I filed here.
In order to implement the sortBubbleSize feature, we sort the bubbles. Unfortunately we are also sorting crossfilter's internal array of groups, which it trusted us with. (group.all() returns an internal data structure which must never be modified.)
The fix will be easy; we just need to copy the array before sorting it. You can test it out in your code by commenting out sortBubbleSize and instead supplying the data function, which is what it does internally:
bubbleChart.data(function (group) {
var data = group.all().slice(0);
if (true) { // (_sortBubbleSize) {
// sort descending so smaller bubbles are on top
var radiusAccessor = bubbleChart.radiusValueAccessor();
data.sort(function (a, b) { return d3.descending(radiusAccessor(a), radiusAccessor(b)); });
}
return data;
});
Notice the .slice(0) at the top.
Hope to fix this in the next release, but this workaround is pretty solid in case it takes longer.
Here is a fiddle demonstrating the workaround.

Export crossfilter dataset to excel in dc.js

I made a visualization page using crossfilter.js and dc.js . I want to export the filtered dataset to excel. Is any way to do this.?
I think the best way to do this is to create another dimension and then call dimension.top(Infinity) to get all the records (sorted by that dimension's key).
Jacob Rideout created a pull request for a new method to do just this without the overhead, but it was not accepted (doesn't look like it was rejected either ;):
https://github.com/square/crossfilter/pull/95
But I doubt you will notice any performance penalty for creating the extra dimension. (Please comment on that PR if you do!)
function groupArrayAdd(keyfn) {
var bisect = d3.bisector(keyfn);
return function (elements, item) {
var pos = bisect.right(elements, keyfn(item));
elements.splice(pos, 0, item);
return elements;
};
}
function groupArrayRemove(keyfn) {
var bisect = d3.bisector(keyfn);
return function (elements, item) {
var pos = bisect.left(elements, keyfn(item));
if (keyfn(elements[pos]) === keyfn(item))
elements.splice(pos, 1);
return elements;
};
}
function groupArrayInit() {
return [];
}
var facts = crossfilter(data); //pass your mater dataset here.
var filteredRows = facts.groupAll().reduce(
groupArrayAdd(dc.pluck('shift')),
groupArrayRemove(dc.pluck('shift')),
groupArrayInit}
);
filteredRows.value() will give you the crossfilted data. Every time the data is filteded, this function will give automatically five the filted output which you can use to export to excel using any jquery plugin.
Another way to find out filtered data is using below dc function:
dimension.top(Infinity)

How do I prevent selection of data outside my selected range from highlighting when using crossfilter.js?

I'm new to crossfilter.js. Whenever I used the range selector I get the strange blocky highlighted area on my bar chart outside of my filter range. I can't figure out what I'm doing wrong.
How I can prevent this blocky section from highlighting outside my range?
d3.csv("mydata.csv", function(data) {
data.forEach(function(d) {
d.conf = d3.round(+d.Conf,1);
console.log(d.conf);
});
var facts = crossfilter(data); // Put data into crossfilter
var confDim = facts.dimension(function (d) { return d.conf;}); // conf # filter
var confTotal = confDim.group().reduceCount(function (d) { return d.conf;});
var confChart = dc.barChart("#conf-chart");
confChart
.width(500).height(200)
.dimension(confDim)
.group(confTotal,"Confidence Number")
.x(d3.scale.linear().domain([0,5]).range([10,400]))
.yAxisLabel("Number of data points")
.brushOn(true);
dc.renderAll();
});
This line that you've commented as being a filter is actually just setting up the dimension in its entirety:
var confDim = facts.dimension(function (d) { return d.conf;}); // conf # filter
You need to define a filter using one of the filter functions. See the documentation here.
Be careful to read the documentation for dimension.group carefully as it contains a gotcha where the filter on the dimension being grouped won't be applied. To work around this you may need to actually define 2 dimensions that are identical and use one for filtering and the other for grouping.

facing performance issues with knockout mapping plugin

I have decent large data set of around 1100 records. This data set is mapped to an observable array which is then bound to a view. Since these records are updated frequently, the observable array is updated every time using the ko.mapping.fromJS helper.
This particular command takes around 40s to process all the rows. The user interface just locks for that period of time.
Here is the code -
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
ko.mapping.fromJS(data,transactionList)
Is there a workaround for this? Or should I just opt of web workers to improve performances?
Knockout.viewmodel is a replacement for knockout.mapping that is significantly faster at creating viewmodels for large object arrays like this. You should notice a significant performance increase.
http://coderenaissance.github.com/knockout.viewmodel/
I have also thought of a workaround as follows, this uses less amount of code-
var transactionList = ko.mapping.fromJS([]);
//Getting the latest transactions which are around 1100 in number;
var data = storage.transactions();
//Mapping the data to the observable array, which takes around 40s
// Instead of - ko.mapping.fromJS(data,transactionList)
var i = 0;
//clear the list completely first
transactionList.destroyAll();
//Set an interval of 0 and keep pushing the content to the list one by one.
var interval = setInterval(function () {if (i == data.length - 1 ) {
clearInterval(interval);}
transactionList.push(ko.mapping.fromJS(data[i++]));
}, 0);
I had the same problem with mapping plugin. Knockout team says that mapping plugin is not intended to work with large arrays. If you have to load such big data to the page then likely you have improper design of the system.
The best way to fix this is to use server pagination instead of loading all the data on page load. If you don't want to change design of your application there are some workarounds which maybe help you:
Map your array manually:
var data = storage.transactions();
var mappedData = ko.utils.arrayMap(data , function(item){
return ko.mapping.fromJS(item);
});
var transactionList = ko.observableArray(mappedData);
Map array asynchronously. I have written a function that processes array by portions in another thread and reports progress to the user:
function processArrayAsync(array, itemFunc, afterStepFunc, finishFunc) {
var itemsPerStep = 20;
var processor = new function () {
var self = this;
self.array = array;
self.processedCount = 0;
self.itemFunc = itemFunc;
self.afterStepFunc = afterStepFunc;
self.finishFunc = finishFunc;
self.step = function () {
var tillCount = Math.min(self.processedCount + itemsPerStep, self.array.length);
for (; self.processedCount < tillCount; self.processedCount++) {
self.itemFunc(self.array[self.processedCount], self.processedCount);
}
self.afterStepFunc(self.processedCount);
if (self.processedCount < self.array.length - 1)
setTimeout(self.step, 1);
else
self.finishFunc();
};
};
processor.step();
};
Your code:
var data = storage.transactions();
var transactionList = ko.observableArray([]);
processArrayAsync(data,
function (item) { // Step function
var transaction = ko.mapping.fromJS(item);
transactionList().push(transaction);
},
function (processedCount) {
var percent = Math.ceil(processedCount * 100 / data.length);
// Show progress to the user.
ShowMessage(percent);
},
function () { // Final function
// This function will fire when all data are mapped. Do some work (i.e. Apply bindings).
});
Also you can try alternative mapping library: knockout.wrap. It should be faster than mapping plugin.
I have chosen the second option.
Mapping is not magic. In most of the cases this simple recursive function can be sufficient:
function MyMapJS(a_what, a_path)
{
a_path = a_path || [];
if (a_what != null && a_what.constructor == Object)
{
var result = {};
for (var key in a_what)
result[key] = MyMapJS(a_what[key], a_path.concat(key));
return result;
}
if (a_what != null && a_what.constructor == Array)
{
var result = ko.observableArray();
for (var index in a_what)
result.push(MyMapJS(a_what[index], a_path.concat(index)));
return result;
}
// Write your condition here:
switch (a_path[a_path.length-1])
{
case 'mapThisProperty':
case 'andAlsoThisOne':
result = ko.observable(a_what);
break;
default:
result = a_what;
break;
}
return result;
}
The code above makes observables from the mapThisProperty and andAlsoThisOne properties at any level of the object hierarchy; other properties are left constant. You can express more complex conditions using a_path.length for the level (depth) the value is at, or using more elements of a_path. For example:
if (a_path.length >= 2
&& a_path[a_path.length-1] == 'mapThisProperty'
&& a_path[a_path.length-2] == 'insideThisProperty')
result = ko.observable(a_what);
You can use typeOf a_what in the condition, e.g. to make all strings observable.
You can ignore some properties, and insert new ones at certain levels.
Or, you can even omit a_path. Etc.
The advantages are:
Customizable (more easily than knockout.mapping).
Short enough to copy-paste it and write individual mappings for different objects if needed.
Smaller code, knockout.mapping-latest.js is not included into your page.
Should be faster as it does only what is absolutely necessary.

Resources