Related
I have CSV data with the following pattern:
Quarter,productCategory,unitsSold
2018-01-01,A,21766
2018-01-01,B,10076
2018-01-01,C,4060
2018-04-01,A,27014
2018-04-01,B,12219
2018-04-01,C,4740
2018-07-01,A,29503
2018-07-01,B,13020
2018-07-01,C,5549
2018-10-01,A,3796
2018-10-01,B,15110
2018-10-01,C,6137
2019-01-01,A,25008
2019-01-01,B,11655
2019-01-01,C,4630
2019-04-01,A,31633
2019-04-01,B,14837
2019-04-01,C,5863
2019-07-01,A,33813
2019-07-01,B,15442
2019-07-01,C,6293
2019-10-01,A,35732
2019-10-01,B,19482
2019-10-01,C,6841
As you can see, there are 3 product categories sold every day. I can make a histogram and count how many Quarters are involved per bin of unitsSold. The problem here is that every Quarter is counted separately. What I would like is a histogram where the bins of unitsSold are already grouped with a reduceSum on the Quarter.
This would result in something like this:
Quarter, unitsSold
2018-01-01,35902
2018-04-01,43973
2018-07-01,48072
2018-10-01,25043
2019-01-01,41293
2019-04-01,52333
2019-07-01,55548
2019-10-01,62055
Where, based on the bins of unitsSold, a number of Quarters would fall into. For example a bin of 50.000 - 70.000 would count 3 Quarters (2019-04-01, 2019-07-01 and 2019-10-01)
Normally I would do something like this:
const histogramChart = new dc.BarChart('#histogram');
const histogramDim = ndx.dimension(d => Math.round(d.unitsSold / binSize) * binSize);
const histogramGroup = histogramDim.group().reduceCount();
But in the desired situation the histogram is kind of created on something that has already been "reducedSummed". Ending up in a barchart histogram like this (data does not match with this example):
How can this be done with dc.js/crossfilter.js?
Regrouping the data by value
I think the major difference between your question and this previous question is that you want to bin the data when you "regroup" it. (Sometimes this is called a "double reduce"... no clear names for this stuff.)
Here's one way to do that, using an offset and width:
function regroup(group, width, offset = 0) {
return {
all: function() {
const bins = {};
group.all().forEach(({key, value}) => {
const bin = Math.floor((value - offset) / width);
bins[bin] = (bins[bin] || 0) + 1;
});
return Object.entries(bins).map(
([bin, count]) => ({key: bin*width + offset, value: count}));
}
}
}
What we do here is loop through the original group and
map each value to its bin number
increment the count for that bin number, or start at 1
map the bins back to original numbers, with counts
Testing it out
I displayed your original data with the following chart (too lazy to figure out quarters, although I think it's not hard with recent D3):
const quarterDim = cf.dimension(({Quarter}) => Quarter),
unitsGroup = quarterDim.group().reduceSum(({unitsSold}) => unitsSold);
quarterChart.width(300)
.height(200)
.margins({left: 50, top: 0, right: 0, bottom: 20})
.dimension(quarterDim)
.group(unitsGroup)
.x(d3.scaleTime().domain([d3.min(data, d => d.Quarter), d3.timeMonth.offset(d3.max(data, d => d.Quarter), 3)]))
.elasticY(true)
.xUnits(d3.timeMonths);
and the new chart with
const rg = regroup(unitsGroup, 10000);
countQuartersChart.width(500)
.height(200)
.dimension({})
.group(rg)
.x(d3.scaleLinear())
.xUnits(dc.units.fp.precision(10000))
.elasticX(true)
.elasticY(true);
(Note the empty dimension, which disables filtering. Filtering may be possible but you have to map back to the original dimension keys so I’m skipping that for now.)
Here are the charts I get, which look correct at a glance:
Demo fiddle.
Adding filtering to the chart
To implement filtering on this "number of quarters by values" histogram, first let's enable filtering between the by-values chart and the quarters chart by putting the by-values chart on its own dimension:
const quarterDim2 = cf.dimension(({Quarter}) => Quarter),
unitsGroup2 = quarterDim2.group().reduceSum(({unitsSold}) => unitsSold);
const byvaluesGroup = regroup(unitsGroup2, 10000);
countQuartersChart.width(500)
.height(200)
.dimension(quarterDim2)
.group(byvaluesGroup)
.x(d3.scaleLinear())
.xUnits(dc.units.fp.precision(10000))
.elasticX(true)
.elasticY(true);
Then, we implement filtering with
countQuartersChart.filterHandler((dimension, filters) => {
if(filters.length === 0)
dimension.filter(null);
else {
console.assert(filters.length === 1 && filters[0].filterType === 'RangedFilter');
const range = filters[0];
const included_quarters = unitsGroup2.all()
.filter(({value}) => range[0] <= value && value < range[1])
.map(({key}) => key.getTime());
dimension.filterFunction(k => included_quarters.includes(k.getTime()));
}
return filters;
});
This finds all quarters in unitsGroup2 that have a value which falls in the range. Then it sets the dimension's filter to accept only the dates of those quarters.
Odds and ends
Quarters
D3 supports quarters with interval.every:
const quarterInterval = d3.timeMonth.every(3);
chart.xUnits(quarterInterval.range);
Eliminating the zeroth bin
As discussed in the comments, when other charts have filters active, there may end up being many quarters with less than 10000 units sold, resulting in a very tall zero bar which distorts the chart.
The zeroth bin can be removed with
delete bins[0];
before the return in regroup()
Rounding the by-values brush
If snapping to the bars is desired, you can enable it with
.round(x => Math.round(x/10000)*10000)
Otherwise, the filtered range can start or end inside of a bar, and the way the bars are colored when brushed is somewhat inaccurate as seen below.
Here's the new fiddle.
I'm newbie and I'm working on a dashboard. I want to show with a pie chart the total value of one dimension (100% when all the registers all selected, and change it with the other filters). I've tried it with groupAll() but it doesn't work. This code works but it shows the groups separate. How can I do this? Thanks a lot!!!
CSV
CausaRaiz,probabilidad,costeReparacion,costePerdidaProduccion,impacto,noDetectabilidad,criticidad,codigo,coste,duracion,recursosRequeridos
PR.CR01,2,1.3,1,1,1,2,AM.PR.01,1,2,Operarios
PR.CR02,4,2.3,3,2.5,2,20,AM.PR.02,2,3,Ingenieria
PR.CR03,4,3.3,4,3.5,4,25,AM.PR.03,3,4,Externos
PR.CR04,2,2.7,2,2,2,8,AM.PR.04,3,4,Externos
FR.CR01,3,2.9,3,2.5,3,22,AM.FR.01,4,5,Ingenieria
FR.CR02,2,2.1,2,2,2,8,AM.FR.02,4,3,Operarios
FR.CR03,1,1.7,1,1,1,1,AM.FR.03,3,5,Operarios
RF.CR01,1,1.9,2,2,3,6,AM.RF.01,3,5,Externos
RF.CR02,3,3.5,4,3.5,4,20,AM.RF.02,4,4,Ingenieria
RF.CR03,4,3.9,4,3.5,4,25,AM.RF.03,4,5,Operarios
Code working
var pieCri = dc.pieChart("#criPie")
var criDimension = ndx.dimension(function(d) { return +d.criticidad; });
var criGroup =criDimension.group().reduceCount();
pieCri
.width(270)
.height(270)
.innerRadius(20)
.dimension(criDimension)
.group(criGroup)
.on('pretransition', function(chart) {
chart.selectAll('text.pie-slice').text(function(d) {
return d.data.key + ' ' + dc.utils.printSingleValue((d.endAngle - d.startAngle) / (2*Math.PI) * 100) + '%';
})
});
pieCri.render();
I can show the total percentage with a number:
var critTotal = ndx.groupAll().reduceSum(function(d) { return +d.criticidad; });
var numbCriPerc = dc.numberDisplay("#criPerc");
numbCriPerc
.group(critTotal)
.formatNumber(d3.format(".3s"))
.valueAccessor( function(d) { return d/critTotalValue*100; } );
But I prefer in a pie chart to show the difference between all the registers and the selection.
If I understand your question correctly, you want to show a pie chart with exactly two slices: the count of items included, and the count of items excluded.
You're on the right track with using groupAll, which is great for taking a count of rows (or sum of a field) based on the current filters. There are just two parts missing:
finding the full total with no filters applied
putting the data in the right format for the pie chart to read it
This kind of preprocessing is really easy to do with a fake group, which will adapt as the filters change.
Here is one way to do it:
// takes a groupAll and produces a fake group with two key/value pairs:
// included: the total value currently filtered
// excluded: the total value currently excluded from the filter
// "includeKey" and "excludeKey" are the key names to give to the two pairs
// note: this must be constructed before any filters are applied!
function portion_group(groupAll, includeKey, excludeKey) {
includeKey = includeKey || "included";
excludeKey = excludeKey || "excluded";
var total = groupAll.value();
return {
all: function() {
var current = groupAll.value();
return [
{
key: includeKey,
value: current
},
{
key: excludeKey,
value: total - current
}
]
}
}
}
You'll construct a groupAll to find the total under the current filters:
var criGroupAll = criDimension.groupAll().reduceCount();
And you can construct the fake group when passing it to the chart:
.group(portion_group(criGroupAll))
Note: you must have no filters active when constructing the fake group this way, since it will grab the unfiltered total at that point.
Finally, I noticed that the way you were customizing pie chart labels, they would be shown even if the slice is empty. That looked especially bad in this example, so I fixed it like this:
.on('pretransition', function(chart) {
chart.selectAll('text.pie-slice').text(function(d) {
return d3.select(this).text() && (d.data.key + ' ' + dc.utils.printSingleValue((d.endAngle - d.startAngle) / (2*Math.PI) * 100) + '%');
})
});
This detects whether the label text is empty because of minAngleForLabel, and doesn't try to replace it in that case.
Example fiddle based on your code.
I have a similar problem as the one described in DC.JS get last month value as filter.
That is, I want to display in bar charts the value of stock for the last month (by company and by product type), while simultaneously I show the evolution through time of the stock in a line chart.
In order to do so I need to keep track of the last month selected by brushing the line chart. See my jsFiddle here: https://jsfiddle.net/BernG/wo60z64j/12/
var data = [{date:"201501",company:"A", product: "Prod1", stock:575}
, {date:"201502",company:"A", product: "Prod1", stock:325}
, {date:"201503",company:"A", product: "Prod1", stock:200}
, {date:"201504",company:"A", product: "Prod1", stock:450}
, {date:"201501",company:"A", product: "Prod2", stock:279}
, {date:"201502",company:"A", product: "Prod2", stock:93}
, {date:"201503",company:"A", product: "Prod2", stock:0}
, {date:"201504",company:"A", product: "Prod2", stock:372}
, {date:"201501",company:"A", product: "Prod3", stock:510}
, {date:"201502",company:"A", product: "Prod3", stock:340}
, {date:"201503",company:"A", product: "Prod3", stock:680}
, {date:"201504",company:"A", product: "Prod3", stock:170}
, {date:"201501",company:"B",product: "Prod1", stock:1000}
, {date:"201502",company:"B",product: "Prod1", stock:1100}
, {date:"201503",company:"B",product: "Prod1", stock:900}
, {date:"201504",company:"B",product: "Prod1", stock:1200}
, {date:"201501",company:"B",product: "Prod2", stock:1000}
, {date:"201502",company:"B",product: "Prod2", stock:1200}
, {date:"201503",company:"B",product: "Prod2", stock:900}
, {date:"201504",company:"B",product: "Prod2", stock:1200}
, {date:"201501",company:"B",product: "Prod3", stock:1000}
, {date:"201502",company:"B",product: "Prod3", stock:1100}
, {date:"201503",company:"B",product: "Prod3", stock:900}
, {date:"201504",company:"B",product: "Prod3", stock:600}];
// Reading and formatting values
var dateFormat = d3.time.format('%Y%m');
data.forEach(function (d) {
d.dd = dateFormat.parse(d.date);
d.year = d3.time.year(d.dd);
d.month = d3.time.month(d.dd);
});
// Definition of crossfilter and dimensions
var ndx = crossfilter(data)
, dimMonth = ndx.dimension(function(d){return d.month})
, dimProduct = ndx.dimension(function(d){return d.product})
, dimCompany = ndx.dimension(function(d){return d.company;});
var lastStaticDate = dimMonth.top(1)[0].month; // identify last date in full time domain
var firstStaticDate = dimMonth.bottom(1)[0].month; // identify first date in full time domain
// Definition of a function to keep track of the last date value in the brush attached to the chartMonth.
// If chartMonth object does not exist or its corresponding brush is empty (has not been brushed), then it returns the lastStaticDate
// otherwise, it returns the last date in the brush selection
var getLastDate = function(){
if (typeof chartMonth === "undefined"){ // test if chartMonth is already in the DOM, if not return lastStaticDate
return lastStaticDate;
}
else {
if (chartMonth.brush().empty()) { // if chartMonth has been created but brush does not have have a selection
return lastStaticDate;
}
else {
return chartMonth.brush().extent()[1];
};
}
};
var lastDate = d3.time.month.ceil(getLastDate()); // lastDate is snapped to return a date where we have data in the x-domain
dateBal.innerHTML = lastDate;
var grpMonth = dimMonth.group().reduceSum(function(d){return d.stock;}); // the line chart displays all values in the x-axis to show stock evolution
// Definition of custom reduce functions
function reduceAdd(p,v) {
if (p.month===lastDate){
p.stock += v.stock;
return p;
}
else {return p;}
};
function reduceRemove(p,v) {
if (p.month!== lastDate){
p.stock -= v.stock;
return p;}
else {return p;}
};
function reduceInitial() {
return {stock: 0}
};
// Application of reduce functions
var grpCompany = dimCompany
.group()
.reduce(reduceAdd, reduceRemove, reduceInitial);
var grpProduct = dimProduct
.group()
.reduce(reduceAdd, reduceRemove, reduceInitial);
var chartCompany = dc.barChart('#chartCompany');
var chartProduct = dc.barChart("#chartProduct");
var chartMonth = dc.lineChart('#chartMonth');
chartCompany
.width(400)
.height(400)
.margins({top: 50, right: 50, bottom: 50, left: 50})
.dimension(dimCompany)
.group(grpCompany)
.x(d3.scale.ordinal())
.xUnits(dc.units.ordinal)
.elasticY(true);
chartProduct
.width(400)
.height(400)
.margins({top: 50, right: 50, bottom: 50, left: 50})
.dimension(dimProduct)
.group(grpProduct)
.x(d3.scale.ordinal())
.xUnits(dc.units.ordinal)
.elasticY(true);
chartMonth
.width(400)
.height(400)
.margins({top: 50, right: 50, bottom: 50, left: 50})
.renderlet(function (chart) {
// rotate x-axis labels
chart.selectAll('g.x text')
.attr('transform', 'translate(-10,10) rotate(315)');
})
.dimension(dimMonth)
.group(grpMonth)
.x(d3.time.scale().domain([firstStaticDate, lastStaticDate]))
.xUnits(d3.time.months)
.elasticX(true);
dc.renderAll();
The solution I tried is based on the only answer to the above question (which, by the way, was not marked as accepted) and follows this logic:
1) First, I try to identify the last date in the crossfiltered data set. Upon initialization it would be the last date of my time dimension before the brush is moved in the line chart. Otherwise, it returns the date from the second element in the brush extent. To this date I apply the ceiling function to make sure that the date it returns exists as exact date in my time dimension.
2) I apply custom reduce functions excluding data from months different than the current selection.
My specific questions are:
How can I make the variable (lastDate) reactive? The following works well in the console: d3.time.month.ceil(getLastDate()). However, it does not react to an interactive brushing event.
What changes need to be done in my custom reduce functions to accumulate only the values that correspond to the lastDate and excludes all others? For some reason the customer reduce functions as they are currently defined do not accumulate stock values correctly. For instance, upon initialization, if I inspect the object where grpCompany resides, it shows the value of the stock as 0.
grpCompany.all() in console
Finally, before you vote to mark this question as duplicate please consider the following:
Original post does not seem to have an accepted answer.
There was an offer to help if OP provided a working jsFiddle, but none was provided.
I wanted to make a follow up comment in the original post asking for clarification but it was not possible to do so, because I do not have yet the required reputation to post a comment.
Ok, this is really complicated, which is why I started my comment on the previous question by saying that Crossfilter isn't good at this. My answer was quite complicated and you haven't managed to do what I meant to suggest, but that's not your fault. The Universe library supports some of what you are suggesting, but overall I would recommend rethinking your approach if it is getting this complicated. However, your scenario as stated is also a little different from the one in the previous question and has a simpler answer:
In your case, what you should actually do is use a custom dc.js filterFunction to make the stock balance chart only filter to the last date in the brush:
chartMonth
...
.filterHandler(function(d,f) {
if(f.length > 0) {
d.filterExact(d3.time.month.ceil(f[0][1]));
dateBal.innerHTML = d3.time.month.ceil(f[0][1]);
} else {
d.filterAll();
dateBal.innerHTML = "No month selected"
}
return f;
})
Here's a working version based on your JSFiddle: https://jsfiddle.net/esjewett/utohm7mq/2/
So, that's the simple answer. What it actually does is just filter your whole Crossfilter to the last month selected in the brush (or actually the subsequent month). If you also want to filter some other charts to the actual dates selected by the brush, then you are talking about having multiple conflicting filters and Crossfilter currently doesn't support this very well. We are interested in adding functionality to support multiple filter groups, but have not begun to tackle that yet. In this case, the best approach right now is probably to maintain 2 separate Crossfilters and drive different groups of charts off of different Crossfilters.
I have a dataset (data) with the following row/column structure:
Date Category1 Category2 Revenue
30/12/2014 a x 10
30/12/2014 b x 15
31/12/2014 a x 11
1/1/2015 a x 13
2/1/2015 a x 14
2/1/2015 b x 9
2/1/2015 c z 4
...
Based on data I create a couple of dimensions and groups:
var ndx = crossfilter(data);
var cat1Dim = ndx.dimension(function(d) {return d.Category1;});
var revenuePerCat1 = cat1Dim.group().reduceSum(function(d) { return d.Revenue; });
var cat2Dim = ndx.dimension(function(d) {return d.Category2;});
var revenuePerCat2 = cat2Dim.group().reduceSum(function(d) { return d.Revenue; });
var dateDim = ndx.dimension(function(d) { return d.Date; });
var revenuePerDate = dateDim.group().reduceSum(function(d) { return d.Revenue; });
Next, I create the following charts:
a line chart; dimension = dateDim, group = revenuePerDate
a pie-chart; dimension = cat1Dim, group = revenuePerCat1
a pie-chart; dimension = cat2Dim, group = revenuePerCat2
Besides the charts I would also like to show the year-to-date value of the revenues via a numberDisplay. Initially I thought to achieve this by adding a simple if condition to the reduceSum function where I reduce the data to contain only items of the current year, like so:
var ytdRev = ndx.groupAll().reduceSum(function(d) { if(d.Date.getFullYear() == curYear) {return d.Revenue;} else{return 0;}});
A box containing a numberDisplay item is then called by:
box_ytd
.formatNumber("$,.4s")
.valueAccessor(function(d) {return Math.round(d * 1000) / 1000; })
.group(ytdRev);
This works perfectly fine if one selects one of the categories displayed in the pie-charts, but is incorrect when one also starts to filter date ranges in the line chart. Namely, instead of a year-to-date value, actually a 'date-to-date' value for the specific selection will be returned. Although this behaviour is correct from a technical perspective, I would like to know how I can instruct dc.js such that it will only take into account chart selections from a certain set of charts when rendering a numberDisplay. The selections made in the pie-charts should, however, both update the displayed selection in the line chart and the numberDisplay.
Ideally, I would like to use one crossfilter instance only, but I am open to any suggestions that involve a second crossfilter as well.
EDIT:
Based on Gordon's comment I played around with a custom reduce function. Instead of ndx.groupAll() I applied the following reduce function with a .groupAll() on the dimension level:
function reduceAdd(p,v) {
if(v.Date.getFullYear() == curYear)
p.revenue += +v.Revenue;
return p;}
function reduceRemove(p,v) {
if v.Date.getFullYear() == curYear)
p.revenue -= +v.Revenue;
return p;}
function reduceInitial() {
return {revenue:0 };}
var ytdRev = dateDim.groupAll().reduce(reduceAdd, reduceRemove, reduceInitial);
The .valueAccessor in the numberDisplay is changed from d.value.revenue to d.revenue:
box_ytd
.formatNumber("$,.4s")
.valueAccessor(function(d) {return Math.round(d.revenue * 1000) / 1000; })
.group(ytdRev);
The numberDisplay will now reflect the total value for the current year for each of the selections made in the pie-charts. Date selections will only affect the pie-charts' values; the numberDisplay shares the same dimension with the line chart and hence the numberDisplay is unaffected by any selections on that dimension.
Based on Gordon's comment I played around with a custom reduce function. Instead of ndx.groupAll() I applied the following reduce function with a .groupAll() on the dimension level:
function reduceAdd(p,v) {
if(v.Date.getFullYear() == curYear)
p.revenue += +v.Revenue;
return p;}
function reduceRemove(p,v) {
if v.Date.getFullYear() == curYear)
p.revenue -= +v.Revenue;
return p;}
function reduceInitial() {
return {revenue:0 };}
var ytdRev = dateDim.groupAll().reduce(reduceAdd, reduceRemove, reduceInitial);
The .valueAccessor in the numberDisplay is changed from d.value.revenue to d.revenue:
box_ytd
.formatNumber("$,.4s")
.valueAccessor(function(d) {return Math.round(d.revenue * 1000) / 1000; })
.group(ytdRev);
The numberDisplay will now reflect the total value for the current year for each of the selections made in the pie-charts. Date selections will only affect the pie-charts' values; the numberDisplay shares the same dimension with the line chart and hence the numberDisplay is unaffected by any selections on that dimension.
My dataset is an array of json of the like :
var data = [ { company: "A", date_round_1: "21/05/2002", round_1: 5, date_round_2: "21/05/2004", round_2: 20 },
...
{ company: "Z", date_round_1: "16/01/2004", round_1: 10, date_round_2: "20/12/2006", round_2: 45 }]
and I wish to display both 'round_1' and 'round_2' time series as stacked line charts.
The base line would look like this :
var fundsChart = dc.lineChart("#fundsChart");
var ndx = crossfilter(data);
var all = ndx.groupAll();
var date_1 = ndx.dimension(function(d){
return d3.time.year(d.date_round_1);
})
fundsChart
.renderArea(true)
.renderHorizontalGridLines(true)
.width(400)
.height(360)
.dimension(date_1)
.group(date_1.group().reduceSum(function(d) { return +d.round_1 }))
.x(d3.time.scale().domain([new Date(2000, 0, 1), new Date(2015, 0, 1)]))
I have tried using the stack method to add the series but the problem resides in the fact that only a single dimension can be passed as argument of the lineChart.
Can you think of a turnaround to display both series while still using a dc chart?
Are you going to be filtering on this chart? If not, just create a different group on a date_2 dimension and use that in the stack. Should work.
If you are going to be filtering, I think you'll have to change your data model a bit. You'll want to switch to have 1 record per round, so in this case you'll have 2 records for every 1 record you have now. There should be 1 date property (the date for that round), an amount property (the contents of round_x in the current structure), and a 'round' property (which would be '1', or '2', for example).
Then you need to create a date dimension and multiple groups on that dimension. The group will have a reduceSum function that looks something like:
var round1Group = dateDim.group().reduceSum(function(d) {
return d.round === '1' ? d.amount : 0;
});
So, what happens here is that we have a group that will only aggregate values from round 1. You'll create similar groups for round 2, etc. Then stack these groups in the dc.js chart.
Hopefully that helps!