I'm building a data dashboard for a project and I want to be able to compare data from two distinct groups in the same dataset.
My dataset looks like this:
Number,Name,Gender,Race,Height,Publisher,Alignment,Weight,Superpower,Strength,Costume Colour
1,A-Bomb,Male,Metahuman,203,Marvel Comics,Good,441,Superhuman Strength,10,None
2,Abin Sur,Male,Alien,185,DC Comics,Good,90,Cosmic Power,40,Green
3,Abomination,Male,Metahuman,203,Marvel Comics,Bad,441,Superhuman Strength,10,None
4,Abraxas,Male,Cosmic Entity,1000,Marvel Comics,Bad,1000,Reality Warping,40,Green
5,Absorbing Man,Male,Metahuman,193,Marvel Comics,Bad,122,Matter Duplication,5,None
6,Adam Strange,Male,Human,185,DC Comics,Good,88,None,0,Red
I want to create two separate selectMenus which list the character names, but with each of the two menus filtered on Publisher name.
So one drop down will have all the characters associated with Marvel Comics, and the other will have all the characters associated with DC Comics.
Once this is set up, the idea is that the dashboard can then show a set of graphs which work as a comparison between the two characters that have been selected - so I don't want the entire dataset to be split out, I just want the selection to be filtered by Publisher.
I've been through dozens of different Stack Overflow threads about similar stuff but still struggling. I've created the dimensions for character name and character publisher but I'm getting really lost trying to use one to filter the other.
This is what I've got so far (ignore the costume color stuff, that's for something further down the line) - the data in 'heroes-information.csv' is in the same format I shared above.
// Bring in data from csv files
Promise.all([d3.csv("../data/heroes-information.csv"), d3.csv("../data/costume-colors.csv")])
.then(function(data) {
// Tidy data before use
data.forEach(function(d) {
d.Height = +d.Height;
d.Weight = +d.Weight;
d.Strength = +d.Strength;
});
// Bring in Heroes data
var ndx = crossfilter(data[0]);
// Bring in costume color data
var ndxcol = crossfilter(data[1]);
// Create colorScale to dynamically color pie chat slices
var colorScale = d3.scaleOrdinal()
.domain(data[1].map(row => row.Name))
.range(data[1].map(row => row.RGB));
// Define chart type
var dccomicsSelector = dc.selectMenu('#dccomics-selector');
// Define chart dimension
var character = ndx.dimension(dc.pluck('Name'));
var characterPublisher = ndx.dimension(dc.pluck('Publisher'));
var dccomicsCharacters = character.group();
var dccomicsPublisher = characterPublisher.group();
dccomicsSelector
.dimension(character)
.group(dccomicsCharacters);
dc.renderAll();
});
I'm probably missing something really obvious but I'm fairly new to DC.js and Crossfilter so a bit lost in the weeds with this one, any help would be much appreciated!
Related
I'm building a data dashboard using DC.js and was wondering if it was possible to change the color of the slices in a pie chart dynamically based on the value in the field it is referring to.
Basically I've built a pie chart aggregating the costume colors of different superheroes and I'd love to be able to color each slice with the color it is referring to - so the slice for 'Black' is colored black, the slice for 'Green' is colored green and so forth.
I'm fairly new to DC.js so accept that it may not be possible, but wanted to throw it out there and see if it could be done!
I tried including an array within .ordinalColors but couldn't figure out if there was a way to pull in the data from the field dynamically. I'm assuming that I'd have to change the data in the .csv file to a string that could be recognised as a color reference, but not sure how to go about doing that.
function show_costume_color(ndx) {
var costume_color_dim = ndx.dimension(dc.pluck('Costume Colour'));
var costume_color = costume_color_dim.group();
dc.pieChart('#costume-color')
.width(500)
.height(500)
.radius(500)
.innerRadius(100)
.slicesCap([7])
.transitionDuration(1500)
.dimension(costume_color_dim)
.group(costume_color);
}
CSV data comes in the below format
ID,name,Gender,Eye color,Race,Hair color,Publisher,Alignment,Superpower,Superpower Strength Level,Costume
Colour
0,A-Bomb,Male,Yellow,Human,No Hair,Marvel Comics,Good,Superhuman
Strength,10,None
1,Abin Sur,Male,Blue,Ungaran,No Hair,DC Comics,Good,Cosmic Power,40,Green
Yes, of course. Everything is specified dynamically in dc.js.
Assuming you are using dc.js v3 (and d3 v4+) the way I would suggest doing this is by creating another CSV file with the color assignments you want, something like
Name, RGB
Red, #ff1122
Blue, #1133ff
...
Then you can load the second file in parallel with your data using Promise.all(),
Promise.all([d3.csv('data.csv'), d3.csv('colors.csv')])
.then(function(data, colors) {
// rest of code will go here
});
ordinalColors is a nice convenience method, but if you want complete control, and to understand exactly what's going on, it's better to supply your own color scale. In this case, we want an ordinal scale, which maps specific discrete values to specific colors.
Under the covers, dc.js always deals with colors by using the colorAccessor to fetch a value for the the item, and then mapping this value using a color scale. You can think of the value that the accessor returns as a "color name", which is pretty convenient because it's exactly what you want here.
So you can populate a d3.scaleOrdinal with the domain of color names and the range of RGB colors:
var colorScale = d3.scaleOrdinal()
.domain(colors.map(row => row.Name))
.range(colors.map(row => row.RGB));
Now supply it to your chart using .colors():
chart.colors(colorScale);
What's really handy about this approach is that you can supply the same color scale for multiple charts, in order to make sure they are consistent. This is something that you don't get automatically in dc.js, because charts don't know very much about each other.
So, I managed to figure it out through an extensive period of trial and error and now I'm off and away with my dashboard. Thanks for your help, Gordon - it really made the difference! It needs a bit of tidying up but my working test code is below.
// Bring in data from both csv files
Promise.all([d3.csv("../data/heroes_information.csv"),
d3.csv("../data/costume_colors.csv")])
.then(function(data) {
// Tidy up data before use
data.forEach(function(d) {
d.Height = +d.Height;
d.Weight = +d.Weight;
d.Strength = +d.Strength;
});
// Bring in colorScale to dynamically color pie chart slices
var ndxcol = crossfilter(data[1]);
var colorScale = d3.scaleOrdinal()
.domain(data[1].map(row => row.Name))
.range(data[1].map(row => row.RGB));
// Bring in superhero data
var ndx = crossfilter(data[0]);
// Define chart types
var publisherSelector = dc.selectMenu('#publisher-selector')
var genderChart = dc.rowChart('#gender-balance');
// Define chart dimensions
var publisherChoice = ndx.dimension(dc.pluck('Publisher'));
var genderBalance = ndx.dimension(dc.pluck('Gender'));
// Define chart groups
var genderNumber = genderBalance.group();
var publisherNumber = publisherChoice.group();
// Draw charts
publisherSelector
.dimension(publisherChoice)
.group(publisherNumber);
genderChart
.width(500)
.height(200)
.margins({ top: 30, right: 30, bottom: 30, left: 30 })
.dimension(genderBalance)
.group(genderNumber)
.gap(6)
.colors(colorScale)
.transitionDuration(500)
.x(d3.scaleOrdinal())
.elasticX(true);
dc.renderAll();
});
I'm trying to implement a live data visualization (i.e. with new data arriving periodically) using dc.js. The problem I'm having is the following - when new data is added to the plot, already existing points often start to "dance around", even though they were not changed. Can this be avoided?
The following fiddle illustrates this.
My guess is that crossfilter sorts data internally, which results in points moving on the chart for data items that changed their position (index) in the internal storage. Data is added in the following way:
var data = [];
var ndx = crossfilter(data)
setInterval(function() {
var value = ndx.size() + 1;
if (value > 50) {
return;
}
var newElement = {
x: myRandom(),
y: myRandom()
};
ndx.add([newElement]);
dc.redrawAll();
}, 1000);
Any ideas?
I stand by my comments above. dc.js should be fixed by binding the data using a key function, and probably the best way to deal with the problem is just to disable transitions on the scatterplot using .transitionDuration(0)
However, I was curious if it was possible to work around the current problems by keeping the group in a set order using a fake group. And it is indeed, at least for this example where there is no aggregation and we just want to display the original data points.
First, we add a third field, index, to the data. This has to order the data in the same order in which it comes in. As noted in the discussion above, the scatter plot is currently binding data by its index, so we need to keep the points in a set order; nothing should be inserted.
var newElement = {
index: value,
x: myRandom(),
y: myRandom()
};
Next, we have to preserve this index through the binning and aggregation. We could keep it either in the key or in the value, but keeping it in the key seems more fitting:
xyiDimension = ndx.dimension(function(d) {
return [+d.x, +d.y, d.index];
}),
xyiGroup = xyiDimension.group();
The original reduction didn't make sense to me, so I dropped it. We'll just use the default behavior, which counts the number of rows which fall into each bin. The counts should be 1 if included, or 0 if filtered out. Including the index in the key also ensures uniqueness, which the original keys were not guaranteed to have.
Now we can create a fake group that keeps everything sorted by index:
var xyiGroupSorted = {
all: function() {
var ret = xyiGroup.all().slice().sort((a,b) => a.key[2] - b.key[2]);
return ret;
}
}
This will fetch the original data whenever it's requested by the chart, create a copy of the array (because the original is owned by crossfilter), and sort it to return it to the correct order.
And voila, we have a scatter plot that behaves the way it should, even though the data has gone through crossfilter.
Fork of your fiddle: https://jsfiddle.net/gordonwoodhull/mj81m42v/13/
[After all this, maybe we shouldn't have given the data to crossfilter in the first place! We could have just created a fake group which exposes the original data. But maybe there's some use to this technique. At least it proves that there's almost always a way to work around any problems in dc.js & crossfilter.]
Good Evening Everyone,
I'm trying to take the data from a database full of hour reports (name, timestamp, hours worked, etc.) and create a plot using dc.js to visualize the data. I would like the timestamp to be on the x-axis, the sum of hours for the particular timestamp on the y-axis, and a new bar graph for each unique name all on the same chart.
It appears based on my objectives that using crossfilter.js the timestamp should be my 'dimension' and then the sum of hours should be my 'group'.
Question 1, how would I then use the dimension and group to further split the data based on the person's name and then create a bar graph to add to my composite graph? I would like for the crossfilter.js functionality to remain intact so that if I add a date range tool or some other user controllable filter, everything updates accordingly.
Question 2, my timestamps are in MySQL datetime format: YYYY-mm-dd HH:MM:SS so how would I go about dropping precision? For instance, if I want to combine all entries from the same day into one entry (day precision) or combine all entries in one month into a single entry (month precision).
Thanks in advance!
---- Added on 2017/01/28 16:06
To further clarify, I'm referencing the Crossfilter & DC APIs alongside the DC NASDAQ and Composite examples. The Composite example has shown me how to place multiple line/bar charts on a single graph. On the composite chart I've created, each of the bar charts I've added a dimension based off of the timestamps in the data-set. Now I'm trying to figure out how to define the groups for each. I want each bar chart to represent the total time worked per timestamp.
For example, I have five people in my database, so I want there to be five bar charts within the single composite chart. Today all five submitted reports saying they worked 8 hours, so now all five bar charts should show a mark at 01/28/2017 on the x-axis and 8 hours on the y-axis.
var parseDate = d3.time.format('%Y-%m-%d %H:%M:%S').parse;
data.forEach(function(d) {
d.timestamp = parseDate(d.timestamp);
});
var ndx = crossfilter(data);
var writtenDimension = ndx.dimension(function(d) {
return d.timestamp;
});
var hoursSumGroup = writtenDimension.group().reduceSum(function(d) {
return d.time_total;
});
var minDate = parseDate('2017-01-01 00:00:00');
var maxDate = parseDate('2017-01-31 23:59:59');
var mybarChart = dc.compositeChart("#my_chart");
mybarChart
.width(window.innerWidth)
.height(480)
.x(d3.time.scale().domain([minDate,maxDate]))
.brushOn(false)
.clipPadding(10)
.yAxisLabel("This is the Y Axis!")
.compose([
dc.barChart(mybarChart)
.dimension(writtenDimension)
.colors('red')
.group(hoursSumGroup, "Top Line")
]);
So based on what I have right now and the example I've provided, in the compose section I should have 5 charts because there are 5 people (obviously this needs to be dynamic in the end) and each of those charts should only show the timestamp: total_time data for that person.
At this point I don't know how to further breakup the group hoursSumGroup based on each person and this is where my Question #1 comes in and I need help figuring out.
Question #2 above is that I want to make sure that the code is both dynamic (more people can be handled without code change), when minDate and maxDate are later tied to user input fields, the charts update automatically (I assume through adjusting the dimension variable in some way), and if I add a names filter that if I unselect names that the chart will update by removing the data for that person.
A Question #3 that I'm now realizing I'll want to figure out is how to get the person's name to show up in the pointer tooltip (the title) along with timestamp and total_time values.
There are a number of ways to go about this, but I think the easiest thing to do is to create a custom reduction which reduces each person into a sub-bin.
First off, addressing question #2, you'll want to set up your dimension based on the time interval you're interested in. For instance, if you're looking at days:
var writtenDimension = ndx.dimension(function(d) {
return d3.time.hour(d.timestamp);
});
chart.xUnits(d3.time.hours);
This will cause each timestamp to be rounded down to the nearest hour, and tell the chart to calculate the bar width accordingly.
Next, here's a custom reduction (from the FAQ) which will create an object for each reduced value, with values for each person's name:
var hoursSumGroup = writtenDimension.group().reduce(
function(p, v) { // add
p[v.name] = (p[v.name] || 0) + d.time_total;
return p;
},
function(p, v) { // remove
p[v.name] -= d.time_total;
return p;
},
function() { // init
return {};
});
I did not go with the series example I mentioned in the comments, because I think composite keys can be difficult to deal with. That's another option, and I'll expand my answer if that's necessary.
Next, we can feed the composite line charts with value accessors that can fetch the value by name.
Assume we have an array names.
compositeChart.shareTitle(false);
compositeChart.compose(
names.map(function(name) {
return dc.lineChart(compositeChart)
.dimension(writtenDimension)
.colors('red')
.group(hoursSumGroup)
.valueAccessor(function(kv) {
return kv.value[name];
})
.title(function(kv) {
return name + ' ' + kv.key + ': ' + kv.value;
});
}));
Again, it wouldn't make sense to use bar charts here, because they would obscure each other.
If you filter a name elsewhere, it will cause the line for the name to drop to zero. Having the line disappear entirely would probably not be so simple.
The above shareTitle(false) ensures that the child charts will draw their own titles; the title functions just add the current name to those titles (which would usually just be key:value).
So this is a question regarding a rather specific problem. As I know from Gordon, main contributor of dc.js, there is no support for elasticY(true) function for logarithmic scales.
So, after knowing this, I tried to implement my own solution, by building a workaround, inside dc.js's renderlet event. This event is always triggered by a click of the user onto the barchart. What I wanted to do is this:
let groupSize = this.getGroupSize(fakeGroup, this.yValue);
let maximum = group.top(1)[0].value;
let minimum = group.top(groupSize)[groupSize-1].value;
console.log(minimum, maximum);
chart.y(d3.scale.log().domain([minimum, maximum])
.range(this.height, 0)
.nice()
.clamp(true));
I thought, that at this point the "fakeGroup" (which is just group.top(50)) contains only the data points that are NOT filtered out after the user clicked somewhere. However, this group always contains all data points that are in the top 50 and doesn't change on filter events.
What I really wanted is get all data points that are NOT filtered out, to get a new maximum and minimum for the yScale and rescale the yAxis accordingly by calling chart.y(...) again.
Is there any way to get only data rows that are still in the chart and not filtered out. I also tried using remove_empty_bins(group) but didn't have any luck with that. Somewhere is always all() or top() missing, even after giving remove_empty_bins both functions.
This is how i solved it:
I made a function called rescale(), which looks like this:
rescale(chart, group, fakeGroup) {
let groupSize = this.getGroupSize(fakeGroup, this.yValue);
let minTop = group.top(groupSize)[groupSize-1].value;
let minimum = minTop > 0 ? minTop : 0.0001;
let maximum = group.top(1)[0].value;
chart.y(d3.scale.log().domain([minimum, maximum])
.range(this.height, 0)
.nice()
.clamp(true));}
I think the parameters are pretty self-explanatory, I just get my chart, the whole group as set by dimension.group.reduceSum and a fake group I created, which contains the top 50 elements, to reduce bar count of my chart.
The rescale() method is called in the event listener
chart.on('preRedraw', (chart) => {
this.rescale(chart, group, fakeGroup);
}
So what I do is re-defining (re-setting min and max values regarding filtered data) the charts yAxis everytime the chart gets redrawn, which happens to also be every time one of my charts is filtered. So now, the scale always fits the filtered data the chart contains after filtering another chart.
I'm using code similar to that in the dc.js annotated example:
var ndx = crossfilter(data);
...
var dayName=["0.Sun","1.Mon","2.Tue","3.Wed","4.Thu","5.Fri","6.Sat"];
var dayOfWeek = ndx.dimension(function (d) {
var day = d.dd.getDay();
return dayName[day];
});
var dayOfWeekGroup = dayOfWeek.group();
var dayOfWeekChart = dc.rowChart("#day-of-week-chart");
dayOfWeekChart.width(180)
.height(180)
.group(dayOfWeekGroup)
.label(function(d){return d.key.substr(2);})
.dimension(dayOfWeek);
The issue I've got is that only days of the week present in the data are displayed in my rowChart, and there's no guarantee every day will be represented in all of my data sets.
This is desirable behaviour for many types of categories, but it's a bit disconcerting to omit them for short and well-known lists like day and month names and I'd rather an empty row was included instead.
For a barChart, I can use .xUnits(dc.units.ordinal) and something like .x(d3.scale.ordinal.domain(dayName)).
Is there some way to do the same thing for a rowChart so that all days of the week are displayed, whether present in data or not?
From my understanding of the crossfilter library, I need to do this at the chart level, and the dimension is OK as is. I've been digging around in the dc.js 1.6.0 api reference, and the d3 scales documentation but haven't had any luck finding what I'm looking for.
Solution
Based on #Gordon's answer, I've added the following function:
function ordinal_groups(keys, group) {
return {
all: function () {
var values = {};
group.all().forEach(function(d, i) {
values[d.key] = d.value;
});
var g = [];
keys.forEach(function(key) {
g.push({key: key,
value: values[key] || 0});
});
return g;
}
};
}
Calling this as follows will fill in any missing rows with 0s:
.group(ordinal_groups(dayNames, dayOfWeekGroup))
Actually, I think you are better off making sure that the groups exist before passing them off to dc.js.
One way to do this is the "fake group" pattern described here:
https://github.com/dc-js/dc.js/wiki/FAQ#filter-the-data-before-its-charted
This way you can make sure the extra entries are created every time the data changes.
Are you saying that you tried adding the extra entries to the ordinal domain and they still weren't represented in the row chart, whereas this did work for bar charts? That sounds like a bug to me. Specifically, it looks like support for ordinal domains needs to be added to the row chart.