I have a list of objects that represent points in a line chart that grows as data is fetched and can exceed more than 20,000 (twenty thousand) objects.
But the graph can only display 1000 points at a time, for this I did the following function:
private pointPush(model: ViewModel, data: Point[]) {
model.values.push(...data);
if (model.values.length > 1000) {
model.values.splice(0, model.values.length - 1000);
}
}
but I realize that when it exceeds 1000 points it gets very slow, because I think that internally the splice function created a new array.
You can do something similar to a buffer ring that has a constant size but the data access pointer will spin as new data is added overwriting older ones.
Remembering that I'm using an ObservableArray with nativescript-ui.
Related
I want to show the most recent 10 bins for box plot.
If a filter is applied to the bar chart or line chart, the box plot should show the most recent 10 records according to those filters.
I made dimension by date(ordinal). But I am unable to get the result.
I didn’t get how to do it with a fake group. I am new to dc.js.
The pic of scenario is attached. Let me know if anyone need more detail to help me.
in image i tried some solution by time scale.
You can do this with two fake groups, one to remove the empty box plots, and one to take the last N elements of the resulting data.
Removing empty box plots:
function remove_empty_array_bins(group) {
return {
all: function() {
return group.all().filter(d => d.value.length);
}
};
}
This just filters the bins, removing any where the .value array is of length 0.
Taking the last N elements:
function cap_group(group, N) {
return {
all: function() {
var all = group.all();
return all.slice(all.length - N);
}
};
}
This is essentially what the cap mixin does, except without creating a bin for "others" (which is somewhat tricky).
We fetch the data from the original group, see how long it is, and then slice that array from all.length - N to the end.
Chain these fake together when passing them to the chart:
chart
.group(cap_group(remove_empty_array_bins(closeGroup), 5))
I'm using 5 instead of 10 because I have a smaller data set to work with.
Demo fiddle.
This example uses a "real" time scale rather than ordinal dates. There are a few ways to do ordinal dates, but if your group is still sorted from low to high dates, this should still work.
If not, you'll have to edit your question to include an example of the code you are using to generate the ordinal date group.
I'm trying to implement a live data visualization (i.e. with new data arriving periodically) using dc.js. The problem I'm having is the following - when new data is added to the plot, already existing points often start to "dance around", even though they were not changed. Can this be avoided?
The following fiddle illustrates this.
My guess is that crossfilter sorts data internally, which results in points moving on the chart for data items that changed their position (index) in the internal storage. Data is added in the following way:
var data = [];
var ndx = crossfilter(data)
setInterval(function() {
var value = ndx.size() + 1;
if (value > 50) {
return;
}
var newElement = {
x: myRandom(),
y: myRandom()
};
ndx.add([newElement]);
dc.redrawAll();
}, 1000);
Any ideas?
I stand by my comments above. dc.js should be fixed by binding the data using a key function, and probably the best way to deal with the problem is just to disable transitions on the scatterplot using .transitionDuration(0)
However, I was curious if it was possible to work around the current problems by keeping the group in a set order using a fake group. And it is indeed, at least for this example where there is no aggregation and we just want to display the original data points.
First, we add a third field, index, to the data. This has to order the data in the same order in which it comes in. As noted in the discussion above, the scatter plot is currently binding data by its index, so we need to keep the points in a set order; nothing should be inserted.
var newElement = {
index: value,
x: myRandom(),
y: myRandom()
};
Next, we have to preserve this index through the binning and aggregation. We could keep it either in the key or in the value, but keeping it in the key seems more fitting:
xyiDimension = ndx.dimension(function(d) {
return [+d.x, +d.y, d.index];
}),
xyiGroup = xyiDimension.group();
The original reduction didn't make sense to me, so I dropped it. We'll just use the default behavior, which counts the number of rows which fall into each bin. The counts should be 1 if included, or 0 if filtered out. Including the index in the key also ensures uniqueness, which the original keys were not guaranteed to have.
Now we can create a fake group that keeps everything sorted by index:
var xyiGroupSorted = {
all: function() {
var ret = xyiGroup.all().slice().sort((a,b) => a.key[2] - b.key[2]);
return ret;
}
}
This will fetch the original data whenever it's requested by the chart, create a copy of the array (because the original is owned by crossfilter), and sort it to return it to the correct order.
And voila, we have a scatter plot that behaves the way it should, even though the data has gone through crossfilter.
Fork of your fiddle: https://jsfiddle.net/gordonwoodhull/mj81m42v/13/
[After all this, maybe we shouldn't have given the data to crossfilter in the first place! We could have just created a fake group which exposes the original data. But maybe there's some use to this technique. At least it proves that there's almost always a way to work around any problems in dc.js & crossfilter.]
I'm starting to go into random world generating, I have an idea on how random number generating works (Actually pseudorandom numbers), but I don't have a good idea of how to make the world look "nice", in other words not just place blocks based on a random x, y that it gives me, but make it look smooth.
This will be a 1 time generation per world. So everything is created at start.
I was thinking of an algorithm a few moments ago, but the problem is that it would just use be an endless amount of nested if loops, which would probably take a more than the necessary time. I was thinking of the following:
Choose a random location on the map and place the spawn point in that location.
Start building the street based on the spawn location, like if the spawn location is 16
spaces near the edge of the world build a house, otherwise start building a street.
Based on the previously generated street's place structures around.
Place misc.
Conceptualizing the algorithm isn't much of a problem, what I'm having difficulty with is starting the actual code from step 2 and below. Based on the above algorithm or an algorithm you think of, how would you start the code? I'm not asking for the actual code to be made, just an idea of how it would look.
I know this question isn't precise and can have multiple answers, but I've seen many questions similar to this one having a strange approach.
hmm looks like planar(or cubic) map filling. from my point of view firstly you need some databases
struct _object
{
string name,help,info; // texts for later use
int siz[3]; // grid size of object
int pos[3]; // placement pos (center or what ever)
// other stuff like: mesh-id,color,hit points,...
};
struct _dependency
{
int objid
List<int> near; // can be near this objects (can add probability)
List<int> far; // cannot be near this objects (can add probability,min distance)
};
List<_object> object; // DBS of all object types
List<_dependency> depend; // DBS of object dependency
Then you need to initialize this DBS from ini files or whatever. After that you need to create world map. For simplicity let it by only a single squared town and single layer(no sub-terrain), size and placement can be random.
List<_object> obj; // DBS of placed objects, could be lesser derivate of previous _object to conserve memory requirements
const int N=100;
int[N][N] map; // placement map, containing placed obj id, or -1 for empty space
so now you need some town generating function that fills map[N][N]:
void genere()
{
int i,j,x,y,xx,yy,objid,objix;
int _min_complexity=N/10; // this can also be random
int _max_complexity=N; // this can also be random
// clear map
for (i=0;i<N;i++)
for (j=0;j<N;j++)
map[i][j]=-1;
int complexity=_min_complexity+random(_max_complexity-_min_complexity);
for (i=0;i<complexity;)
{
// random placenet position
x=random(N);
y=random(N);
// random object, should take in mind object[].near and closest objects in map[y][x]
objid=random(object.num);
if (check if map[][] is empty enough to place object[objid] to x,y,z)
if (check if near x,y position is not bad type of object already object[].far)
{
// add new object to list
objix=obj.add(object[objid]);
// add new object to map
int *siz=obj[objix].siz
int *pos=obj[objix].pos
x+=pos[0];
y+=pos[1];
for (yy=0;yy<siz[1];yy++)
for (xx=0;xx<siz[0];xx++)
map[y+yy][x+xx]=objid;
i++;
}
}
}
also the position can be double[3] + orientation matrix and map coordinates will than be aligned to grid. There are many ways to tweak this code, its just an starting template. Hope it helps a little.
P.S.
List<type> l;
l.num - number of items in list
l[i] - i item from list
i=l.add(a) - add new item a to list and returns its index
I am trying to use d3 to animate text using an svg text with d3 transitions. I have it working as desired for a single string.
I want to iterate through strings from an array of json objects.
I can do this as well.
All the painting and transitions work great. Problem is, they all happen at once, and appear piled up on each other, and all animate all at once.
I have tried putting them in a setTimeout() to get them to appear sequentially.
Still does not work.
for ( i in haikuStr ) {
if( i !=0 ){
//Make it wait if an appropriate time it is not the first one
setTimeout( function() {
showText();
}, 11000 * i );
} else {
//if i=0, don't make folks wait
showText();
}
}
The showText() function is the full create container -> finish transitions.
I use 11000 * i to ensure that >2 iterations have 11 additional seconds per i.
I have spent quite a bit of time reading and trying to figure out how to get the loop to pause before cycling through to paint the next line.
Any thoughts or ideas would be appreciated.
The un-timed example is here, if you wish to see the text jumble up:
http://www.mysalmagundi.com/js/svg-d3-no-timing.html
Have you read Thinking with Joins? Or some of the other introductory D3 tutorials, such as those by Scott Murray? Or Three Little Circles, or Working with Selections? I ask because your showText function is misusing data joins; it creates text elements for every element in the global haikuStr array:
var text = haikuContainer.selectAll("text")
.data(haikuStr)
.html(String)
.enter().append("text");
And all your text elements are overlapping because you set them to have the same y-attribute:
var thisHaiku = text
.attr("x", -800)
.attr("y", 120)
(Also, that selection.html call is a no-op because the update selection is guaranteed to be empty, since you just created haikuContainer it is guaranteed to not have any descendant text elements. And thisHaiku is the same value as the var text, because when method chaining selection.attr and similar methods return the current selection; so there’s no reason to create a separate var. Also, you shouldn’t use a for-in loop to iterate over arrays.)
If you wait 11 seconds, you’ll see the second SVG appear, but because of your data join as described above, it has the same overlapping text content.
If you just want to show a single piece of text, then pass that string to your showText function (e.g., showText("hello")). Then, since you’re just creating individual elements, just selection.append them rather than using a data-join. You only need the data-join when you’re creating (or updating or removing) a variable number of elements based on data; in this case it looks like you’re trying to just create a single element.
I have a line chart. Its purpose is to show the amount of transactions per user over a given time period.
To do this I'm getting the dates of all users transactions. I'm working off this example : http://bl.ocks.org/mbostock/3884955 and have the line chart renedering fine.
My x-axis is time and the y-axis is number of transactions. The problem I have is to do with displaying dates when there is no activity.
Say I have 4 transactions on Tuesday and 5 transactions on Thursday..I need to show that there has been 0 transactions on Wednesday. As no data exists in my database explicitly stating that a user has made no transactions on Wedensday do I need to pass in the Wednesday time (and all other times, depending on the timeframe) with a 0 value? or can I do it with d3? I can't seem to find any examples that fit my problem.
This seems like a pretty common issue, so I worked up an example implementation here: http://jsfiddle.net/nrabinowitz/dhW2F/2/
Relevant code:
// get the min/max dates
var extent = d3.extent(data, function(d) { return d.date; }),
// hash the existing days for easy lookup
dateHash = data.reduce(function(agg, d) {
agg[d.date] = true;
return agg;
}, {}),
// note that this leverages some "get all headers but date" processing
// already present in the example
headers = color.domain();
// make even intervals
d3.time.days(extent[0], extent[1])
// drop the existing ones
.filter(function(date) {
return !dateHash[date];
})
// and push them into the array
.forEach(function(date) {
var emptyRow = { date: date };
headers.forEach(function(header) {
emptyRow[header] = null;
});
data.push(emptyRow);
});
// re-sort the data
data.sort(function(a, b) { return d3.ascending(a.date, b.date); });
As you can see, it's a bit convoluted, but seems to work well - you make an array of evenly spaced dates using the handy d3.interval.range method, filter out those dates already present in your data, and use the remaining ones to push empty rows. One downside is that performance could be slow for a big dataset - and this assumes full rows are empty, rather than different empty dates in different series.
An alternate representation, with gaps (using line.defined) instead of zero points, is here: http://jsfiddle.net/nrabinowitz/dhW2F/3/