I have very long array of arrays.
arr[97] =[['2001-03-12',73.3333],['2001-03-13',73.12],['2001-03-14',69.99],['2001-03-15',70.97],['2001-03-16',67.46] ... ];
I'm using jqplot to draw a chart with these values. But I wish to have charts with data restrictions.
Now I'm creating new subarray every time the date restriction was changed.
But it is very unefficnet, takes time and memory.
Can I use only part of already existing array and give it as an agrument for jqplot-chart?
Related
I have a schematiq table containig data I have read from a CSV and parsed into columns. I want to create myself a data link to a range object representing each column so I can then pass those into multiple stats functions.
I've tried =rng.Subset(B5,1,1) on the table which I had hoped would create a range for the first column, but in the views I see I this results in a 1x1 range containing the whole table.
What's the correct syntax?
When you return an array from a Schematiq function, there are two possible forms for that array to take. One is a "range", i.e. a data-linked object that will exist in a single cell regardless of its size, the other is an array function result, which will fill whatever range an array formula has been entered in.
The easiest way to get all the values out of a table as an array is to use tbl.GetValues(), which if you use it without a column argument just returns a rectangular array containing all the data in the table.
However, this is returned as an array function result, so in order to put it into a range you will need to use rng.Create() on that result. The following should work for you in this case:
=rng.Create(tbl.GetValues(B5))
Or, for a specific column:
=rng.Create(tbl.GetValues(B5, "ColumnName"))
Heres my problem. I have a working dc.js based dashboard and some data within it. A column of the data contains text data (twitter info). Is it somehow possible to extract that specific column from crossfilter? My aim is to create some charts and the crossfilter containing the text data should feed into a d3 based word cloud so that i can do the drill down based filtering as well which dc and crossfilter provide out of the box. I tried a dimension.top(infinity) but that returns all the key value pairs in the data. I just need the values for a particular key across the whole data set. I hope my question makes some sense.
EDIT:
More research reveals that the wordcloud will accept data in key value pair where the key is the word and value is its frequency of appearance. So i am guessing that will need to be implemented as well. If there is a ready to implement library out there kindly let me know as well. This changes things a bit as far as crossfilter is concerned.I need to throw this calculated key value pair (fit for the word cloud consumption) whenever a filter is triggered. How to go about it?
Looking forward to hearing from you all.
Best,
Anmol
Answer to the first part of the question: Probably dimension.top(Infinity) and then use an accessor to get the values you need. Not exactly efficient, but it is what it is.
Answer to the 2nd part of the question:
You need groupAll, I think. You want to take a tweet, generate an array of tokens (words), then generate a Crossfilter grouping that is a count per word, right? You can code your own custom crossfilter.dimension.groupAll reducers (if you want to do that, create a working example and I can probably cook it up). Or if you want to use Reductio:
tweetWords = data.dimension(function(d) { return d.tweetText.split(' '); });
wordCounts = tweetWords.groupAll();
reducer = reductio()
.groupAll(function(d) {
return d.tweetText.split(' ');
})
.count(true);
reducer(wordCounts);
wordCounts.all();
If you want to filter on this dimension you'll have to override the filter handler and check if the group key is in the dimension array for the record using a filterFunction.
I have two datasets having two fields each. In a table I have added two fields from one data set and one column from another dataset. I have used Lookup function to add the field from another database. But when I am calculating the Sum of this lookup function by Sum (Lookup(....)) Its giving me blank.
I have also tried the SumLookup() method but its still giving me blank.
Please suggest what needs to be done.
I have built an MVC 5 application, using EF 6 to query the database. One page show a cross table of two dimensions: substances against properties of these substances. It is rendered as an html table.Many cells do not have a value. This is what it looks like:
sub 1 sub 2 sub 3
prop A 1.0
prop B 1.5 X
prop C 0.6 Y
The cell values are actually more complex, including tool tips, footnotes, etc.
I implemented the generation of the html table, by the following steps:
create a list of unique properties;
create a list of unique substances;
loop through the properties;
render a row for each;
loop through the substances;
See if there is a value for the combination of property and substances;
render the cell's value or an empty one.
Using the ANTS performance profiler, I found out that step 6 has a huge performance issue with increasing numbers of substances and properties, the hit count exploding to hundreds of millions, with a few hundred substances and a few tens of properties (the largest selection the user can make). The execution time is many minutes. It seems to scale N(substances)^2 * N(properties)^2.
The code looks like:
Value currentValue =
values.Where(val => val.substance.Id == currentSubstanceId
&& val.property.Id == currentPropertyId).SingleOrDefault();
where values is a List and Value is an entity, which I read from to render the cells. values had been pre-loaded from the database and no queries are shown by the SQL Server Profiler.
Since not all cells have a value, I thought it best to loop through the row and columns and see if there is a value. I cannot just loop through the list of values.
What could I try to improve this? I thought about:
Create some sort of C# object, using the substance.Id and property.Id as a compound key and fill it from the List object. Which would be fastest?
Create some Linq query which returns an object which already contains the empty cells, like (substance cross join properties) left join values. I could do this in SQL easily, but could this be done with Linq? Could the object which stores the result have the Value as a member field, so I can still use it to render the cells?
Stop pre-loading and just run a database query for the value of each combination, possibly benefiting from database indexes.
I am considering restricting the number of substances and properties the user may select, but I would rather not do that.
Addtional info
As requested by C.Zonnenberg, some more info about the query.
The query to fill the list of values is basically as follows:
I create an IQueryable to which I add filters for requested substances and properties. I then include the substances, property and value details, found in related entities. I then execute query.ToList(). The actual SQL query, as seen by the SQL Profiler looks complex, involving SubstanceId IN () and PropertyId IN (), but it takes far less then a second to execute.
It returns a list of proxies, like: {System.Data.Entity.DynamicProxies.SubstancePropertyValue_078F758A4FF9831024D2690C4B546F07240FAC82A1E9D95D3826A834DCD91D1E}
I think your best bet is your first option. But to do that efficiently I would also modify the source data (values) and turn it into a dictionary, so you have a structure that's optimized for indexed lookup:
var dict = values.ToDictionary(e =>
Tuple.Create(e.substance.id, e.propertyid),
e => e.Value);
Then for each cell:
Value currentValue ;
dict.TryGetValue(Tuple.Create(currentSubstanceId, currentPropertyId),
out currentValue );
Further, you may benefit from parallelization by fetching the cell values in a Parallel.ForEach looping through all substances, for instance.
I'm fairly new to the more complex parts of Core Data.
My application has a core data store with 15K rows. There is a single entity.
I need to display a subset of those rows in a table view filtered on a calculated search criteria, and for each row displayed add a value that I calculate in real time but don't store in the entity.
The calculation needs to use a couple of values supplied by the user.
A hypothetical example:
Entity: contains fields "id", "first", and "second"
User inputs: 10 and 20
Search / Filter Criteria: only display records where the entity field "id" is a prime number between the two supplied numbers. (I need to build some sort of complex predicate method here I assume?)
Display: all fields of all records that meet the criteria, along with a derived field (not in the the core data entity) that is the sum of the "id" field and a random number, so each row in the tableview would contain 4 fields:
"id", "first", "second", -calculated value-
From my reading / Googling it seems that a transient property might be the way to go, but I can't work out how to do this given that the search criteria and the resultant property need to calculate based on user input.
Could anyone give me any pointers that will help me implement this code? I'm pretty lost right now, and the examples I can find in books etc. don't match my particular needs well enough for me to adapt them as far as I can tell.
Thanks
Darren.
The first thing you need to do is to stop thinking in terms of fields, rows and columns as none of those structures are actually part of Core Data. In this case, it is important because Core Data supports arbitrarily complex fetches but the sqlite store does not. So, if you use a sqlite store your fetches are restricted those supported by SQLite.
In this case, predicates aimed at SQLite can't perform complex operations such as calculating whether an attribute value is prime.
The best solution for your first case would be to add a boolean attribute of isPrime and then modify the setter for your id attribute to calculate whether the set id value is prime or not and then set the isPrime accordingly. That will be store in the SQLite store and can be fetched against e.g. isPrime==YES &&((first<=%#) && (second>=%#))
The second case would simply use a transient property for which you would supply a custom getter to calculate its value when the managed object was in memory.
One often overlooked option is to not use an sqlite store but to use an XML store instead. If the amount of data is relatively small e.g. a few thousand text attributes with a total memory footprint of a few dozen meg, then an XML store will be super fast and can handle more complex operations.
SQLite is sort of the stunted stepchild in Core Data. It's is useful for large data sets and low memory but with memory becoming ever more plentiful, its loosing its edge. I find myself using it less these days. You should consider whether you need sqlite in this particular case.