DC.js, crossfilter - generic grouping with varying number of columns - d3.js

I'm trying to create a generic composite chart that would take any csv file, read the columns and create a composite line chart with one line per column.
In this use case, the first column is always TimeStamp in all csv's and this will not be changed, while the rest of the columns may vary.
To get the column names from any csv, I am using the following line of code.
var mappedArray = d3.entries(data[0]);
Say, my mappedArray is like this :
["Date", "c1", "c2", "c3"]
I have created the Dimension for timestamp as follows:
var DateDim = ndx.dimension(function(d){return timeFmt(d[mappedArray[0].key]); });
For a single valued group, i would use something like this:
var testGrp = DateDim.group().reduceSum(function(d){return +d[mappedArray[1].key]; });
For a multi valued group, I tried something like this:
var testGrp1 = testDim.group().reduce(
function(p,v){
++p.count;
p.col1 = +v[mappedArray[1].key];
p.col2 = +v[mappedArray[2].key];
...
p.coln = +v[mappedArray[n].key];
return p;
},...
But, I cannot hard code like this, as the 'n' in mappedArray[n] may keep changing.
How can I recreate this group by using mappedArray.length?

You recreate by using the index field operator []. Use a for loop to create as many p.colX fields as needed.
var testGrp1 = testDim.group().reduce(
function(p,v){
++p.count;
for (let i = 1; i < mappedArray.length; i++) {
p["col" + i] = +v[mappedArray[i].key];
}
return p;
},...
Edit
The answer to your question in the comment about the chart.compose.
A possible way of doing is to create a function that constructs the array needed.
function makeComposeArray() {
var composeArr = [];
for(let i=1; i<=mappedArray.length; i++) {
composeArr.push(
dc.lineChart(chart)
.group(testGrp1, function(d){return d['col'+i]})
.valueAccessor(function(d){return d.value["col" + i]})
);
}
return composeArr;
}
chart.compose(makeComposeArray());

Related

For loop to check values from two spreadsheets

I have two spreadsheets:
Column A on sheet 6th&7thRoster lists all IDs in a sample, contains 853 items.
Column C on sheet alreadySubmitted contains the IDs of users who've completed a task. Contains 632 items.
I'm trying to parse through both columns. If a user from Column A of sheet 6th&7thRoster matches a user from Column C of sheet sandboxAlreadySubmitted, I want to write the word "Yes" on Column I of the current row of sheet 6th&7thRoster. When using the code below, I'm not seeing not seeing any instances of the word "Yes" on Column I of 6th&7thRoster, even though I know there's multiple places where that should be the case.
function checkRoster() {
var mainSheet = SpreadsheetApp.openById('XXXXXXX');
var roster = mainSheet.getSheetByName('6th&7thRoster');
var submissions = mainSheet.getSheetByName('alreadySubmitted');
var rosterLastRow = roster.getLastRow();
var submissionsLastRow = submissions.getLastRow();
var rosterArray = roster.getRange('A2:A853').getValues();
var submissionsArray = submissions.getRange('C2:C632').getValues;
var i;
var x;
for (i = 1; i < 853; i++) {
for (x = 1; x < 632; x++){
if (rosterArray[i] == submissionsArray[x]){
roster.getRange(i, 9).setValue("Yes");
}
}
}
}
Feedback on how to solve and achieve this task will be much appreciated. For confidentiality, I cannot share the original sheets.
You want to compate the values of A2:A853 of 6th&7thRoster and C2:C632 of alreadySubmitted.
When the values of C2:C632 of alreadySubmitted are the same with the values of A2:A853 of 6th&7thRoster, you want to put Yes to the column "I".
If my understanding is correct, how about this modification? Please think of this as just one of several possible answers.
Modified script:
function checkRoster() {
var mainSheet = SpreadsheetApp.openById('XXXXXXX');
var roster = mainSheet.getSheetByName('6th&7thRoster');
var submissions = mainSheet.getSheetByName('alreadySubmitted');
var rosterLastRow = roster.getLastRow();
var submissionsLastRow = submissions.getLastRow();
var rosterArray = roster.getRange('A2:A853').getValues();
var submissionsArray = submissions.getRange('C2:C632').getValues(); // Modified
// I modified below script.
var obj = submissionsArray.reduce(function(o, [v]) {
if (v) o[v] = true;
return o;
}, {});
var values = rosterArray.map(function([v]) {return [obj[v] ? "Yes" : ""]});
roster.getRange(2, 9, values.length, values[0].length).setValues(values);
}
Flow:
Retrieve values from A2:A853 of 6th&7thRoster and C2:C632 of alreadySubmitted.
Create an object for searching the values from the values of alreadySubmitted.
Create the row values for putting to 6th&7thRoster.
References:
reduce()
map()
If I misunderstood your question and this was not the direction you want, I apologize.

crossfilter: obtain the count of values falling into the product of two columns

I have a data set like
{"parent":"/home","inside":"/files","filename":"type.jar",
"extension":"jar","type":"modified","archive"}
Likewise many there are many rows in the json array. I am using crossfilter to read the data and plot graphs and datatables. the Type in the data set has values "added", "modified" and "deleted".
I want to create a data table like
Extension | Added | Modified | Deleted
where added, modified and deleted will hold the count of the files with the specific extension. Can anyone suggest me a way to do so?
So far I have created a dimension like this:
var extensionType = facts.dimension(function(d) {
return d.extension; });
var extensionTypeGroup=extensionType.group();
and I get a grouped output like this,
{"key":"class","value":424},
{"key":"js","value":176},
{"key":"properties","value":26},
{"key":"jar","value":10},
{"key":"css","value":8},
{"key":"txt","value":6},
{"key":"war","value":4},
{"key":"png","value":4},
{"key":"handlebars","value":4},
{"key":"jar_local","value":2},
{"key":"aar","value":2}
How do I get the separate count of added deleted and modified?
Probably the easiest way to do this is to reduce to an object rather than a single value.
This is covered in the FAQ: How do I reduce multiple values at once? What if rows contain a single value but a different value per row? You probably just needed the right search terms to find it.
Actually it looks like the code from the FAQ will work for you unmodified:
var extensionTypeGroup = extensionType.group().reduce(
function(p, v) { // add
p[v.type] = (p[v.type] || 0) + v.value;
return p;
},
function(p, v) { // remove
p[v.type] -= v.value;
return p;
},
function() { // initial
return {};
});

Date Validation with If/Then Function in Google Apps Script

Thanks already to Serge insas for his insight both here and here, which have been a godsend for me already. But...I'm having trouble tying everything together with date validation.
To clarify, I have a GAS intended to verify that the date in Column A is (a) more than seven days old and (b) not null. If both pass, the script determines the first empty row in Column G, and then pauses before completing various functions. The beginning of the script looks like...
function getStats() {
var doc = SpreadsheetApp.getActiveSpreadsheet();
var sheet = doc.getSheetByName("Main");
var TITLE_ROW = 1;
var DATE_COL = 1;
var URL_COL = 4;
var sevendaysBefore = new Date(new Date().getTime()-7*24*60*60*1000);
if (DATE_COL != ''||(DATE_COL != null || DATE_COL< sevendaysBefore)) {
var end = sheet.getLastRow();
for( var i = 1; i < end; i++) {
var Gvals = sheet.getRange("G1:G").getValues();
var Glast = Gvals.filter(String).length;
var rowNum = TITLE_ROW+Glast;
var itemurl = sheet.getRange(rowNum,URL_COL).getValues();
Utilities.sleep(500);
...
I've clearly implemented something wrong, though, because the date validation doesn't work—the script appears to function as though the data in Column A doesn't matter. I'm sure I've done something incredibly idiotic, but I'm too ignorant to spot it on my own. So...anyone know what I've overlooked?
While the other answer is probably working (didn't test), its approach is very different from yours.
Below is code that follows the same logic as yours but works at the array level (to follow recommendations in Best practices).
I added a few comments to show the differences, hoping it will help you to understand how it works.
function getStats() {
var doc = SpreadsheetApp.getActiveSpreadsheet();
var sheet = doc.getSheetByName("Main");
var Glast; // define the variable for later use
var vals = sheet.getDataRange().getValues();// get all data in an array (do that before loop)
var TITLE_ROW = 0;// use array index instead of real row numbers
var DATE_COL = 0;// use array index instead of real column numbers
var URL_COL = 3;// use array index instead of real column numbers
var sevendaysBefore = new Date(new Date().getTime()-7*24*60*60*1000).getTime();// get native value in milliseconds to make comparison easier below
for( var i = 1; i < vals.length; i++) { // start loop from Row 2 (=array index 1)
if(vals[i][0]!='' && vals[i][0]!=null&&vals[i][0].getTime()<sevendaysBefore){continue};// use && instead of ||, we want ALL conditions to be true ( see !='' and !=null)
Glast = i; break ;// first occurrence of data meeting above condition (non null and date < 7 days before)
}
var itemurl = vals[Glast][URL_COL];// get the value from the array
Utilities.sleep(500);
//...
Mistake : You are hard coding DATE_COL = 1 and you are using this in if statement. It doesn't get the value of the cell. Also I am not getting your statement "date in Column A is (a) more than seven days old". Is that date is from a cell or you are iterating through all the cells in column A ?.
Below code will satisfy your need and I tested. Here as example I am checking date validation for cell R1C1(A1).
1)Get the date from cell. You can change it or Iterate the cells in column for date.
2) We have date.valueOf() method which returns the number of milliseconds since midnight 1970-01-01.
3) Validation : check the cell data is date and greater than 7 days
function compDate()
{
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getActiveSheet();
var cell = sheet.getRange("A1"); //point1
var date01 = new Date();
var date02 = cell.getValue(); //point2
var dateDiff = (date01.valueOf()-date02.valueOf())/(24*60*60*1000);
if((isValidDate(date02)) == true && dateDiff > 7) //point3
Logger.log("success");
}
//below function will return true if the arg is valid date and false if not.
function isValidDate(d) {
if ( Object.prototype.toString.call(d) !== "[object Date]" )
return false;
return !isNaN(d.getTime());
}

How to populate the table dynamically and correctly with Ajax?

I have a form where the user submits a query and then have a Servlet that processes this query and returns the results in XML. With this result trying to populate a table dynamically via Ajax, for such, I use the following code below.
var thead = $("<thead>");
var rowsTHead = $("<tr>");
var tbody = $("<tbody>");
var numberOfColumns;
$(xml).find("head").each(function(){
var variable = $(this).find("variable");
numberOfColumns = variable.length;
for (var i = 0; i < variable.length; i++){
var name = $(variable[i]).attr("name");
rowsTHead.append($("<th>").html(name));
}
});
thead.append(rowsTHead);
$(xml).find("result").each(function(){
var literal = $(this).find("literal");
var rowsTBody = $("<tr class=\"even\">");
literal.length = numberOfColumns;
for (var j = 0; j < literal.length; j++){
var tdBody = $("<td>");
tdBody.html($(literal[j]).text());
rowsTBody.append(tdBody);
}
tbody.append(rowsTBody);
});
$(".tablesorter").empty()
.append(thead)
.append(tbody);
This code works perfectly until it was used in a UNION query. When using a UNION the returned xml comes in the following way http://pastebin.com/y7hXK1Zy
As can be observed, this query has 4 variables that are: gn1, indication1, gn2, indication2.
What is going wrong is that the values of all the variables being written in columns corresponding to gn1 and indication1.
What I wish I was to write the value of each variable in its corresponding column. I wonder what should I change in my code to make this possible.
You need to respect the name values of the binding elements, and relate them back to the columns that you correctly built from parsing the element. When you are doing the find "literal", you are skipping the parsing of the binding elements. You should find "binding", respect the name and look up which column to use based on that, and then for each of those, find the "literal" elements for the actual values.

Row number in LINQ

I have a linq query like this:
var accounts =
from account in context.Accounts
from guranteer in account.Gurantors
where guranteer.GuarantorRegistryId == guranteerRegistryId
select new AccountsReport
{
recordIndex = ?
CreditRegistryId = account.CreditRegistryId,
AccountNumber = account.AccountNo,
}
I want to populate recordIndex with the value of current row number in collection returned by the LINQ. How can I get row number ?
Row number is not supported in linq-to-entities. You must first retrieve records from database without row number and then add row number by linq-to-objects. Something like:
var accounts =
(from account in context.Accounts
from guranteer in account.Gurantors
where guranteer.GuarantorRegistryId == guranteerRegistryId
select new
{
CreditRegistryId = account.CreditRegistryId,
AccountNumber = account.AccountNo,
})
.AsEnumerable() // Moving to linq-to-objects
.Select((r, i) => new AccountReport
{
RecordIndex = i,
CreditRegistryId = r.CreditRegistryId,
AccountNumber = r.AccountNo,
});
LINQ to objects has this builtin for any enumerator:
http://weblogs.asp.net/fmarguerie/archive/2008/11/10/using-the-select-linq-query-operator-with-indexes.aspx
Edit: Although IQueryable supports it too (here and here) it has been mentioned that this does unfortunately not work for LINQ to SQL/Entities.
new []{"aap", "noot", "mies"}
.Select( (element, index) => new { element, index });
Will result in:
{ { element = aap, index = 0 },
{ element = noot, index = 1 },
{ element = mies, index = 2 } }
There are other LINQ Extension methods (like .Where) with the extra index parameter overload
Try using let like this:
int[] ints = new[] { 1, 2, 3, 4, 5 };
int counter = 0;
var result = from i in ints
where i % 2 == 0
let number = ++counter
select new { I = i, Number = number };
foreach (var r in result)
{
Console.WriteLine(r.Number + ": " + r.I);
}
I cannot test it with actual LINQ to SQL or Entity Framework right now. Note that the above code will retain the value of the counter between multiple executions of the query.
If this is not supported with your specific provider you can always foreach (thus forcing the execution of the query) and assign the number manually in code.
Because the query inside the question filters by a single id, I think the answers given wont help out. Ofcourse you can do it all in memory client side, but depending how large the dataset is, and whether network is involved, this could be an issue.
If you need a SQL ROW_NUMBER [..] OVER [..] equivalent, the only way I know is to create a view in your SQL server and query against that.
This Tested and Works:
Amend your code as follows:
int counter = 0;
var accounts =
from account in context.Accounts
from guranteer in account.Gurantors
where guranteer.GuarantorRegistryId == guranteerRegistryId
select new AccountsReport
{
recordIndex = counter++
CreditRegistryId = account.CreditRegistryId,
AccountNumber = account.AccountNo,
}
Hope this helps.. Though its late:)

Resources