How to reduce time of script execution with FormApp...getResponses()? - performance

The Email collection function built into the form is in the Text format. We have a lot of employees filling out this form, so it would be more convenient to choose from a drop-down list than to manually enter an email for each employee. The built-in function of collecting emails does not allow using the Dropdown list question type, so I had to disable it and send the response to the email using a script.
In addition, when you select one of the options (Transfer) in the form, you need to insert an additional correction row with data. This is also done by the script.
The script works fine, but unfortunately, its execution time reaches 45 seconds. I think this is abnormally large.
Updated code:
var FORM_ID = '#####';
var SHEET_NAME = 'Operations';
function sendFormToEmail() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
console.time('section10'); // 25579ms
var sheet = ss.getSheetByName(SHEET_NAME);
console.timeEnd('section10');
// Open a form by ID and log the responses to each question.
var form = FormApp.openById(FORM_ID);
var formResponses = form.getResponses();
var i = formResponses.length - 1;
var formResponse = formResponses[i]; // Last item
var itemResponses = formResponse.getItemResponses();
var length = itemResponses.length;
Logger.log("length = " + length);
var emailTo = itemResponses[0].getResponse(); // Returns the email if given
var cp = itemResponses[1].getResponse(); // Counterparty
var subject = "Input form: "+ cp;
var datePay = itemResponses[2].getResponse(); // Date of the operation
var dateAccept = itemResponses[3].getResponse(); // Date of acceptance
var sum = itemResponses[4].getResponse() ; // Amount
var what = itemResponses[5].getResponse(); // Operation type
var comm = "Correction " + itemResponses[length - 1].getResponse(); // Last response
var sum_1 = parseFloat(sum.replace(/,/, '.')) * (-1); // Amount * (-1)
var timestamp = new Date();
var textBody = "Operation: " + timestamp + ";\n";
for (var j = 0; j < itemResponses.length; j++) {
var itemResponse = itemResponses[j];
var resp = itemResponse.getResponse();
textBody += itemResponse.getItem().getTitle() + "=" + resp + ";\n";
}
if (what == 'Transfer') { // If the transfer between your accounts
var lr = sheet.getLastRow();
sheet.insertRowBefore(lr);
var values = [datePay, dateAccept, "", sum_1, "", "", "", "", comm, "", "", "", what, timestamp, "", "", ""];
Logger.log(values);
console.time('section12'); // 19449ms
sheet.getRange(lr, 2, 1, 17).setValues([values]);
console.timeEnd('section12')
}
if(emailTo !== undefined){
GmailApp.sendEmail(emailTo, subject, textBody);
}
}
At first I thought it was the large size of the acceptance sheet (about 13 thousand rows). I created a copy of the table, reduced it to several dozen rows, but the speed did not increase much.
Then I deleted the answers from the form (there were just under 9000 of them) - the same thing, I didn't get much performance gain.
Anyone have any ideas how to change the script algorithm to improve its performance?
===
Conclusions
Thanks to #TheMaster for the help with console.time(). Thanks to this tool I found bottlenecks in the code. More precisely, the code works well, but it's about the structure of the spreadsheets system with which the code interacts. The structure needs to be optimized.
There was also an idea addressed, probably, to Google developers. It would be great if there was a tool that visually (graphically) displays the relationships between spreadsheets and sheets that make up a single system. Perhaps with some kind of numerical characteristics that reflect, for example, the interaction time between its blocks. This would make it possible to quickly eliminate such bottlenecks in the system and improve it.

Investigation:
Pinpointing the issue is done using console.time() and console.timeEnd() of each section of code and calculating the time taken by reachl each section of code.
Issue:
As discussed in the question comment chain, This is due to a bloated spreadsheet with many sheets and import formulas. This caused more time to get the exact sheet and to use setValues:
console.time('section10'); // 25579ms
var sheet = ss.getSheetByName(SHEET_NAME);
console.timeEnd('section10');
//....
console.time('section12'); // 19449ms
sheet.getRange(lr, 2, 1, 17).setValues([values]);
console.timeEnd('section12')
Possible solutions:
Reduce number of sheets
Reduce interconnected spreadsheets (=import* formulas)
Delete empty rows on the bottom and empty columns on the right of each sheet.

Related

What is the most efficient way to get borders in Google Apps Script

What is the best way to get border details in Google Apps Script?
I cannot see anything with borders in the documentation for GAS, so I have had to resort to getting borders through the Spreadsheet API.
This works ok, other than when the number of borders gets large where it will take a long time to return, or not return at all.
Is there a better way to do this?
var fieldsBorders = 'sheets(data(rowData/values/userEnteredFormat/borders))';
var currSsId = SpreadsheetApp.getActiveSpreadsheet().getId();
var activeSheet = SpreadsheetApp.getActiveSheet();
var name = activeSheet.getName();
var data = Sheets.Spreadsheets.get(currSsId, {
ranges: name,
fields: fieldsBorders
});
You want to reduce the process cost for retrieving the borders from a sheet in Spreadsheet.
When I set the borders for 26 x 1000 cells and run your script, the process time was about 50 s in my environment.
For this situation, you want to reduce the cost more.
Your reply comment is perhaps it was a bigger sheet it was on, either way 50s is a long time to get the borders. The other calls to GAS take a very small amount of time to complete. Can you confirm this is the only way to get borders?
If my understanding is correct, how about this workaround? In this workaround, I request directly to the endpoint of Sheets API for retrieving the borders.
Workaround:
Sample situation
In this sample script, as a sample situation, I supposes the default sheet which has 26 columns x 1000 rows, and the borders are set to all cells.
Sample script 1:
In this sample script, the borders are retrieved by one API call.
var ss = SpreadsheetApp.getActiveSpreadsheet();
var fileId = ss.getId();
var sheetName = ss.getActiveSheet().getName();
var token = ScriptApp.getOAuthToken();
var fields = "sheets/data/rowData/values/userEnteredFormat/borders";
var params = {
method: "get",
headers: {Authorization: "Bearer " + token},
muteHttpExceptions: true,
};
var range = sheetName + "!A1:Z1000";
var url = "https://sheets.googleapis.com/v4/spreadsheets/" + fileId + "?ranges=" + encodeURIComponent(range) + "&fields=" + encodeURIComponent(fields);
var res = UrlFetchApp.fetch(url, params);
var result = JSON.parse(res.getContentText());
Result:
When the sample script 1 was used, the average process time was 2.2 seconds.
Although I'm not sure about the internal process of Sheets API of Advanced Google Service, it was found that when it directly requests to the endpoint, the process cost can be reduced.
Sample script 2:
In this sample script, the borders are retrieved with the asynchronous process by several API calls.
var sep = 500; // Rows retrieving by 1 request.
var ss = SpreadsheetApp.getActiveSpreadsheet();
var fileId = ss.getId();
var sheetName = ss.getActiveSheet().getName();
var token = ScriptApp.getOAuthToken();
var fields = "sheets/data/rowData/values/userEnteredFormat/borders";
var requests = [];
var maxRows = 1000;
var row = 1;
for (var i = 0; i < maxRows / sep; i++) {
var range = sheetName + "!A" + row + ":Z" + (row + sep - 1);
requests.push({
method: "get",
url: "https://sheets.googleapis.com/v4/spreadsheets/" + fileId + "?ranges=" + encodeURIComponent(range) + "&fields=" + encodeURIComponent(fields),
headers: {Authorization: "Bearer " + token},
});
row += sep;
}
var response = UrlFetchApp.fetchAll(requests);
var result = response.reduce(function(ar, e) {
var obj = JSON.parse(e.getContentText());
Array.prototype.push.apply(ar.sheets[0].data[0].rowData, obj.sheets[0].data[0].rowData);
return ar;
}, {sheets: [{data: [{rowData: []}]}]});
Result:
When the sample script 2 was used, the following results were obtained.
When sep is 500 (in this case, 2 API calls are run.), the average process time was 1.9 seconds.
When sep is 200 (in this case, 5 API calls are run.), the average process time was 1.3 seconds.
But if the number of requests in one run are increased, the error related to the over of quotas occurs. Please be careaful this.
Note:
This is a simple sample for testing above situation. So I think that above script cannot be used for all situations. If you use above sample script, please modify it for your situation.
References:
fetchAll(requests)
Benchmark: fetchAll method in UrlFetch service for Google Apps Script

Google Sheets - Multiple dependent drop-down lists

I am trying to create a dependent list as described and answered (with a script) here.
I would like to achieve that if selecting a certain value (e.g. "First") from a cell in column 1, then the drop-down options from the next cell in the same row should offer a range of values from the column in a different sheet with the same heading as the value in the first - left - cell (i.e. the first sheet is called "Selector" - in which there are dropdowns, in the second sheet called "KAT" I have the options for these dropdowns). This should then be possible for every row depending on the value of each first cell of the row.
I have tried to use and adapt the suggested script and have reviewed the sample files in the article but I apparently lack some basic understanding of the script to be able to adapt and implement it properly.
Could anybody kindly help me with making this dynamic dropdown work properly?
Just to clarify my final intention: I would like to have this script working first to be able to use it on multiple files. My final goal, though, is to make self-filling dropdown lists and selectors, so that I could simply fill in the data in the "Selector" sheet and would then be able to select these same values later in the cells below (depending on the name (value) of the first cell in the row = first cell of the column holding validation range). I hope to be able to achieve this by using either Pivot table or any other formula in the "KAT" sheet that would aggregate my data from "Selector" sheet and feed them back as drop-down options ...).
Thank you for your help.
See the example sheet here
Code I used (as above):
function onEdit()
{
var ss = SpreadsheetApp.getActiveSpreadsheet(),
sheet = ss.getActiveSheet(),
name = sheet.getName();
if (name != 'Selector') return;
var range = sheet.getActiveRange(),
col = range.getColumn();
if (col != 1) return;
var val = range.getValue(),
dv = ss.getSheetByName('KAT'),
data = dv.getDataRange().getValues(),
catCol = data[0].indexOf(val),
list = [];
Logger.log(catCol)
for (var i = 1, len = 100; i < len; i++) // Problem is here, you have too many items in list! Cannot have more 500 items for validation
list.push(data[i][catCol]);
var listRange = dv.getRange(2,catCol +1,dv.getLastRow() - 1, 1)
Logger.log(list)
var cell = sheet.getRange(range.getRow(), col-1)
var rule = SpreadsheetApp.newDataValidation()
.requireValueInRange(listRange) // Use requireValueIn Range instead to fix the problem
.build();
cell.setDataValidation(rule);
Logger.log(cell.getRow())
}
This question deals with dynamic dropdown lists. A previous question and answer on StackOverflow (Google Sheets - Dependent drop-down lists) were referenced, and code from that answer was being unsuccessfully re-purposed.
The code in the question was not working for one reason: Line 20
var cell = sheet.getRange(range.getRow(), col-1)
In the referenced code, the dropdown list begins in Column F (col=6). The dependant dropdowns ranged to the left so the definition of the dependant column was "col-1". In the questioner's scenario, the dropdown list begins in Column A (col=1) and the dependant dropdowns range from left to right. However, this line of code was not changed to take into account the different layout. Rather than "col-1", it should be "col+1".
Other matters
In addition to this, lines 16 and 17 perform a loop to create an array that might be used for the dependant dropdown. However the loop is redundant because the dropdown is actual defined by creating and assigning a range on the "KAT" sheet.
Cell A2 of KAT includes a formula:
=sort(unique(Selector!$A$2:$A),1,true)
This may appear to be useful because it automatically adds any new dropdown value entered in "Selector" to a list of values in KAT. In reality it is unproductive, because the dependant dropdown build by the code works vertically rather than horizontally. So an additional row added to KAT does not, of itself, contribute to building the dependant dropdown.
The following code works to build the dependant drop down list. I have deliberately left a number of "Logger" entries in the code to assist the questioner in understanding how the code works.
function onEdit() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getActiveSheet();
var name = sheet.getName();
if (name != 'Selector') return;
var range = sheet.getActiveRange();
var col = range.getColumn();
var dropdownrow = range.getRow(); // added for debugging and informationm
if (col != 1) return;
var val = range.getValue();
Logger.log("the cursor is in 'Selector' in cell = " + range.getA1Notation()); //DEBUG
Logger.log("That's row " + dropdownrow + ", and column " + col + ". The value selected = " + val); // DEBUG
var dv = ss.getSheetByName('KAT');
var data = dv.getDataRange().getValues();
var catCol = data[0].indexOf(val);
var list = [];
var KAT_data = dv.getDataRange();
var KAT_data_len = KAT_data.getLastRow(); // added to give 'for' loop a sensible range
Logger.log("The data range on KAT is " + KAT_data.getA1Notation() + ", and the last row of data = " + KAT_data_len); //DEBUG
Logger.log("KAT data = '" + data + "'"); // DEBUG
Logger.log("Found the dropdown cell value of '" + val + "' in KAT as item #" + catCol); //DEBUG
for (var i = 1, len = KAT_data_len; i < len; i++) { // Irrelevant because the data validation range is obtained by defining a range on KAT
// Problem is here, the unique command in A2 creates a blank row
// Logger.log("i="+i+", data = "+data[i][catCol]); // DEBUG
list.push(data[i][catCol]);
}
var listRange = dv.getRange(2, catCol + 1, dv.getLastRow() - 1, 1);
Logger.log("FWIW, this is the list after the loop= " + list); // DEBUG
Logger.log("The contents for the new data validation range (taken from KAT) is " + listRange.getA1Notation()); // DEBUG
Logger.log("The new validation range gets added to col = " + (col + 1)); // DEBUG
//var cell = sheet.getRange(range.getRow(), col-1); // governs the next validation range. Example validation worked right to left, but this sheet works left to right. So must ADD 1, not subtract 1.
var cell = sheet.getRange(range.getRow(), col + 1);
Logger.log("The cell to be assigned the new validation range will be " + cell.getA1Notation()); // DEBUG
var rule = SpreadsheetApp.newDataValidation().requireValueInRange(listRange).build(); // Build validation rule
cell.setDataValidation(rule); // assign validation range to new cell
}
Is this code worthwhile?
The code, as written and referenced, is limited to creating only one level of dependant dropdowns. To this extent it has very limited value. A different approach to creating dependant dropdowns is justified.
"How do you do dynamic / dependent drop downs in Google Sheets?" on StackOverflow has been a meeting place for discussing and updating techniques for dynamic dependant dropdowns since 2014. The latest update was in February 2018 by Max Makhrov. Thye code described here may be useful for the questioner.

Applying Google Apps Script for Dynamic Data Validation to Existing Sheet

So I'm using this script (credit to Chicago Computer Classes) for populating dynamic data validation of a Google Sheets cell based on what the user entered in a different cell.
For example, if they enter the sport "Football" in one cell, the next cell has data validation for "CFB, CFL, or NFL" but if they enter "Basketball" in the first cell then the second cell's data validation changes to "ABL, CBB, NBA, or WNBA" for examples.
The script is working fantastic and you are welcome to play with the sheet here
However ... here's my problem:
I have an existing spreadsheet with 9000 rows of data. I would like to apply this new data validation scheme to this spreadsheet. The script is triggered with the onEdit() function which works great when you are entering things one row at a time. But if I try to copy and paste a whole bunch of rows in the first column, only the first row of the second column triggers the onEdit and gets the new data validation while all the other rows of the second column are unchanged. I've also tried to "Fill Down" or "Fill Range" on the first column and they have the same result where the first row in the selected range gets the new data validation but the rest of the selection is unchanged.
And while it would work just fine if I was manually entering rows, I really don't feel like doing that 9000 times :)
How do I modify the script to trigger the function with data that's copy/pasted or filled down?
Thanks!
Script here:
function onEdit(){
var tabLists = "Leagues";
var tabValidation = "2018";
var ss = SpreadsheetApp.getActiveSpreadsheet().getActiveSheet();
var datass = SpreadsheetApp.getActiveSpreadsheet().getSheetByName(tabLists);
var activeCell = ss.getActiveCell();
if(activeCell.getColumn() == 6 && activeCell.getRow() > 1 && ss.getSheetName() == tabValidation){
activeCell.offset(0, 1).clearContent().clearDataValidations();
var makes = datass.getRange(1, 1, 1, datass.getLastColumn()).getValues();
var makeIndex = makes[0].indexOf(activeCell.getValue()) + 1;
if(makeIndex != 0){
var validationRange = datass.getRange(3, makeIndex, datass.getLastRow());
var validationRule = SpreadsheetApp.newDataValidation().requireValueInRange(validationRange).build();
activeCell.offset(0, 1).setDataValidation(validationRule);
}
}
}
You should use the event object, which will provide you with the range that was edited. What you're doing now is looking only at the "active cell", which doesn't leverage the benefits of the event object, and can also lead to bugginess when you make rapid changes.
Using the event object, when you make an edit to multiple cells at once (from copy/paste), you can then loop through the range and set your validations.
function onEdit(e) {
var editedRange = e.range;
var ss = editedRange.getSheet();
var tabValidation = "2018";
if(editedRange.getColumn() == 6 && editedRange.getRow() > 1 && ss.getSheetName() == tabValidation) {
var tabLists = "Leagues";
var tabListsSheet = e.source.getSheetByName(tabLists);
var makes = tabListsSheet.getRange(1, 1, 1, tabListsSheet.getLastColumn()).getValues(); // This won't change during execution, so call only once
var activeCell = editedRange.getCell(1,1); // Start with the first cell
var remainingRows = editedRange.getHeight();
while(remainingRows > 0) {
var cellValue = activeCell.getValue();
activeCell.offset(0, 1).clearContent().clearDataValidations(); // Always clear content & validations
if (cellValue != "") { // Add validations if cell isn't blank
var makeIndex = makes[0].indexOf(cellValue) + 1;
if(makeIndex != 0) {
var validationRange = tabListsSheet.getRange(3, makeIndex, tabListsSheet.getLastRow()-2);
var validationRule = SpreadsheetApp.newDataValidation().requireValueInRange(validationRange).build();
activeCell.offset(0, 1).setDataValidation(validationRule);
}
}
activeCell = activeCell.offset(1, 0); // Get the next cell down
remainingRows--; // Decrement the counter
}
}
}

Is there a way to mass input data validation in google sheets

I'm trying to create a drop down menu with contents based on a another cell in the same row. For example if A1 = 'yes' then the drop down in B2 gives you the options of 'yes' or 'no'. I can do this I have the list data set up and to code works. The problem is I need to do this 155 times in 4 different sheets. Is there a faster way to do this than right clicking and editing the data validation rules for each cell. Here's a link to the test sheet I'm working on :
https://docs.google.com/spreadsheets/d/1rd_Ig_wpof9R_L0IiA1aZ9syO7BWxb6jvBhPqG8Jmm4/edit?usp=sharing
You can set data validation rules with a script, as documented here. Here's a reference for starting with Apps scripts.
I wrote a function that does approximately what you described. It works with the range B3:B157 of the sheet '9th grade' in the current spreadsheet. For each of them, it sets the validation rule to be: a value in the same row, columns B and C of sheet 'List Data'. The line with
....... = listData.getRange(i+3, 2, 1, 2);
will need to be modified if the source range of validation is to be different. Here, the parameters are: starting row, starting column, number of rows, number of columns. So, 2 columns starting with the second, in row numbered i+3.
function setRules() {
var ss = SpreadsheetApp.getActiveSpreadsheet();
var grade = ss.getSheetByName('9th Grade');
var listData = ss.getSheetByName('List Data');
var range = grade.getRange('B3:B157');
var rules = range.getDataValidations();
for (var i = 0; i < rules.length; i++) {
var sourceRange = listData.getRange(i+3, 2, 1, 2);
rules[i][0] = SpreadsheetApp.newDataValidation().requireValueInRange(sourceRange).build();
}
range.setDataValidations(rules);
}
I land in this issue for a diferent reason: "Just mass DataValidation copy (or update) in one column". Thanks, to user3717023 that bring me a light.
I hope that helps someone this simplification.
function setRules() {
//select spreadsheet
var ss = SpreadsheetApp.getActiveSpreadsheet();
var leads = ss.getSheetByName('Leads');
//Select correct Datavalidation
var rangeNewDataValidation = leads.getRange('M2:M2');
var rule = rangeNewDataValidation.getDataValidations();
//Copy (or Update) Datavalidation in a specific (13 or 'M') column
var newRule = rule[0][0].copy();
Logger.log(leads.getMaxRows())
for( var i=3; i <= leads.getMaxRows(); i++){
var range = leads.getRange(i, 13);
range.setDataValidations([[newRule.build()]]);
}
}

Faster Iterative Calculation

I've made a script that performs an iterative calculation to find values based on a gross percentage of project cost.
This is what I have so far:
//<!-- FEE, BOND, & INSURANCE CALCULATOR --!>
function feeBondInsCalculator (){
var ss = SpreadsheetApp.getActiveSpreadsheet();
var sheet = ss.getSheetByName("Pretty Sum Form");
sheet.activate();
// Get all variables from named ranges
var bondValue = ss.getRangeByName("bondValue");
var bondCalc = ss.getRangeByName("bondCalc");
var insValue = ss.getRangeByName("insValue");
var insCalc = ss.getRangeByName("insCalc");
var feeValue = ss.getRangeByName("feeValue");
var feeCalc = ss.getRangeByName("feeCalc");
var interCount = ss.getRangeByName("interCount")
// Interative calculation to paste calculated values into the body of the spreadsheet
for (var i = 0; i<11; i++){
feeCalc.copyTo(ss.getRangeByName("feeValue"), {contentsOnly: true});
bondCalc.copyTo(ss.getRangeByName("bondValue"), {contentsOnly: true});
insCalc.copyTo(ss.getRangeByName("insValue"), {contentsOnly: true});
interCount.setValue(i)
}
}
I used range names so that users can add/delete rows and columns and not have to reset the code. When executed, the code works fine, but takes about two seconds per iteration. Is there a more efficient way to make this work?
Instead of using copy, you might try referencing your ranges directly in a formula like:
=arrayformula(if(bondValue <>"",bondValue,""))

Resources