I am trying to develop a webapp using Google Apps Script to be embedded into a Google Site which simply displays the contents of a Google Sheet and filters it using some simple parameters. For the time being, at least. I may add more features later.
I got a functional app, but found that filtering could often take a while as the client sometimes had to wait up to 5 seconds for a response from the server. I decided that this was most likely due to the fact that I was loading the spreadsheet by ID using the SpreadsheetApp class every time it was called.
I decided to cache the spreadsheet values in my doGet function using the CacheService and retrieve the data from the cache each time instead.
However, for some reason this has meant that what was a 2-dimensional array is now treated as a 1-dimensional array. And, so, when displaying the data in an HTML table, I end up with a single column, with each cell being occupied by a single character.
This is how I have implemented the caching; as far as I can tell from the API reference I am not doing anything wrong:
function doGet() {
CacheService.getScriptCache().put('data', SpreadsheetApp
.openById('####')
.getActiveSheet()
.getDataRange()
.getValues());
return HtmlService
.createTemplateFromFile('index')
.evaluate()
.setSandboxMode(HtmlService.SandboxMode.IFRAME);
}
function getData() {
return CacheService.getScriptCache().get('data');
}
This is my first time developing a proper application using GAS (I have used it in Sheets before). Is there something very obvious I am missing? I didn't see any type restrictions on the CacheService reference page...
CacheService stores Strings, so objects such as your two-dimensional array will be coerced to Strings, which may not meet your needs.
Use the JSON utility to take control of the results.
myCache.put( 'tag', JSON.stringify( myObj ) );
...
var cachedObj = JSON.parse( myCache.get( 'tag' ) );
Cache expires. The put method, without an expirationInSeconds parameter expires in 10 minutes. If you need your data to stay alive for more than 10 minutes, you need to specify an expirationInSeconds, and the maximum is 6 hours. So, if you specifically do NOT need the data to expire, Cache might not be the best use.
You can use Cache for something like controlling how long a user can be logged in.
You could also try using a global variable, which some people would tell you to never use. To declare a global variable, define the variable outside of any function.
I never quite understood the if needed part of the description.
.fetchAll()
Fetches the given list of Parse.Object.
.fetchAllIfNeeded()
Fetches the given list of Parse.Object if needed.
What is the situation where I might use this and what exactly determines the need? I feel like it's something super elementary but I haven't been able to find a satisfactory and clear definition.
In the example in the API, I notice that the fetchAllIfNeeded() has:
// Objects were fetched and updated.
In the success while the fetchAll only has:
// All the objects were fetched.
So does the fetchAllIfNeeded() also save stuff too? Very confused here.
UPDATES
TEST 1
Going on some of the hints #danh left in the comments I tried the following things.
var todos = [];
var x = new Todo({content:'Test A'}); // Parse.Object
todos.push(x);
x.save();
// So now we have a todo saved to parse and x has an id. Async assumed.
x.set({content:'Test B'});
Parse.Object.fetchAllIfNeeded(todos);
So in this scenario, my client x is different than the server. But the x.hasChanged() is false since we used the set function and the change event is triggered. fetchAllIfNeeded returns no results. So it isn't that it's trying to compare this outright to what is on the server to sync and fetch.
I notice that in the request payload, running the fetchAllIfNeeded is sending the following interesting thing.
{where: {objectId: {$in: []}}, _method: "GET",…}
So it seems that on the clientside something determines whether an object isNeeded
Test 2
So now, based on the comments I tried manipulating the changed state of the object by setting with silent.
x.set({content:'Test C'}, {silent:true});
x.hasChanged(); // true
Parse.Object.fetchAllIfNeeded(todos);
Still nothing interesting. Clearly the server state ("Test A") is different than clientside ("Test C"). and I still results [] and the request payload is:
{where: {objectId: {$in: []}}, _method: "GET",…}
UPDATE 2
Figured it out by looking at the Parse source. See answer.
After many manipulations, then taking a look at the source - I figured this out. Basically fetchAllIfNeeded will fetch models in an array that have no data, meaning there are no attribute properties and values.
So the use case would be you have lets say a parent object with an array of nested Parse Objects. When you fetch the parent object, the nested child objects in the array will not be included (unless you have the include query constraint set). Instead, the pointers are sent back to clientside and in your client, those pointers are translated into 'empty' models with no data, basically just blank Parse.Objects with ids.
Specifically, the Parse.Object has an internal Boolean property called _hasData which seems to be toggled true any time stuff like set, or fetch, or whatever gives that model attributes.
So, lets say you need to fetch those child objects. You can just do something like
var childObjects = parent.get('children'); // Array
Parse.Object.fetchAllIfNeeded(childObjects);
And it will search for those children who are currently only represented as empty Objects with id.
It's useful as opposed to fetchAll in that you might go through the children array and lazily load one at a time as needed, then at a later time need to "get the rest". fetchAllIfNeeded essentially just filters "the rest" and sends a whereIn query that limits fetching to those child objects that have no data.
In the Parse documentation, they have a comment in the callback response to fetchAllIfNeeded as:
// Objects were fetched and UPDATED.
I think they mean the clientside objects were updated. fetchAllIfNeeded is definitely sending GET calls so I doubt anything updates on the serverside. So this isn't some sync function. This really confused me as I instantly thought of serverside updating when they really mean:
// Client objects were fetched and updated.
I am trying to develop a webapp using Google Apps Script to be embedded into a Google Site which simply displays the contents of a Google Sheet and filters it using some simple parameters. For the time being, at least. I may add more features later.
I got a functional app, but found that filtering could often take a while as the client sometimes had to wait up to 5 seconds for a response from the server. I decided that this was most likely due to the fact that I was loading the spreadsheet by ID using the SpreadsheetApp class every time it was called.
I decided to cache the spreadsheet values in my doGet function using the CacheService and retrieve the data from the cache each time instead.
However, for some reason this has meant that what was a 2-dimensional array is now treated as a 1-dimensional array. And, so, when displaying the data in an HTML table, I end up with a single column, with each cell being occupied by a single character.
This is how I have implemented the caching; as far as I can tell from the API reference I am not doing anything wrong:
function doGet() {
CacheService.getScriptCache().put('data', SpreadsheetApp
.openById('####')
.getActiveSheet()
.getDataRange()
.getValues());
return HtmlService
.createTemplateFromFile('index')
.evaluate()
.setSandboxMode(HtmlService.SandboxMode.IFRAME);
}
function getData() {
return CacheService.getScriptCache().get('data');
}
This is my first time developing a proper application using GAS (I have used it in Sheets before). Is there something very obvious I am missing? I didn't see any type restrictions on the CacheService reference page...
CacheService stores Strings, so objects such as your two-dimensional array will be coerced to Strings, which may not meet your needs.
Use the JSON utility to take control of the results.
myCache.put( 'tag', JSON.stringify( myObj ) );
...
var cachedObj = JSON.parse( myCache.get( 'tag' ) );
Cache expires. The put method, without an expirationInSeconds parameter expires in 10 minutes. If you need your data to stay alive for more than 10 minutes, you need to specify an expirationInSeconds, and the maximum is 6 hours. So, if you specifically do NOT need the data to expire, Cache might not be the best use.
You can use Cache for something like controlling how long a user can be logged in.
You could also try using a global variable, which some people would tell you to never use. To declare a global variable, define the variable outside of any function.
I am using libRets for .NET, and querying http://retsgw.flexmls.com/rets2_1/, using a valid user account. From C#, after calling Search() I check the count using GetCount() and I get 6300 results, but when I call HasNext() the first time returns false.
Checking the XML response, it looks like the results are empty () even though the result count provides a number.
So... where did the results go?
The exact query is the following:
http://retsgw.flexmls.com/rets2_1/Search?Class=OpenHouse&Count=1&QueryType=DMQL2&SearchType=OpenHouse&Select=ListingID&StandardNames=1
Here is the request:
SearchRequest request = client.CreateSearchRequest("OpenHouse", "OpenHouse", "");
request.SetStandardNames(true);
request.SetSelect("ListingID");
Here is how the request is made:
SearchResultSet result = client.Search(request);
Here is how the result is handled:
while (result.HasNext()) {
// Do something
}
So, it looks like the FlexMLS Support was able to help (rather quickly).
I needed to add &Format=COMPACT-DECODED to the query string.
So, in the code it would look like this:
request.SetFormatType(SearchRequest.FormatType.COMPACT_DECODED);
1) You are setting StandardNames to true AND then setting a selection. That selection may not exist in StandardNames. (You're reviewed the metadata returned by the server, right?) Its possible the server doesn't take the select into account when doing the count, but does on a full query, therefor it doesn't have any information to send back because it doesn't have a Table matching what you selected. What happens when you don't set the Select?
2) Have you done a packet trace or set libRETS to log the network traffic to a file? (I can't tell if that's what you mean by "Checking the XML response, it looks like the results are empty () even though the result count provides a number.") If you haven't, do that and see if the server is passing back any information.
If the server is passing pack information, you might have discovered a bug in libRETS and I invite you to join the libRETS-users mailing list and share this data (and that network trace) there.
If the server is passing back 0 results, you'll may need to contact your MLS and/or FlexMLS to see if you don't have permissions to view the results. Some RETS servers have fine grained results where you could get the count, but not get the data.
This question is about load data embedded into DOM structure.
I am using jQuery2, but the question is valid for any other framework or single Javascript code.
There are two scenarios:
When the data is load once (with the page), no "refresh data" is need.
When data is refreshed by some event.
The average performance can be changed with either one or other
Suppose a typical case of scenario-2, where a page fragment must be reloaded, with new HTML and new data. So, the $('#myDiv').load('newHtmlFragment') will be used any way... And, for jQuery programmer, using AJAX, there are two ways to load an "DOM-based data":
by HTML: expressing all data into the "newHtmlFragment" HTML. Suppose many paragraphs, each like <p id="someId" data-X="someContent">...more content</p>. There are some "verbose overhead" for each data-X1="contentX1" data-X2="contentX2" ..., and is not elegant for webservice script if it is not an XHTML-oriented one (I am using PHP, my data is an array, and I preffer to use json_encode).
by jQuery evaluation: using the same $('#myDiv').load('newHtmlFragment') only for <p id="someId">...more content</p>, with no data-X. A second AJAX load an jQuery script like $('#someId').data(...) and evaluate it. So this is an overhead, for node-selection and data-inclusion, but with big item-data, each data can be enconded by JSON.
by pure JSON: similar to "by jQuery", but the second AJAX load an JSON object like var ALLDATA={'someId1':{...}, 'someId2':{...}, ...}. So this is an overhead for a static function that executes something like $('#myDiv p').each(function(){... foreach ... $(this).data('x',ALLDATA[id]['x']);}) retrive selection, but with big data, all data can be enconded by JSON.
The question: what the best choice? It depends on scenarios or another context parameter? There are signfificative performance tradeoffs?
PS: a complete answer needs to address the issue of performance... If no significative performance differences, the best choice relies on "best programming style" and software engineering considerations.
More context, if you need as reference to answer. My practical problem is at scenario-1, and I am using the second choice, "by jQuery script", executing:
$('#someId1').data({'bbox':[201733.2,7559711.5,202469.4,7560794.9],'2011-07':[3,6],'2011-08':[2,3],'2011-10':[4,4],'2012-01':[1,2],'2012-02':[12,12],'2012-03':[3,6],'2012-04':[6,12],'2012-05':[3,4],'2012-06':[2,4],'2012-07':[3,5],'2012-08':[10,11],'2012-09':[7,10],'2012-10':[1,2],'2012-12':[2,2],'2013-01':[6,10],'2013-02':[19,26],'2013-03':[2,4],'2013-04':[5,8],'2013-05':[4,5],'2013-06':[4,4]});
$('#someId2').data({'bbox':[197131.7,7564525.9,198932.0,7565798.1],'2011-07':[39,51],'2011-08':[2,3],'2011-09':[4,5],'2011-10':[13,14],'2011-11':[40,42],'2011-12':[21,25],'2012-01':[10,11],'2012-02':[26,31],'2012-03':[27,35],'2012-04':[8,10],'2012-05':[24,36],'2012-06':[4,7],'2012-07':[25,30],'2012-08':[9,11],'2012-09':[42,52],'2012-10':[4,7],'2012-11':[17,22],'2012-12':[7,8],'2013-01':[21,25],'2013-02':[5,8],'2013-03':[8,11],'2013-04':[28,40],'2013-05':[55,63],'2013-06':[1,1]});
$('#...').data(...); ... more 50 similar lines...
This question can be discussed from different aspects. Two that I can think of right now would be software engineering and end-user experience. A comprehensive solution covering first can also cover the later but as usually it's not possible to come up with such a solution (due to its cost) these two hardly overlap.
Software engineering point of view
In this POV it is strongly suggested that different parts of the system to be as isolated as possible. This means you better postpone the marriage of data and view as late as you can. It helps you to divide your developers into two separate groups; those who know server-side programming and have no idea how HTML (or any other interface layer technology) works and those who are solely experienced on HTML and Javascript. Just this division alone is a blessing for management and it helps greatly in big projects where teamwork is essential. It also helps the maintenance and expansion of the system, all the things software engineering aims at.
User experience point of view
As good as the previous solution sounds, it comes with (solvable) drawbacks. As you mentioned in your question, if we are to load view and data separately, it elevates the number of requests we have to send to server to retrieve them. It imposes two problems, first the overhead that comes with each request and second the delay user has to wait for each request to be responded. The second is more obvious so let's start with that. With all the advances to the Internet and bandwidths, yet our users' exceeding expectations enforce us to consider this delay. One way to reduce the number of requests would be your first choice: data within HTML fragments. Multiple number of requests also has an overhead problem as well. This can be explained by HTTP protocol's handshake (both on client-side and server-side) and by the fact that each request will lead to loading the session on server which in a large scale could be pretty considerable. So again your first option could be the answer to this problem.
Tie breaker
Having both sides of the story said, what then? The ultimate solution is a combination of both where data and view are married on client but they are downloaded at the same time. To be honest I don't know of such a library. But the principle is simple, you need a mechanism to package data and empty HTML fragments within the same response and combine them into what user will see on client. This solution is costy (to implement) but it is sort of the cost that once paid you can benefit from it for a life time.
It depends on how you are going to use the data that is stored. Consider this example:
Example
You have 300 items in a database (say server access logs of the last 300 days). Now you want to display 300 <div> tags, each tag representing one database-item.
Now there are 2 options to do this:
<div data-stat1="stat1" data-stat2="stat2" data-stat3="stat3">(...)</div>
<!-- again, this is repeated 300 times -->
<script>
// Example on how to show "stat1" value in all <div>s
function showStat1() {
for(var i=1; i<=300; i+= 1) {
var theID = '#id-' + i;
jQuery(theID).text(jQuery(theID).data('stat1'));
}
}
</script>
OR
<div id="id-1">(...)</div>
<!-- repeat this 300 times, for all DB items -->
<script>
data = { // JSON data which is displayed in the <div> tags
'1': ['stat1', 'stat2', 'stat3'],
// same for all 300 items
}
// Example on how to show "stat1" value in all <div>s
function showStat1() {
for(var i=1; i<=300; i+= 1) {
var theID = '#id-' + i;
jQuery(theID).text(data[i][0]);
}
}
</script>
Which scenario is better?
Case 1:
The data is directly coded into the DOM elements, which makes this one an easy to implement solution. You can see in my examples that the first code block takes a lot less code to generate and the code is less complex.
This solution is very powerful, when you want to store data that is directly related to an DOM element (as you do not have to create a logical connection between JSON and DOM): It is very simple to access data for the correct element.
However, your main concern is performance - this is NOT the way to go. Because every time you access data, you have to first select the correct DOM element and attribute via javascript, which takes considerable amount of time. So the simplicity costs you a lot of performance overhead when you want to read/write data in the element.
Case 2:
This scenario very cleanly separates the DISPLAY from the DATA storage. It has major advantages compared to the first case.
A) Data should not be mingled with the display elements - imagine you want to switch from <div> to <table> and suddenly you have to rewrite all javascript to correctly select the correct table cells
B) Data is directly accessible without traversing the DOM tree. Imagine you want to calculate an average sum of all values. In first case you need to loop all DOM elements and read value from them. In the second you simply loop a normal array
C) The data can be loaded or refreshed via ajax. You only transfer the JSON object but not all the HTML stuff that is required for displaying the data
D) Performance is way better, mostly because of the above mentioned points. Javascript is a lot better in handling simple arrays or (not too complex) JSON objects than it is in filtering data out of the DOM tree.
In general this solution is more than twice as fast as the first case. You will also discover - even though it is not obvious - that the second case is also easier to code and maintain! The code simply is easier to read and understand, as you clearly separate the data from the UI elements...
Performance comparison
You can compare the two solutions on this JSPerf scenario:
http://jsperf.com/performance-on-data-storage
Taking it further
For implementation of the second case I generally use this approach:
I generate the HTML code which will serve as the UI. It often happens that I have to generate HTML via javascript, but now I assume that the DOM tree will not change after the page is loaded
At the end of the HTML I include a tag with a JSON object for initial page display
Via jQuery onReady event I then parse the JSON object and update the UI elements as required (e.g. populating the table with data)
When data is loaded dynamically I simply transfer a new JSON object with the new data and use exact the same javascript function as in step 3 to display the new results.
Example of the HTML file:
<div>ID: <span class="the-id">-</span></div>
<div>Date: <span class="the-date">-</span></div>
<div>Total page hits: <span class="the-value">-</span></div>
<button type="button" class="refresh">Load new data</button>
<script src="my-function.js"></script>
<script>
function initial_data() {
return {"id":"1", "date":"2013-07-30", "hits":"1583"};
}
</script>
The "my-function.js" file:
jQuery(function initialize_ui() {
// Initialize the global variables we will use in the app
window.my_data = initial_data();
window.app = {};
app.the_id = jQuery('.the-id');
app.the_date = jQuery('.the-date');
app.the_value = jQuery('.the-value');
app.btn_refresh = jQuery('.refresh');
// Add event handler: When user clicks refresh button then load new data
app.btn_refresh.click(refresh_data);
// Display the initial data
render_data();
});
function render_data() {
app.the_id.text(my_data.id);
app.the_date.text(my_data.date);
app.the_value.text(my_data.hits);
}
function refresh_data() {
// For example fetch and display the values for date 2013-07-29
jQuery.post(url, {"date":"2013-07-29"}, function(text) {
my_data = jQuery.parseJSON(text);
render_data();
});
}
I did not test this code. Also there is crucial error handling and other optimization missing. But it helps to illustrate the concepts that I try to describe