AngularDart custom filter call() method required to be idempotent? - filter

The main running example of the Angular Dart tutorial is a Recipe Book app. The exercise at the end of the Chapter 5 on filters and services suggests trying to "create a [custom] filter that will multiply all the amounts [of each ingredient listed] in the recipes" thus allowing a "user to double, triple, or quadruple the recipe." E.g. an ingredient of "1/2 cup of flour" would become "1 cup of flour" when doubled.
I have written such a custom filter: it takes a list of Ingredients (consisting of a quantity and a description) and returns a new list of new Ingredients (with increased quantities), but I am getting the following error:
5 $digest() iterations reached. Aborting!
My question is: what is the required and/or permitted behavior of an AngularDart custom filter call() method? E.g., clearly it is permitted to remove (i.e. filter) elements from its input list, but can it also add new or replace elements? The Dart angular.core NgFilter documentation simply says that a "filter is a class with a call method". I have not found more details.
Extrapolating from the answer to this AngularJS post, it would seem that repeated invocations of call() should (eventually?) yield "the same result". If so, this would be a reasonable constraint.
Yielding "the same result" could mean that call() needs to be idempotent, but in the case of Dart such idempotence should be relative to == (object equivalence) not identical() (object identity), IMHO. I ran a few tests using the following small example to illustrate the issues:
main.dart
import 'package:angular/angular.dart';
class A { }
#NgFilter(name:'myFilter') class MutatingCustomFilter {
final A _a = new A();
call(List list) => new List.from(list)..add(_a); // runs ok.
// call(List list) => new List.from(list)..add(new A()); // gives error
}
class MyAppModule extends Module {
MyAppModule() { type(MutatingCustomFilter); }
}
main() => ngBootstrap(module: new MyAppModule());
index.html excerpt
<ul>
<li ng-repeat="x in [1,2,3] | myFilter">{{x}}</li>
</ul>
If I change the body of class A to be
#override bool operator==(other) => true;
#override int get hashCode => 1;
which makes all instances of A considered ==, then the second implementation of call() in main.dart (the one with add(new A())) still gives an error (though a different one).
I can see how to solve the tutorial exercise without use of a custom filter, but I am trying to not give up on the challenge of finding a filter that will work as requested. I am new to Angular and decided to jump in with AngularDart, so any help in explaining the effects of the various flavors of call(), or in finding documentation for the expected behavior of call(), (or letting me know if you think such a custom filter simply cannot be written!) would be appreciated.

Too many iterations
When angular detects a change in the model, it executes a reaction function. The reaction function can further change the model. This would leave the model in inconsistent state. For this reason we re-run the change detection, which can further create more changes. For this reason we keep re-running the changes until the model stabilizes. But how many times should we rerun the change detection before giving up? By default it is 5 times. If the model does not stabilize after 5 iteration we give up. This is what is going on in your case.
Change Detection
When has object changed? one can use identical or == (equals). Good arguments can be made for each, but we have chosen to use identical because it is fast and consistent. Using == (equals) is tricky and it would negatively impact the change detection algorithm.
Filters and arrays
When a filter which operates an an array, executes it has no choice but to create a new instance of the array. This breaks identical, but luckily it is fed into ng-repeat which uses its own algorithm for array contents detection rather the array detection. While the array does not have to be identical between runs, its content must be. Otherwise ng-repeat can not tell the difference between insertions and changes, which it needs to do proper animations.
Your code
The issue with your filter is that it creates new instance on each iteration of the digest loop. These new instances prevent the model from stabilizing and hence the error. (There are plans to solve this issue, but it will be few weeks before we get there.)
Solution
Your solutions is attempting to create a filter which consumes the whole array and then attempts to create a new array, for the ng-repeat. A different (prefered) solution would be to leave the ng-repeat iteration as is, and instead place the filter on the binding which is creating the qty and apply it there.
<span>{{recipe.qty | myFilter:multiply}}</span>

Related

Second parameter in knockout applyBindings. Performance implications

After reading the documentation about the second parameter in knockout applyBindins, I understand the reason behind it:
Optionally, you can pass a second parameter to define which part of
the document you want to search for data-bind attributes. For example,
ko.applyBindings(myViewModel,
document.getElementById('someElementId')). This restricts the
activation to the element with ID someElementId and its descendants,
which is useful if you want to have multiple view models and associate
each with a different region of the page.
But I have not seen anything about performance. My thoughts (which are based on nothing) are that it makes sense that ko bindings will work faster if to restring the binding not to the whole document, but to a small part of it.
So does it make sense to use ko.applyBindings(myViewModel, $('#someElementId')[0]) without having multiple view models just for the sake of performance. (I heard about Mr. Knuth, so it would be nice to refrain from optimization citation).
applyBindings is a thin wrapper over applyBindingsToNodeAndDescendantsInternal (which is where ALL the magic happens) and if you don't supply a node it uses window.document.body
There is no difference between calling applyBindings(viewModel) and applyBindings(viewMode,rootNode) unless you have a very large DOM and only want to bind a small fraction. In this case use the second parameter and Knockout will have a lot less to scan on the initial setup.
Once the bindings are in place they are reacting locally to observable changes, unlike a framework like AngularJS that can potential scan the entire DOM for changes. The root node is irrelevant at this point.
ko.applyBindings = function (viewModel, rootNode) {
if (rootNode && (rootNode.nodeType !== 1) && (rootNode.nodeType !== 8))
throw new Error("ko.applyBindings: first parameter should be your view model; second parameter should be a DOM node");
rootNode = rootNode || window.document.body; // Make "rootNode" parameter optional
applyBindingsToNodeAndDescendantsInternal(viewModel, rootNode, true);
};

Using invariant with Dexterity form and fieldsets

I have a content type derived from plone.directives.form.Schema; it has several dozen fields across four fieldsets. I'm trying to create a zope.interface.invariant that looks at fields from two different fieldsets.
From tracing the behaviour, it looks like the invariant is called once for each fieldset, but not for the entire form.
I'm aware I can provide my own handler and perform all the checks I need there, but that feels chunky compared to distinctly defined invariants. While the obvious solution is to move related fields onto the same fieldset, the current setup reflects a layout that is logical the end user.
Is there an existing hook where I could perform validation on multiple fields across fieldsets?
The answer seems to be no: z3c.form.group.Group.extractData calls z3c.form.form.BaseForm.extractData once for each group/fieldset, and this call already includes invariant validation.
Instead of registering your own handler, you could also overwrite extractData:
from plone.directives import form, dexterity
from z3c.form.interfaces import ActionExecutionError,WidgetActionExecutionError
# ...
class EditForm(dexterity.EditForm):
grok.context(IMyEvent)
def extractData(self, setErrors=True):
data, errors = super(EditForm, self).extractData(setErrors)
if not None in(data['start'], data['end']):
if data['end'] < data['start']:
raise WidgetActionExecutionError('end', Invalid(_(u"End date should not lie before the start date.")))
if data['end'] - data['start'] > datetime.timedelta(days=7):
raise WidgetActionExecutionError('end', Invalid(_(u"Duration of convention should be shorter than seven (7) days.")))
return data, errors
Please note that this class derives from dexterity.EditForm, which includes Dexterity's default handlers, instead of form.SchemaForm.
WidgetActionExecutionError does not work reliably, though. For some fields, it produces a 'KeyError'.

'Maximum number of expressions in a list is 1000' error with Grails and Oracle

I'm using Grails with an Oracle database. Most of the data in my application is part of a hierarchy that goes something like this (each item containing the following one):
Direction
Group
Building site
Contract
Inspection
Non-conformity
Data visible to a user is filtered according to his accesses which can be at the Direction, Group or Building Site level depending on user role.
We easily accomplished this by creating a listWithSecurity method for the BuildingSite domain class which we use instead of list across most of the system. We created another listWithSecurity method for Contract. It basically does a Contract.findAllByContractIn(BuildingSite.listWithSecurity). And so on with the other classes. This has the advantage of keeping all the actual access logic in BuildingSite.listWithsecurity.
The problem came when we started getting real data in the system. We quickly hit the "ora-01795 maximum number of expressions in a list is 1000" error. Fair enough, passing a list of over 1000 literals is not the most efficient thing to do so I tried other ways even though it meant I would have to deport the security logic to each controller.
The obvious way seemed to use a criteria such as this (I only put the Direction level access here for simplicity):
def c = NonConformity.createCriteria()
def listToReturn = c.list(max:params.max, offset: params.offset?.toInteger() ?: 0)
{
inspection {
contract {
buildingSite {
group {
'in'("direction",listOfOneOrTwoDirections)
}
}
}
}
}
I was expecting Grails to generate a single query with joins that would avoid the ora-01795 error but it seems to be calling a separate query for each level and passing the result back to Oracle as literal in an 'in' to query the other level. In other words, it does exactly what I was doing so I get the same error.
Actually, it might be optimising a bit. It seems to be solving the problem but only for one level. In the previous example, I wouldn't get an error for 1001 inspections but I would get it for 1001 contracts or building sites.
I also tried to do basically the same thing with findAll and a single HQL where statement to which I passed a single direction to get the nonConformities in one query. Same thing. It solves the first levels but I get the same error for other levels.
I did manage to patch it by splitting my 'in' criteria into many 'in' inside an 'or' so no single list of literals is more than 1000 long but that's profoundly ugly code. A single findAllBy[…]In becomes over 10 lines of code. And in the long run, it will probably cause performance problems since we're stuck doing queries with a very large amount of parameters.
Has anyone encountered and solved this problem in a more elegant and efficient way?
This won't win any efficiency awards but I thought I'd post it as an option if you just plainly need to query a list of more than 1000 items none of the more efficient options are available/appropriate. (This stackoverflow question is at the top of Google search results for "grails oracle 1000")
In a grails criteria you can make use of Groovy's collate() method to break up your list...
Instead of this:
def result = MyDomain.createCriteria().list {
'in'('id', idList)
}
...which throws this exception:
could not execute query
org.hibernate.exception.SQLGrammarException: could not execute query
at grails.orm.HibernateCriteriaBuilder.invokeMethod(HibernateCriteriaBuilder.java:1616)
at TempIntegrationSpec.oracle 1000 expression max in a list(TempIntegrationSpec.groovy:21)
Caused by: java.sql.SQLSyntaxErrorException: ORA-01795: maximum number of expressions in a list is 1000
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
You'll end up with something like this:
def result = MyDomain.createCriteria().list {
or { idList.collate(1000).each { 'in'('id', it) } }
}
It's unfortunate that Hibernate or Grails doesn't do this for you behind the scenes when you try to do an inList of > 1000 items and you're using an Oracle dialect.
I agree with the many discussions on this topic of refactoring your design to not end up with 1000+ item lists but regardless, the above code will do the job.
Along the same lines as Juergen's comment, I've approached a similar problem by creating a DB view that flattens out user/role access rules at their most granular level (Building Site in your case?) At a minimum, this view might contain just two columns: a Building Site ID and a user/group name. So, in the case where a user has Direction-level access, he/she would have many rows in the security view - one row for each child Building Site of the Direction(s) that the user is permitted to access.
Then, it would be a matter of creating a read-only GORM class that maps to your security view, joining this to your other domain classes, and filtering using the view's user/role field. With any luck, you'll be able to do this entirely in GORM (a few tips here: http://grails.1312388.n4.nabble.com/Grails-Domain-Class-and-Database-View-td3681188.html)
You might, however, need to have some fun with Hibernate: http://grails.org/doc/latest/guide/hibernate.html

Implementing thread-safe, parallel processing

I am trying to convert an existing process in a way that it supports multi-threading and concurrency to make the solution more robust and reliable.
Take the example of an emergency alert system. When a worker clocks-in, a new Recipient object is created with their information and added to the Recipients collection. Conversely, when they clock-out, the object is removed. And in the background, when an alert occurs, the alert engine will iterate through the same list of Recipients (foreach), calling SendAlert(...) on each object.
Here are some of my requirements:
Adding a recipient should not block if an alert is in progress.
Removing a recipient should not block if an alert is in progress.
Adding or removing a recipient should not affect the list of
recipients used by an in-progress alert.
I've been looking at the Task and Parallel classes as well as the BlockingCollection and ConcurrentQueue classes but am not clear what the best approach is.
Is it as simple as using a BlockingCollection? After reading a ton of documentation, I'm still not sure what happens if Add is called while I am enumerating the collection.
UPDATE
A collegue referred me to the following article which describes the ConcurrentBag class and how each operation behaves:
http://www.codethinked.com/net-40-and-system_collections_concurrent_concurrentbag
Based on the author's explanation, it appears that this collection will (almost) serve my purposes. I can do the following:
Create a new collection
var recipients = new ConcurrentBag();
When a worker clocks-in, create a new Recipient and add it to the collection:
recipients.Add(new Recipient());
When an alert occurs, the alert engine can iterate through the collection at that time because GetEnumerator uses a snapshot of the collection items.
foreach (var recipient in recipients)
recipient.SendAlert(...);
When a worker clocks-out, remove the recipient from the collection:
???
The ConcurrentBag does not provide a way to remove a specific item. None of the concurrent classes do as far as I can tell. Am I missing something? Aside from this, ConcurrentBag does everything I need.
ConcurrentBag<T> should definitely be the best performing class out of the bunch for you to use for such a case. Enumeration works exactly as your friend describes and so it should serve well for the scenario you have laid out. However, knowing you have to remove specific items from this set, the only type that's going to work for you is ConcurrentDictionary<K, V>. All the other types only offer a TryTake method which, in the case of ConcurrentBag<T>, is indeterminate or, in the case of ConcurrentQueue<T> or ConcurrentStack<T> ordered only.
For broadcasting you would just do:
ConcurrentDictionary<string, Recipient> myConcurrentDictionary = ...;
...
foreach(Recipient recipient in myConcurrentDictionary.Values)
{
...
}
The enumerator is once again a snapshot of the dictionary in that instant.
I came into work this morning to an e-mail from a friend that gives me the following two answers:
1 - With regards to how the collections in the Concurrent namespace work, most of them are designed to allow additions and subtractions from the collection without blocking and are thread-safe even when in the process of enumerating the collection items.
With a "regular" collection, getting an enumerator (via GetEnumerator) sets a "version" value that is changed by any operation that affects the collection items (such as Add, Remove or Clear). The IEnumerator implementation will compare the version set when it was created against the current version of the collection. If different, an exception is thrown and enumeration ceases.
The Concurrent collections are designed using segments that make it very easy to support multi-threading. But, in the case of enumerating, they actually create a snapshot copy of the collection at the time GetEnumerator is called and the enumerator works against this copy. That allows changes to be made to the collection without adverse affects on the enumerator. Of course this means that the enumeration will know nothing of these changes but it sounds like your use-case allows this.
2 - As far as the specific scenario you are describing, I don't believe that a Concurrent collection is needed. You can wrap a standard collection using a ReaderWriterLock and apply the same logic as the Concurrent collections when you need to enumerate.
Here's what I suggest:
public class RecipientCollection
{
private Collection<Recipient> _recipients = new Collection<Recipient>();
private ReaderWriterLock _lock = new ReaderWriterLock();
public void Add(Recipient r)
{
_lock.AcquireWriterLock(Timeout.Infinite);
try
{
_recipients.Add(r);
}
finally
{
_lock.ReleaseWriterLock();
}
}
public void Remove(Recipient r)
{
_lock.AcquireWriterLock(Timeout.Infinite);
try
{
_recipients.Remove(r);
}
finally
{
_lock.ReleaseWriterLock();
}
}
public IEnumerable<Recipient> ToEnumerable()
{
_lock.AcquireReaderLock(Timeout.Infinite);
try
{
var list = _recipients.ToArray();
return list;
}
finally
{
_lock.ReleaseReaderLock();
}
}
}
The ReaderWriterLock ensures that operations are only blocked if another operation that changes the collection's contents is in progress. As soon as that operation completes, the lock is released and the next operation can proceed.
Your alert engine would use the ToEnumerable() method to obtain a snapshot copy of the collection at that time and enumerate the copy.
Depending on how often an alert is sent and changes are made to the collection, this could be an issue but you might be able to still implement some type of version property that is changed when an item is added or removed and the alert engine can check this property to see if it needs to call ToEnumerable() again to get the latest version. Or encapsulate this by caching the array inside the RecipientCollection class and invalidating the cache when an item is added or removed.
HTH
There is much more to an implementation like this than just the parallel processing aspects, durability probably being paramount among them. Have you considered building this using an existing PubSub technology like say... Azure Topics or NServiceBus?
Your requirements strike me as an good fit for the way standard .NET events are triggered in C#. I don't know offhand if the VB syntax gets compiled to similar code or not. The standard pattern looks something like:
public event EventHandler Triggered;
protected void OnTriggered()
{
//capture the list so that you don't see changes while the
//event is being dispatched.
EventHandler h = Triggered;
if (h != null)
h(this, EventArgs.Empty);
}
Alternatively, you could use an immutable list class to store the recipients. Then when the alert is sent, it will first take the current list and use it as a "snapshot" that cannot be modified by adding and removing while you are sending the alert. For example:
class Alerter
{
private ImmutableList<Recipient> recipients;
public void Add(Recipient recipient)
{
recipients = recipients.Add(recipient);
}
public void Remove(Recipient recipient)
{
recipients = recipients.Remove(recipient);
}
public void SendAlert()
{
//make a local reference to the current list so
//you are not affected by any calls to Add/Remove
var current = recipients;
foreach (var r in current)
{
//send alert to r
}
}
}
You will have to find an implementation of an ImmutableList, but you should be able to find several without too much work. In the SendAlert method as I wrote it, I probably didn't need to make an explicit local to avoid problems as the foreach loop would have done that itself, but I think the copy makes the intention clearer.

Reliable and efficient way to handle Azure Table Batch updates

I have an IEnumerable that I'd like to add to Azure Table in the most efficient way possible. Since every batch write has to be directed to the same PartitionKey, with a limit of 100 rows per write...
Does anyone want to take a crack at implementing this the "right" way as referenced in the TODO section? I'm not sure why MSFT didn't finish the task here...
Also I'm not sure if error handling will complicate this, or the correct way to implement it. Here is the code from the Microsoft Patterns and Practices team for Windows Azure "Tailspin Toys" demo
public void Add(IEnumerable<T> objs)
{
// todo: Optimize: The Add method that takes an IEnumerable parameter should check the number of items in the batch and the size of the payload before calling the SaveChanges method with the SaveChangesOptions.Batch option. For more information about batches and Windows Azure table storage, see the section, "Transactions in aExpense," in Chapter 5, "Phase 2: Automating Deployment and Using Windows Azure Storage," of the book, Windows Azure Architecture Guide, Part 1: Moving Applications to the Cloud, available at http://msdn.microsoft.com/en-us/library/ff728592.aspx.
TableServiceContext context = this.CreateContext();
foreach (var obj in objs)
{
context.AddObject(this.tableName, obj);
}
var saveChangesOptions = SaveChangesOptions.None;
if (objs.Distinct(new PartitionKeyComparer()).Count() == 1)
{
saveChangesOptions = SaveChangesOptions.Batch;
}
context.SaveChanges(saveChangesOptions);
}
private class PartitionKeyComparer : IEqualityComparer<TableServiceEntity>
{
public bool Equals(TableServiceEntity x, TableServiceEntity y)
{
return string.Compare(x.PartitionKey, y.PartitionKey, true, System.Globalization.CultureInfo.InvariantCulture) == 0;
}
public int GetHashCode(TableServiceEntity obj)
{
return obj.PartitionKey.GetHashCode();
}
}
Well, we (the patterns & practices team) just optimized for showing other things we considered useful. The code above is not really a "general purpose library", but rather a specific method for the sample that uses it.
At that moment we thought that adding that extra error handling would not add much, and we diceided to keep it simple, but....we might have been wrong.
Anyway, if you follow the link in the //TODO:, you will find another section of a previous guide we wrote that talks a little bit more on error handling in "complex" storage transactions (not in the "ACID" form though as transactions "ala DTC" are not supported in Windows Azure Storage).
Link is this: http://msdn.microsoft.com/en-us/library/ff803365.aspx
The limitations are listed in more detail there:
Only one instance of the entity should be present in the batch
Max 100 entities or 4 MB payload
Same PartitionKey (which is being handled in the code: notice that "batch" is only specified if there's a single Partition key)
etc.
Adding some extra error handling should not overcomplicate things too much, but depends on the type of app you are building on top of this and your preference to handle this higher or lower in your app stack. In our example, the app would never expect > 100 entities anyway, so it would simply bubble the exception up if that situation happens (because it should be truly exceptional). Same with the total size. The use cases implemented in the app make it impossible to have the same entity in the same collection, so again, that should never happen (and if it happens, it wouls simply throw)
All "entity group transactions" limitations are documented here: http://msdn.microsoft.com/en-us/library/dd894038.aspx
Let us know how it goes! I'm also interested to know if other pieces of the guide were useful for you.

Resources