Giving a quick overview of my situation:
I am working on an N-Tier app that relies a lot on serialization, objects interact with the database mainly in a serialized fashion, objects and collections are inserted, updated and read as XML from within stored procedures.
For some of the smaller data classes I am simply using ExecuteNonQuery, Reader, etc to interact with the data, as its easier, but I have encountered a problem.
Data is inserted into the database using ExecuteNonQuery, using Parameters - some of the data inserted are properties that are Enums (stored in the DB as ints) that have the FlagAttribute attached. On a Enum such as:
<Flags()> _
Public Enum Fruit As Integer
<Description("None"), XmlEnum("0")> None = 0
<Description("Apple"), XmlEnum("1")> Apple = 1
<Description("Banana"), XmlEnum("2")> Banana = 2
<Description("Orange"), XmlEnum("4")> Orange = 4
End Enum
The value read back might be an Integer value of 1, 3, 7, etc, and inserted into the database not using serialization, when it is read back however as part of a larger group of classes using the ExecuteXmlReader (filling a XmlReader object) and then needing to be deserialized, it can not be, as 7 for example, causes 'Instance validation error: '7' is not a valid value for Fruit', as it is expecting it to be serialized in the format of:
<fruitOptions>1 2 4<fruitOptions>
All in all it is a little confusing, and I could probably work around it by storing it in the database in the 1, 2, 4 format, but sadly not in the int type that it currently is in.
Does anyone have any ideas on this?
You're going to have to ask your DBA or the author of the stored procedure you're using. It appears they've specified that this field is not an enum, but rather a list.
Related
My task is to check the value of data from the global data sheet within different UIs, each of them having lots of data.
My idea was to do this in a generic way.
I create a array with the objects name, which corresponds with the name of the data sheet column
And then I just compare the content
Browser("").Page("").GENERIC_TYPE(label).GetROProperty("value") = datasheet.GetParameter(label)
Is there such a Generic Type that works for WebEdit and WebList?
You can use WebElement and this is generic as all elements on the page are web elements.
If you are reading the objects from OR, then you might have to update the element type to WebElement and it's tidious (if you are dealing with multiple objects). So the alternative way is using the below approach.
Browser("").Page("").WebElement("xpath:=//*[#common_attribute=" + element_attribute_value + "]").GetROProperty("value") = datasheet.GetParameter(label)
I’m experimenting with scripting a batch of OmniFocus tasks in JXA and running into some big speed issues. I don't think the problem is specific to OmniFocus or JXA; rather I think this is a more general misunderstanding of how getting objects works - I'm expecting it to work like a single SQL query that loads all objects in memory but instead it seems to do each operation on demand.
Here’s a simple example - let’s get the names of all uncompleted tasks (which are stored in a SQLite DB on the backend):
var tasks = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false})
var totalTasks = tasks.length
for (var i = 0; i < totalTasks; i++) {
tasks[i].name()
}
[Finished in 46.68s]
Actually getting the list of 900 tasks takes ~7 seconds - already slow - but then looping and reading basic properties takes another 40 seconds, presumably because it's hitting the DB for each one. (Also, tasks doesn't behave like an array - it seems to be recomputed every time it's accessed.)
Is there any way to do this quickly - to read a batch of objects and all their properties into memory at once?
Introduction
With AppleEvents, the IPC technology that JavaScript for Automation (JXA) is built upon, the way you request information from another application is by sending it an "object specifier," which works a little bit like dot notation for accessing object properties, and a little bit like a SQL or GraphQL query.
The receiving application evaluates the object specifier and determines which objects, if any, it refers to. It then returns a value representing the referred-to objects. The returned value may be a list of values, if the referred-to object was a collection of objects. The object specifier may also refer to properties of objects. The values returned may be strings, or numbers, or even new object specifiers.
Object specifiers
An example of a fully-qualified object specifier written in AppleScript is:
a reference to the name of the first window of application "Safari"
In JXA, that same object specifier would be expressed:
Application("Safari").windows[0].name
To send an IPC request to Safari to ask it to evaluate this object specifier and respond with a value, you can invoke the .get() function on an object specifier:
Application("Safari").windows[0].name.get()
As a shorthand for the .get() function, you can invoke the object specifier directly:
Application("Safari").windows[0].name()
A single request is sent to Safari, and a single value (a string in this case) is returned.
In this way, object specifiers work a little bit like dot notation for accessing object properties. But object specifiers are much more powerful than that.
Collections
You can effectively perform maps or comprehensions over collections. In AppleScript this looks like:
get the name of every window of Application "Safari"
In JXA it looks like:
Application("Safari").windows.name.get()
Even though this requests multiple values, it requires only a single request to be sent to Safari, which then iterates over its own windows, collecting the name of each one, and then sends back a single list value containing all of the name strings. No matter how many windows Safari has open, this statement only results in a single request/response.
For-loop anti-pattern
Contrast that approach to the for-loop anti-pattern:
var nameOfEveryWindow = []
var everyWindowSpecifier = Application("Safari").windows
var numberOfWindows = everyWindowSpecifier.length
for (var i = 0; i < numberOfWindows; i++) {
var windowNameSpecifier = everyWindowSpecifier[i].name
var windowName = windowNameSpecifier.get()
nameOfEveryWindow.push(windowName)
}
This approach may take much longer, as it requires length+1 number of requests to get the collection of names.
(Note that the length property of collection object specifiers is handled specially, because collection object specifiers in JXA attempt to behave like native JavaScript Arrays. No .get() invocation is needed (or allowed) on the length property.)
Filtering, and why your code example is slow
The really interesting part of AppleEvents is the so-called "whose clause". This allows you provide criteria with which to filter the objects from which the values will be returned from.
In the code you included in your question, tasks is an object specifier that refers to a collection of objects that have been filtered to only include uncompleted tasks using a whose clause. Note that this is still just reference at this point; until you call .get() on the object specifier, it's just a pointer to something, not the thing itself.
The code you included then implements the for-loop anti-pattern, which is probably why your observed performance is so slow. You are sending length+1 requests to OmniFocus. Each invocation of .name() results in another AppleEvent.
Furthermore, you're asking OmniFocus to re-filter the collection of tasks every time, because the object specifier you're sending each time contains a whose clause.
Try this instead:
var taskNames = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false}).name.get()
This should send a single request to OmniFocus, and return an array of the names of each uncompleted task.
Another approach to try would be to ask OmniFocus to evaluate the "whose clause" once, and return an array of object specifiers:
var taskSpecifiers = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false})()
Iterating over the returned array of object specifies and invoking .name.get() on each one would likely be faster than your original approach.
Answer
While JXA can get arrays of single properties of collections of objects, it appears that due to an oversight on the part of the authors, JXA doesn't support getting all of the properties of all of the objects in a collection.
So, to answer you actual question, with JXA, there is not a way to read a batch of objects and all their properties into memory at once.
That said, AppleScript does support it:
tell app "OmniFocus" to get the properties of every flattened task of default document whose completed is false
With JXA, you have to fall back to the for-loop anti-pattern if you really want all of the properties of the objects, but we can avoid evaluating the whose clause more than once by pulling its evaluation outside of the for loop:
var tasks = []
var taskSpecifiers = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false})()
var totalTasks = taskSpecifiers.length
for (var i = 0; i < totalTasks; i++) {
tasks[i] = taskSpecifiers[i].properties()
}
Finally, it should be noted that AppleScript also lets you request specific sets of properties:
get the {name, zoomable} of every window of application "Safari"
But there is no way with JXA to send a single request for multiple properties of an object, or collection of objects.
Try something like:
tell app "OmniFocus"
tell default document
get name of every flattened task whose completed is false
end tell
end tell
Apple event IPC is not OOP, it’s RPC + simple first-class relational queries. AppleScript obfuscates this, and JXA not only obfuscates it even worse but cripples it too; but once you learn to see through the faux-OO syntactic nonsense it makes a lot more sense. This and this may give a bit more insight.
[ETA: Omni recently implemented its own embedded JavaScriptCore-based scripting support in its apps; if JS is your thing you might find that a better bet.]
I believe that I had not drafted my question clearly enough in my previous posting.
My question relates to the old technology. Yet it matters a lot to me to get a through answer. Could a Visual Basic expert answer me or include links to other sites and sheds light on this memory usage question please?
Consider the following VB 6 code in a custom COM plus object on a page on an e-commerce site:
Assign long and short strings to the globalArray elements. Also some of the elements of globalArray hold smaller arrays.
And also load the xml:
Dom globalArray(30, 100000)
Set objGlobalDom = CreateObject("msxml2.FreeThreadedDomDocument.6.0")
objGlobalDom.loadXML (xmlStr)
Application.lock
Set Application(“objGlobalDom”) = objGlobalDom
Application(“globalArray”) = globalArray
Application.unlock
With any new session the following variable assignments are done:
set Session(“objGlobalDom”) = Application(“objGlobalDom”)
session(“globalArray”) = Application(“objGlobalDom”)
The Application(“objGlobalDom”) will contain an xml with some 1000 nodes and each node takes about 3k of memory. The array will take some 50 meg of memory.
Considering VB6 and COM object:
I understand that each instance of the object references the object’s data. What I don’t understand is:
1- If Session(“objGlobalDom”) does not contain a copy of the Application(“objGlobalDom”), why changes in the data of Session(“objGlobalDom”) are not automatically reflected in the data of Application(“objGlobalDom”) or does the Session(“objGlobalDom”) have a copy of the Application(“objGlobalDom”)?
2- According to Microsoft, in a situation such as my globalArray example, the Session(“globalArray”) always gets a copy of the Application(“globalArray”) and so Microsoft discourages assigning the array to session variables. But it is not clear to me that in the case of COM object and object references, does the assignment of set Session(“objGlobalDom”) = Application(“objGlobalDom”) copies the array to the session variable?
I really appreciate your answers and thank you in advance for your response.
COM Object are just UDT 'User Defined Type' (structs int c, record in delphi). that hold pointers to function, and private data related for each instance of the object.
when you see in VB the keyword set it does not mean it will copy all data from var to var, it just copy its reference (address) and add one to the refcount of that object. and when you do test var=nothing the object will not be freed until refcount reach 0. So set Obj=Nothing decrements Object RefCount , and if RefCount =0 the the Object Free Itself.
I'm using Fog to access a cloud environment at Terremark. When I pull down our organizational data it returns a data structure that, while I know it should be straight forward, confuses me.
Using irb I initialize the connection and then request the data using conn.organizations and display it with awesome_print. It returns:
[
[0] <Fog::Compute::Ecloud::Organization
href="/cloudapi/ecloud/organizations/#######",
name="****************************** (***-###-###)",
type="application/vnd.tmrk.cloud.organization",
other_links=[{:href=>"/cloudapi/ecloud/admin/organizations/#######", :name=>"****************************** (***-###-###)", :type=>"application/vnd.tmrk.cloud.admin.organization", :rel=>"alternate"}, {:href=>"/cloudapi/ecloud/environments/organizations/#######", :type=>"application/vnd.tmrk.cloud.environment; type=collection", :rel=>"down"}, {:href=>"/cloudapi/ecloud/devicetags/organizations/#######", :type=>"application/vnd.tmrk.cloud.deviceTag; type=collection", :rel=>"down"}, {:href=>"/cloudapi/ecloud/alerts/organizations/#######", :type=>"application/vnd.tmrk.cloud.alertLog", :rel=>"down"}]
>
]
So it is returning an array with a singular element. That element is comprised of another data structure surrounded by < and >. But I'm not certain if that's accurate because there also appears to be another array containing a hash embedded within that structure.
My issue is that I need to extract the value represented by the ####### but I don't know how to access any of the sections of the output which contain that value.
What am I looking at as far as the data structure is concerned and how do I go about access the data contained within?
It's a Fog::Compute::Ecloud::Organization object and the documentation of that class should tell you what methods are available. Or you can just ask the object itself, by calling Object#methods.
I'm using Grails with an Oracle database. Most of the data in my application is part of a hierarchy that goes something like this (each item containing the following one):
Direction
Group
Building site
Contract
Inspection
Non-conformity
Data visible to a user is filtered according to his accesses which can be at the Direction, Group or Building Site level depending on user role.
We easily accomplished this by creating a listWithSecurity method for the BuildingSite domain class which we use instead of list across most of the system. We created another listWithSecurity method for Contract. It basically does a Contract.findAllByContractIn(BuildingSite.listWithSecurity). And so on with the other classes. This has the advantage of keeping all the actual access logic in BuildingSite.listWithsecurity.
The problem came when we started getting real data in the system. We quickly hit the "ora-01795 maximum number of expressions in a list is 1000" error. Fair enough, passing a list of over 1000 literals is not the most efficient thing to do so I tried other ways even though it meant I would have to deport the security logic to each controller.
The obvious way seemed to use a criteria such as this (I only put the Direction level access here for simplicity):
def c = NonConformity.createCriteria()
def listToReturn = c.list(max:params.max, offset: params.offset?.toInteger() ?: 0)
{
inspection {
contract {
buildingSite {
group {
'in'("direction",listOfOneOrTwoDirections)
}
}
}
}
}
I was expecting Grails to generate a single query with joins that would avoid the ora-01795 error but it seems to be calling a separate query for each level and passing the result back to Oracle as literal in an 'in' to query the other level. In other words, it does exactly what I was doing so I get the same error.
Actually, it might be optimising a bit. It seems to be solving the problem but only for one level. In the previous example, I wouldn't get an error for 1001 inspections but I would get it for 1001 contracts or building sites.
I also tried to do basically the same thing with findAll and a single HQL where statement to which I passed a single direction to get the nonConformities in one query. Same thing. It solves the first levels but I get the same error for other levels.
I did manage to patch it by splitting my 'in' criteria into many 'in' inside an 'or' so no single list of literals is more than 1000 long but that's profoundly ugly code. A single findAllBy[…]In becomes over 10 lines of code. And in the long run, it will probably cause performance problems since we're stuck doing queries with a very large amount of parameters.
Has anyone encountered and solved this problem in a more elegant and efficient way?
This won't win any efficiency awards but I thought I'd post it as an option if you just plainly need to query a list of more than 1000 items none of the more efficient options are available/appropriate. (This stackoverflow question is at the top of Google search results for "grails oracle 1000")
In a grails criteria you can make use of Groovy's collate() method to break up your list...
Instead of this:
def result = MyDomain.createCriteria().list {
'in'('id', idList)
}
...which throws this exception:
could not execute query
org.hibernate.exception.SQLGrammarException: could not execute query
at grails.orm.HibernateCriteriaBuilder.invokeMethod(HibernateCriteriaBuilder.java:1616)
at TempIntegrationSpec.oracle 1000 expression max in a list(TempIntegrationSpec.groovy:21)
Caused by: java.sql.SQLSyntaxErrorException: ORA-01795: maximum number of expressions in a list is 1000
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:440)
You'll end up with something like this:
def result = MyDomain.createCriteria().list {
or { idList.collate(1000).each { 'in'('id', it) } }
}
It's unfortunate that Hibernate or Grails doesn't do this for you behind the scenes when you try to do an inList of > 1000 items and you're using an Oracle dialect.
I agree with the many discussions on this topic of refactoring your design to not end up with 1000+ item lists but regardless, the above code will do the job.
Along the same lines as Juergen's comment, I've approached a similar problem by creating a DB view that flattens out user/role access rules at their most granular level (Building Site in your case?) At a minimum, this view might contain just two columns: a Building Site ID and a user/group name. So, in the case where a user has Direction-level access, he/she would have many rows in the security view - one row for each child Building Site of the Direction(s) that the user is permitted to access.
Then, it would be a matter of creating a read-only GORM class that maps to your security view, joining this to your other domain classes, and filtering using the view's user/role field. With any luck, you'll be able to do this entirely in GORM (a few tips here: http://grails.1312388.n4.nabble.com/Grails-Domain-Class-and-Database-View-td3681188.html)
You might, however, need to have some fun with Hibernate: http://grails.org/doc/latest/guide/hibernate.html