Related
I want to pass an instance of an object between two Ruby processes. Specifically, I want to pass an instance of a Selenium WebDriver from one process to another process. The reason I want to do this is because it takes a lot of time for Ruby to create this object, but I want it to be used by the other process.
I've found some related questions here and here that seem to point towards using DRb, but I've been unable to find any useful examples or sample code.
Is there a tool other than DRb that I should be using? Does anyone have an example similar to this that I could copy from?
It looks like you're going to have to use DRb, although the documentation for it seems to be lacking. There is however an interesting article here. You might also want to consider purchasing The dRuby Book by Masatoshi Seki to get a better idea of how to do this effectively.
Another option to investigate if you are not looking at simultaneous access, but you just want to send the object from one process to another, is to serialize (that is, encode in a way that Ruby can read) the object with YAML (for a human readable file) or Marshall (for a binary encoded file) and send it using a pipe. This was mentioned in another answer that has since been deleted.
Note that either of these solutions require modifying the Selenium code heavily since the objects you want to manipulate neither support copying, nor simultaneous access natively.
TL;DR
Most queue or distributed processes are going to require some sort of serialization to work properly. If you want to pass objects rather than messages, then this will a limiting factor in how you approach the problem.
DRb
I don't know if you can marshal a WebDriver object. If you can't, then DRb may be a good choice for your distributed Ruby programs because it supports DRbObject references for things that can't be marshaled. There are some examples provided in the DRb documentation.
Selenium Wire Protocol
Depending on what you're really trying to do, it may be worth taking a closer look at using the remote bindings for the Remote WebDriver client/server, or Selenium's JSON Wire Protocol as an alternative to passing objects between processes.
Other Alternatives: Fixtures, Factories, Stubs, and Mocks
Whether or not these work in your specific case will depend a lot on why you want to pass objects instead of simply driving the remote server. If it's largely an issue of how long it takes to build your object, then the serialization/de-serialization cycle may not necessarily be faster in all cases.
You might want to revisit why your object is so slow to create. If gathering and processing the data for it is what's taking too long, you can use some sort of test fixture or factory to trim that time, either by using a smaller set of fixed data, or using a pre-serialized object that's optimized for speed.
You might also consider whether you actually need real data or objects for your test at all. In many cases, you can speed up your tests a lot by stubbing methods or creating mock objects that will return the values you need for your integration tests without needing to perform expensive calculations or long-running operations.
There are certainly cases where you need to drive the full stack and perform acceptance tests on real data. Even then, you may be able to devise a set of fixture data that will take less time or memory to process. It's certainly worth at least thinking about.
I'm writing an application that uses Hector to access a Cassandra database. I have some situations where I only need to query one column, and some where I need to query multiple columns at once. Writing one method that takes an array of column names and returns a list of columns using SliceQuery would be simplest in terms of code, but I'm wondering whether there's a significant drawback to using SliceQuery for one column compared to using ColumnQuery.
In short, are there enough (or any) performance benefits of using ColumnQuery over SliceQuery for one column to make it worth the extra code to deal with a one-column case separately?
By looking at Hector's code , the difference between using a ColumnQuery (ThriftColumnQuery.java) and a SliceQuery (ThriftSliceQuery.java) is the different thrift command being sent - "get" or "get_slice" (respectively).
I didn't find an exact documentation of how each of those operations are implemented by Cassandra's server, but I took a quick look in Cassandra's sources and after examining CassandraServer.java I got the impression that the "get" operation is there more for client's convenience than for better performance when querying a single column:
For a "get" request, a SliceByNamesReadCommand instance is created and executed.
For a "get_slice" request (assuming you're using Hector's setColumnNames method and not setRange), a SliceByNamesReadCommand instance is created for each of the wanted columns and then executed (the row is read only once though).
Bottom line, as far as I see it there's not much more than the (negligible) overhead of creating some collections meant for handling the multiple columns.
If you're still worried however, I believe it shouldn't be too difficult to handle the two cases differently when wrapping the use of Hector in your DAOs.
Hope I managed to help.
Given the premise:
There are competent sql programmers
(correlary - writing sql queries are not an issue)
There are competent application developers
(correlary - there is simple/strong/flexible architecture for handling connections and simple queries from code)
Why do people use linq to sql?
There is overhead added to each transaction
There is strong likelihood of performance loss for moderate-complex calculations (DBs are made for processing sets and calculations and had teams of engineers working out optimization - why mess with this?)
There is loss of flexibility (if you want to add another ui (non .NET app) or access method, you either have to put the queries back in the db or make a separate data access layer)
There is loss of security by not having a centralized control of write/update/read on db (for example, a record has changed - if you allow applications to use linq to sql to update, then you cannot prove which application changed it or what instance of an application changed it)
I keep seeing questions about linq to sql and am wondering if I'm missing something.
I keep seeing questions about linq to sql and am wondering if I'm missing something.
It's not that you're missing something. It's that you have something most shops don't have:
There are competent sql programmers
Additionally, in your shop those competent sql programmers prefer to write sql.
Here's a point by point response:
There is overhead added to each transaction
Generally true. This can be avoided by translating the queries before they are needed to run using CompiledQuery for many (but not all!) scenarios.
There is strong likelihood of performance loss for moderate-complex calculations (DBs are made for processing sets and calculations and had teams of engineers working out optimization - why mess with this?)
Either you're writing linq, which is translated to sql, and then a plan is generated from the optimizer - or your writing sql from which a plan is generated by the optimizer. In both cases you are telling the machine what you want and it is supposed to figure out how to do it. Are you suggesting that subverting the optimizer by using query hints is a good practice? Many competent sql programmers will disagree with that suggestion.
There is loss of flexibility (if you want to add another ui (non .NET app) or access method, you either have to put the queries back in the db or make a separate data access layer)
A lot of people using linq are already SOA. The linq lives in a service. The non .NET app calls the service. Bada-bing bada-boom.
There is loss of security by not having a centralized control of write/update/read on db (for example, a record has changed - if you allow applications to use linq to sql to update, then you cannot prove which application changed it or what instance of an application changed it)
This is simply not true. You prove which application is connected and issuing sql commands the same way you prove which application is connected and calling a sproc.
Let me list you a few points:
There are small software companies or mid-sized companies who develop their software in-house who might rather focus on getting many application developers than getting a freelancer DB developer or even permanently hire one.
In most cases the overhead is a non-issue either due to the amount of data to be processed or due to the low traffic. Besides, when used properly, LINQ to SQL can perform as fast as most SQL queries + the associated .net code.
Many companies just stick with the Microsoft stack and they can only enjoy the integration. Some other company develops using SOA there's just no problem. The others aren't forced to choose LINQ-to-SQL and if they make that choice is their problem how to integrate it. Nobody ever said LINQ-to-SQL is a silver bullet :)
I believe security is gained with LINQ-to-SQL because I've bumped across lots of SQL queries taking in unescaped data with string concatenation etc and explaining the whole parametrized query idea has never been easy. Besides since all queries are eventually translated into SQL, unless the tracking issue you describe would happen via a stored procedure, there're again no problems at all.
I also believe your question can be posed more generally to address all ORMs and not just LINQ-to-SQL, and still most of what I said would hold true.
The problem is that it is very rare for somewhere to have a competent SQL developer who likes writing SQL and wouldn't rather be doing something else. I would consider myself competent in SQL, I used to do all my data access layers with stored procs or parametrized queries. Trouble is that it takes ages and is dull. I'd rather be writing great applications than messing around with data access layers that essentially have a select, insert, update and delete SQL statement(or proc) repeated dozens of times for each data object.
Linq-to-SQL takes away some of the repetitive nature. It has a tool to auto generate you business objects from your database schema, and it gives you a nice integrated query language that is compile time type verified and is in your code (Stored procs are a pain to source control neatly)
I can write a DAL in Linq-to-sql several times faster than I can using plain SQL, stored procs or parametrized queries.
If you want to maintain the use of stored procs both linq-to-sql and the EF both support the use of stored procs for all their data access, you just have to set up the appropriate mappings. So, you can still use your stored procs to log details and implement security if you want. We tend to opt for using windows auth, and use that to restrict access to each table for the various users, then we have a bunch of triggers on the tables that track details for audit purposes.
Two things I will quickly note is that firstly, the entity framework seems to be getting more support from MS at the moment, and I suspect that will be considered the kind of default standard for the future in preference to linq-to-sql. Secondly, in .Net 3.5 the EF and linq-to-sql do not have very good support for n-tier disconnected apps. In both of them you kind of have to muck around with either serializing data contexts across your disconnected tiers, or manually detach and re-attach your data objects. This is much improved in the .net 4.0 though. Just something to consider depending on which version you have available to you.
Existing question/answers in the same vein/spirit:
Doesn't Linq to SQL miss the point? Aren't ORM-mappers (SubSonic, etc.) sub-optimal solutions?
LINQ-to-SQL vs stored procedures?
What's wrong with Linq to SQL?
Why do I need Stored Procedures when I have LINQ to SQL
If using LINQ to SQL is there any good reason to learn SQL queries/syntax anymore?
https://stackoverflow.com/questions/216569/are-the-days-of-the-stored-procedure-numbered
I personally believe there's no right or wrong answer. It depends on what you're developing and how you're developing it. If you need razor-sharp performance, have an overly-complex data model, etc... skip the abstraction. If you feel the abstraction speeds up your development time, like the idea of capturing all application logic in a single codebase, etc... use it.
For me, it takes a lot less time to write linq to sql code than it does to write a bunch of stored procedures. That's especially true when the design isn't finished, in that case I don't yet know how much of the work I want to do on C# objects, and how much I want to do in SQL.
So, I can skip building datasets, I don't have to click click click to add queries, basically, linq to sql means I can change my code in less time.
Also, as a big fan of Haskell, I can write lots of functional-style code with linq to sql and it just works.
I'm not saying this is an ideal solution or even a great example (it was the result of a high level constraint on the architecture, not something we necessarily would have chosen from scratch), but...
I worked on an app where the code was completely isolated from the database except through a set of exposed stored procs. The code could not "know" anything about the database schema except was was returned from the stored procs.
While this isn't that unusual and it isn't too hard to write a DAL using ADO or whatever, I decided to try out Linq to Sql, even though it wouldn't be using it for its real intended purpose and wouldn't use most of the features. Turns out it was a great decision.
I created the Linq to Sql class, dragged the stored procs from server explorer onto the right side of the designer, then... Wait, there is no then. I was pretty much done.
Linq created strongly typed methods for each stored proc. For the procs that returned rows of data, Linq automatically created a class for the items in each row and returned a List<generatedClass> for them. I wrapped the calls themselves in a lightweight public DAL class that did some verification and some automatic parameter setting and I was done. I wrote a business object class and mapped the dynamically generated Linq class objects to the business object (did this by hand, but it isn't hard to do or maintain).
The program is now immune to any schema change that doesn't affect the stored procedure signatures. If the signatures do change, we just drag off the old proc from the design and drag it back to regenerate the code. A few passes through the unit tests to make changes (which usually don't go higher than the public DAL interface) and it's done. Things upstream of the DAL use Linq to Objects techniques to select, filter, and sort data that isn't in the right format straight from the stored proc calls.
We have some excellent DBAs writing the stored procedures and an entirely different group writing the other code, so maybe it is a good example of why (and how) you can use LINQ in the scenario you describe.
Some handy features are the debugger picking up sytax errors in your query, compared to writing SQL statements as strings. Mistakes that wont get picked up until runtime.
Plus I find LINQ statements easier to read than SQL.
It may be a case of convenience triumphing over performance. If you're programming at Facebook levels of uber-performance then you might think about every clock cycle but the simple truth is that the majority of applications don't need this attention and benefit from efficiencies in code maintenance and dev time (100k contractor vs. another $erver).
That said, there's a case for outsourcing as much of the query processing from the DB box in very high scale systems, else the DB is the bottle neck and you need to shard or re-architect down the line. Costly.
I think its fair to say that LINQ will scale better/easier both in terms of servers and from many core in that your LINQ codebase will get m-core for 'free' as soon as MS release C# 4.0.
I do see your point in asking and as a non-ASP.NET dev just beginning a www project for the first time, I can't see the point of 80% of ASP.NET (themes, controls etc.) - it seems I need to learn more and code more than the HTML itself! -- again, I'm sure there's an good reason for it all.
--- I haven't got the 50 pts to comment on the post I want to so I'm doing it here ---
David B suggests that writing some SQL is all there is to getting the most out of SQL Server and that using query hints is the steering mechanism. The same task can be achieved in many different ways in SQL and many with 1000s of times the performance gain. I suggest reading Inside T-SQL Querying (Itzik Ben Gan). Over and over, Itzik shows how to rethink the query and use new commands to shrink the logical reads sometimes from thousands into less than ten.
For very simple queries, the overhead of an extra layer adds to the roundtrip cost. For somewhat more complex queries in normal 'business app' scenarios, the optimizations done by the Linq-to-SQL expression->sql translation magic can often save a lot.
As an example, I recently did a 1:1 translation of a customer-supplied 1400+ (!) line stored proc to L2S. Not only did it go from 1400 lines of SQL to 500 lines of much more readable, strongly typed, and commented code. It also started hitting the database with an average of ~1500 reads instead of ~30k reads. This is before I even started looking at db-side optimizations - that saving is something I can 100% attribute to L2S's ability to eliminate predicates that can be evaluated client-side.
simple answer, there are two approaches: create exquisite Rube Goldberg contraptions, or just get the job done in a simple way. Many devs lean towards the former.
Developers get bored easily, and would often personally enjoy doing things a harder way that seems to provide a certain intellectual beauty. Are you developing an app or writing a PhD? As my msft director used to yell in the hallways, "I don't want another research project!"
please repeat after me (min 3x)
there is no silver bullet
I will not use a technology just because its the latest thing from msft
I will not use something just to get it on my resume
Not only are their competent SQL coders, any decent app programmer, especially LOB apps, should write intermediate SQL. If you don't know any SQL and are writing LINQ to SQL, how are you going to debug your data calls? How are you going to profile them to fix bottlenecks?
We're trying out LINQ to SQL and I think there are major issues with it, such as:
There is no simple way to return the query results to another object. This in itself seems insane. Microsoft created the var anonymous datatype, and recommends using it, but there is no way to move this data out of your local method, hence the oo paradigm breaks if you have to use the data in the same function that retrieved it.
Tables are NOT objects. Study up on 3rd normal form etc. Relational databases are for storing data, not using it. I don't want to be restricted or encouraged to use my tables as objects. The data I retrieve from the database will very often be joins of multiple tables, and may include SQL casts, functions, operators, etc.
There is no performance gain, and a slight loss
Now I have way more code to worry about. There are the dbml files and still a DAL to actually write the LINQ. Yes, lots of it is machine-generated, that doesn't mean its not there, its something else that can go wrong (i.e. your dbml files, etc.).
Now that I've given the background, I will attempt to answer you actual question, why do people use LINQ To SQL:
Its the latest thing from Microsoft and I want it on my resume.
Msft has convinced managers/execs that it will decrease coding time
Developers hate SQL. (no good dev environment or debugging except manually--it would be nice to have better intellisense to a sql tool.)
I encourage people not to jump on the bandwagon just because everyone else is, learn enough to put it on your resume, be willing to use it if forced to, but try and really understand the pros and cons first.
I have been playing with some LINQ ORM (LINQ directly to SQL) and I have to admit I like its expressive powers . For small utility-like apps, It also works quite fast: dropping a SQL server on some surface and you're set to linq away.
For larger apps however, the DAL never was that big of an issue to me to setup, nor maintain, and more often than not, once it was set, all the programming was not happening there anyway...
My, honest - I am an ORM newbie - question : what is the big advantage of ORM over writing a decent DAL by hand?
(seems like a double, couldn't find it though)
UPDATE : OK its a double :-) I found it myself eventually :
ORM vs Handcoded Data Access Layer
Strong-typing
No need to write the DAL yourself => time savings
No need to write SQL code yourself =>
less error-prone
I've used Hibernate in the past to dynamically create quite complex queries. The logic involved to create the appropriate SQL would have been very time-consuming to implement, compared with the logic to build the appropriate Criteria. Additionally, Hibernate knew how to work with various different databases, so I didn't need to put any of that logic in our code. We had to test against different databases of course, and I needed to write an extension to handle "like" queries appropriately, but then it ran against SQL Server, Oracle and HSqldb (for testing) with no issues.
There's also the fact that it's more code you don't have to write, which is always a nice thing :) I can't say I've used LINQ to SQL in anything big, but where I've used it for a "quick and dirty" web-site (very small, rarely updated, little benefit from full layer abstraction) it was lovely.
I used JPA in a project, and at first I was extremely impressed. Gosh it saved me all that time writing SQL! Gradually, however, I became a bit disenchanted.
Difficulty defining tables without surrogate keys. Sometimes we need tables that don't have surrogate keys. Sometimes we want a multicolumn primary key. TopLink had difficulties with that.
Forced datastructure relationships. JPA uses annotations to describe the relationship between a field and the container or referencing class. While this may seem great at first site, what do you do when you reference the objects differently in the application? Say for example, you need just specific objects that reference specific records based on some specific criteria (and it needs to be high-performance with no unnecessary object allocation or record retrieval). The effort to modify Entity classes will almost always exceed the effort that would have existed had you never used JPA in the first place (assuming you are at all successful getting JPA to do what you want).
Caching. JPA defines the notion of caches for your objects. It must be remembered that the database has its own cache, typically optimized around minimizing disk reads. Now you're caching your data twice (ignoring the uncollected GC heap). How this can be an advantage is beyond me.
Data != Objects. For high-performance applications, the retrieval of data from the DB must be done very efficiently. Forcing object creation is not always a good thing. For example, sometimes you may want arrays of primitives. This is about 30 minutes of work for an experienced programmer working with straight JDBC.
Performance, debugging.
It is much more difficult to gauge the performance of an application with complex things going on in the (sub-optimal, autogenerated) caching subsystem, further straining project resources and budgets.
Most developers don't really understand the impedence mismatch problem that has always existed when mapping objects to tables. This fact ensures that JPA and friends will probably enjoy considerable (cough cough) success for the forseeable future.
Well, for me it is a lot about not having to reinvent/recreate the wheel each time I need to implement a new domain model. It is simply a lot more efficient to use for instance nHibernate (my ORM of choice) for creating, using and maintaining the data access layer.
You don't specify exactly how you build your DAL, but for me I used to spend quite some time doing the same stuff over and over again. I used to start with the database model and work my way up from there, creating stored procedures etc. Even if I sometimes used little tools to generate parts of the setup, it was a lot of repetitive coding.
Nowadays I start with the domain. I model it in UML, and for most of the time I'm able to generate everything from that model, including the database schema. It need a few tweaks here and there, but with my current setup I get 95% of the job with the data access done in no time at all. The time I save I can use to fine tune the parts that need tuning. I seldom need to write any SQL statements.
That's my two cents. :-)
Portability between different db vendors.
My, honest - i am an ORM newbie - question : what is the big advance of ORM over writing a decent DAL by hand?
Not all programmers are willing or even capable of writing "a decent DAL". Those who can't or get scared from the mere thought of it, find LINQ or any other ORM a blessing.
I personally use LINQ to manipulate collections in the code because of its expressiveness. It offers a very compact and transparent way to perform some common tasks on collections directly in code.
LINQ will stop being useful to you when you will want to create very specific and optimized queries by hand. Then you are likely to get a mixture of LINQ queries intermingled with custom stored procedures wired into it. Because of this considerations, I decided against LINQ to SQL in my current project (since I have a decent (imho) DAL layer). But I'm sure LINW will do just fine for simple sites like maybe your blog (or SO for that matter).
With LINQ/ORM there may also be a consideration of lagging for high traffic sites (since each incoming query will have to be compiled all over again). Though I have to admit I do not see any performance issues on SO.
You can also consider waiting for the Entity Framework v2. It should be more powerful than LINQ (and hopefully not that bad as v1 (according to some people)).
Transparent persistence - changes get saved (and cascaded) without you having to call Save(). At first glance this seems like a nightmare, but once you get used to working with it rather than against it, your domain code can be freed of persistence concerns almost completely. I don't know of any ORM other than Hibernate / NHibernate that does this, though there might be some...
The best way to answer the question is to understand exactly what libraries like Hibernate are actually accomplishing on your behalf. Most of the time abstractions exist for a reason, often to make certain problems less complex, or in the case Hibernate is almost a DSL for expression certain persistance concepts in a simple terse manner.
One can easily change the fetch strategy for collections by changing an annotation rather than writing up lots of code.
Hibernate and Linq are proven and tested by many, there is little chance you can achieve this quality without lots of work.
Hibernate addresses many features that would take you months and years to code.
Also, while the JPA documentation says that composite keys are supported, it can get very (very) tricky quickly. You can easily spend hours (days?) trying to get something quite simple working. If JPA really makes things simpler then developers should be freed from thinking too much about these details. It doesn't, and we are left with having to understand two levels of abstraction, the ORM (JPA) and JDBC. For my current project I'm using a very simple implementation that uses a package protected static get "constructor" that takes a ResultSet and returns an Object. This is about 4 lines of code per class, plus one line of code for each field. It's simple, high-performance, and quite effective, and I retain total control. If I need to access objects differently I can add another method that reads different fields (leaving the others null, for example). I don't require a spec that tells me "how ORMs must (!) be done". If I require caching for that class, I can implement it precisely as required.
I have used Linq, I found it very useful. I saves a lot of your time writing data access code. But for large applications you need more than DAL, for them you can easily extent classes created by it. Believe me, it really improves your productivity.
I am adding some indexes to my DevExpress TdxMemDataset to improve performance. The TdxMemIndex has SortOptions which include the option for soCaseInsensitive. My data is usually a GUID string, so it is not case sensitive. I am wondering if I am better off just forcing all the data to the same case or if the soCaseInsensitive flag and using the loCaseInsensitive flag with the call to Locate has only a minor performance penalty (roughly equal to converting the case of my string every time I need to use the index).
At this point I am leaving the CaseInsentive off and just converting case.
IMHO, The best is to assure the data quality at Post time. Reasonings:
You (usually) know the nature of the data. So, eg. you can use UpperCase (knowing that GUIDs are all in ASCII range) instead of much slower AnsiUpperCase which a general component like TdxMemDataSet is forced to use.
You enter the data only once. Searching/Sorting/Filtering which all implies the internal upercassing engine of TdxMemDataSet it's a repeated action. Also, there are other chained actions which will trigger this engine whithout realizing. (Eg. a TcxGrid which is Sorted by default having GridMode:=True (I assume that you use the DevEx. components) and having a class acting like a broker passing the sort message to the underlying dataset.
Usually the data entry is done in steps, one or few records in a batch. The only notable exception is data aquisition applications. But in both cases above the user's usability culture allows way greater response times for you to play with. (IOW how much would add an UpperCase call to a record post which lasts 0.005 ms?) OTOH, users are very demanding with the speed of data retreival operations (searching, sorting, filtering etc.). Keep the data retreival as fast as you can.
Having the data in the database ready to expose reduces the risk of processing errors when you'll write (if you'll write) other modules (you need to remember to AnsiUpperCase the data in any module in any language you'll write). Also here a classical example is when you'll use other external tools to access the data (for ex. db managers to execute an SQL SELCT over the data).
hth.
Maybe the DevExpress forums (or ever a support email, if you have access to it) would be a better place to seek an authoritative answer on that performance question.
Anyway, is better to guarantee that data is on the format you want - for the reasons plainth already explained - the moment you save it. So, in that specific, make sure the GUID is written in upper(or lower, its a matter of taste)case. If it is SQL Server or another database server that have an guid datatype, make sure the SELECT make the work - if applicable and possible, even the sort.