Are there any good reasons why one would not have transaction management in their code?
The question came up when talking with a dba who gets very nervous when I bring up spring/hibernate. I mention that Spring can handle transactions, in use with Hibernate mapping tables to objects etc, and the issue comes up that the database(Oracle10g) already handles transaction management, so we should just use that. He even offered up the idea that we create a bunch of DB procedures to do inserts/updates so the database can handle things more efficiently, returning a 0/1 on whether the insert/update worked.
Are there any good reasons to not have your application deal with any transactions? Is my dba clueless? I'm thinking he is, but I'm not a great speaker when I'm unsure of the answer... which is why I'm out looking for the answer.
I think there is some misunderstanding here.
The point is that database doesn't manage transactions in the same sense as Spring/Hibernate.
Database "manages transactions" by providing transactional behaviour, and your application "manages transactions" by using that behaviour and defining transaction boundaries (in particular, with the help of Spring or Hibernate).
Since boundaries of transactions are defined by business logic, implementing an application without transaction management would require you to move all your business logic to the database side. Even if you implement simple insert/update operations as stored procedures, it won't be enough to free application from transaction management as long as application needs to define that several inserts/updates should be executed inside the same transaction.
I am not entirely sure if you mean that there will be a bunch of crud stored procedures (that do single inserts or updates), or if there will be stored procedures encompassing business logic (transaction scripts). If you mean the crud stored procedures, that is an entirely bad idea. (Actually even if you start with the crud approach you will end up with transaction scripts as business logic accretes, so it amounts to the same thing.) If you mean transaction scripts, that's an approach some places take. It is painful and there is no reuse, and you end up with a bunch of very complex stored procedures that are terribly hard to test. But DBAs like it because they know what's going on.
There is also an argument (applying to Transaction Scripts) that it's faster because there are less round trips, you have one call to the stored procedure that goes and does everything and returns a result, as opposed to your usual Spring/Hibernate application where you have multiple queries or updates and each statement is going over the network to the database (although Hibernate caches and reorders to try to minimize this). Minimizing network round-trips is probably the most valid reason for this approach, you have to weigh whether it is worth sacrificing flexibility for the reduced network traffic, or if it is a premature optimization.
Another argument made in favor of transaction scripts is that less competence is required to implement the system correctly. In particular Hibernate expertise is not required. You can hire a horde of code monkeys and have them bang out the code. All the hard stuff is removed from them and placed under the DBA's control.
So, to recap, here are the arguments for transaction scripts:
Less network traffic
Cheap developers
total DBA control (from your point of view, he will be a total bottleneck)
As mentioned above, there's no way to "use transactions" from the database standpoint without making your application aware of it at some level. Although, if you're using Spring, you can make this fairly painless by using <tx:annotation-driven> and applying the #Transactional annotations to the relevant methods in the service implementation classes.
That said, there are times when you should bypass transactions and write directly to the database. Specifically any time when speed is more important than guaranteed data integrity.
Related
I always tend to "protect" my persistance layer from violations via the service layer. However, I am beginning to wonder if it's really necessary. What's the point in taking the time in making my database robust, building relationships & data integrity when it never actually comes into play.
For example, consider a User table with a unique contraint on the Email field. I would naturally want to write blocker code in my service layer to ensure the email being added isn't already in the database before attempting to add anything. In the past I have never really seen anything wrong with it, however, as I have been exposed to more & more best practises/design principles I feel that this approach isn't very DRY.
So, is it correct to always ensure data going to the persistance layer is indeed "valid" or is it more natural to let the invalid data get to the database and handle the error?
Please don't do that.
Implementing even "simple" constraints such as keys is decidedly non-trivial in a concurrent environment. For example, it is not enough to query the database in one step and allow the insertion in another only if the first step returned empty result - what if a concurrent transaction inserted the same value you are trying to insert (and committed) in between your steps one and two? You have a race condition that could lead to duplicated data. Probably the simplest solution for this is to have a global lock to serialize transactions, but then scalability goes out of the window...
Similar considerations exist for other combinations of INSERT / UPDATE / DELETE operations on keys, as well as other kinds of constraints such as foreign keys and even CHECKs in some cases.
DBMSes have devised very clever ways over the decades to be both correct and performant in situations like these, yet allow you to easily define constraints in declarative manner, minimizing the chance for mistakes. And all the applications accessing the same database will automatically benefit from these centralized constraints.
If you absolutely must choose which layer of code shouldn't validate the data, the database should be your last choice.
So, is it correct to always ensure data going to the persistance layer is indeed "valid" (service layer) or is it more natural to let the invalid data get to the database and handle the error?
Never assume correct data and always validate at the database level, as much as you can.
Whether to also validate in upper layers of code depends on a situation, but in the case of key violations, I'd let the database do the heavy lifting.
Even though there isn't a conclusive answer, I think it's a great question.
First, I am a big proponent of including at least basic validation in the database and letting the database do what it is good at. At minimum, this means foreign keys, NOT NULL where appropriate, strongly typed fields wherever possible (e.g. don't put a text field where an integer belongs), unique constraints, etc. Letting the database handle concurrency is also paramount (as #Branko Dimitrijevic pointed out) and transaction atomicity should be owned by the database.
If this is moderately redundant, than so be it. Better too much validation than too little.
However, I am of the opinion that the business tier should be aware of the validation it is performing even if the logic lives in the database.
It may be easier to distinguish between exceptions and validation errors. In most languages, a failed data operation will probably manifest as some sort of exception. Most people (me included) are of the opinion that it is bad to use exceptions for regular program flow, and I would argue that email validation failure (for example) is not an "exceptional" case.
Taking it to a more ridiculous level, imagine hitting the database just to determine if a user had filled out all required fields on a form.
In other words, I'd rather call a method IsEmailValid() and receive a boolean than try to have to determine if the database error which was thrown meant that the email was already in use by someone else.
This approach may also perform better, and avoid annoyances like skipped IDs because an INSERT failed (speaking from a SQL Server perspective).
The logic for validating the email might very well live in a reusable stored procedure if it is more complicated than simply a unique constraint.
And ultimately, that simple unique constraint provides final protection in case the business tier makes a mistake.
Some validation simply doesn't need to make a database call to succeed, even though the database could easily handle it.
Some validation is more complicated than can be expressed using database constructs/functions alone.
Business rules across applications may differ even against the same (completely valid) data.
Some validation is so critical or expensive that it should happen prior to data access.
Some simple constraints like field type/length can be automated (anything run through an ORM probably has some level of automation available).
Two reasons to do it. The db may be accessed from another application..
You might make a wee error in your code, and put data in the db, which because your service layer operates on the assumption that this could never happen, makes it fall over if you are lucky, silent data corruption being worst case.
I've always looked at rules in the DB as backstop for that exceptionally rare occasion when I make a mistake in the code. :)
The thing to remember, is if you need to , you can always relax a constraint, tightening them after your users have spent a lot of effort entering data will be far more problematic.
Be real wary of that word never, in IT, it means much sooner than you wished.
question:
i'm building (or trying to build) a front end for an enterprise level web-app. the existing structure is driven by stored procedures (sql 2008 db). A stored proc is implemented as a class that ultimately handles the execution and the results are returned as an object.
i'm new to this game and would welcome an explanation about how my time would best be served...i read a repository pattern is a best practice, but all of the examples i read implement Entity Framework or Linq, etc..do i need an ORM? why or why not? i'd like to be able to have a maximum performance environment so that users can play with those result sets. thanks in advance
Well, I would suggest deciding on your use cases.
Some of the things that nHibernate / ORM's generally are not good for are:
Batch jobs.
Reporting
So if your work primarily involves either of those then you're best off not wasting your time, that being said there's nothing wrong with having multiple strategies... Building out a domain model is great for simplifying complex business rules, performance is generally very good too... Reporting and batch jobs can be built out separately, there's no reason why the different strategies can't co-exist... I woul however do my best to keep them decoupled...
So if you've got a big hairy business logic layer and it's littered with datasets / data access code and business logic IN your stored procedures then you will likely find it worth your while to invest in an ORM, but consider it a re-factoring step... IE you're improving existing code and making it testable before extending it...
In any case there's no one 'best' answer, the smartest thing I've done at previous companies has been to build new functionality (Test driven of course) in whichever data access pattern that seems to make sense to the functionality... Keep interfaces clean and decoupled... After doing that for awhile it usually becomes obvious which strategy / pattern is best suited for the application overall...
Good luck
Your question is somewhat unclear. Stored Procedures are SQL queries stored on the database which are used to interact with the data. However, it sounds like you already have an existing data access layer (DAL) which uses stored procedures and returns objects to you to play with. If this is the case, I would not throw away the entire data access layer and replace it with EF or any other ORM. Unless the existing DAL isn't working for you for either design or performance reasons, there's no reason to reinvent the wheel.
So I was searching the web looking for best practices when implementing the repository pattern with multiple data stores when I found my entire way of looking at the problem turned upside down. Here's what I have...
My application is a BI tool pulling data from (as of now) four different databases. Due to internal constraints, I am currently using LINQ-to-SQL for data access but require a design that will allow me to change to Entity Framework or NHibernate or the next data access du jour. I also hold steadfast to decoupled layers in my apps using an IoC framework (Castle Windsor in this case).
As such, I've used the Repository pattern to abstract the actual data access code from my business layer. As a result, my business object is coded against some I<Entity>Repository interface and the IoC Container is used to manage the actual implementation. In this case, I would expect to have a concrete Linq<Entity>Repository that implements the interface using LINQ-to-SQL to do the work. Later I could replace this with an EF<Entity>Repository with no changes required to my business layer.
Also, because I'm coding against the interface, I can easily mock the repository for unit testing purposes.
So the first question that I have as I begin coding the application is whether I should have one repository per DataContext or per entity (as I've typically done)? Let's say one database contains Customers and Sales with the expected relationship. Should I have a single OrderTrackingRepository with methods that work with both entities or have a separate CustomerRepository and a different SalesRepository?
Next, as a BI tool, the primary interface is for reporting, charting, etc and often will require a "mashup" of data across multiple sources. For instance, the reality is that one database contains customer information while another handles sales information and a third holds other financial information but one of my requirements is to display aggregated information that spans all three. Plus, I have to support dynamic filtering in the UI. Obviously working directly against the LINQ-to-SQL or EF DataContext objects (Table<Entity>, for instance) will allow me to pretty much do anything. What's the best approach to expose that same functionality to my business logic when abstracting the DAL with a repository interface?
This article: link text indicates that EF4 has turned this approach around and that the repository is nothing more than an IQueryable returned from the EF DataContext which brings up a whole other set of questions.
But, I think I've rambled on enough...
UPDATE (Thanks, Steven!)
Okay, let me put a more tangible (for me, at least) example on the table and clarify a few points that will hopefully lead to an approach I can better wrap my head around.
While I understand what Steven has proposed, I have a team of developers I have to consider when implementing such things and I'm afraid they will get lost in the complexity (yes, a real problem here!).
So, let's remove any direct tie-in with Linq-to-Sql because I don't want a solution that is dependant upon the way L2S works - or even EF, for that matter. My intent has been to abstract away the data access technology being used so that I can change it as needed without requiring collateral changes to the consuming code in my business layer. I've accomplished this in the past by presenting the business layer with IRepository interfaces to work against. Perhaps these should have been named IUnitOfWork or, more to my liking, IDataService, but the goal is the same. These interfaces typically exposed methods such as Add, Remove, Contains and GetByKey, for example.
Here's my situation. I have three databases to work with. One is DB2 and contains all of the business information for a customer (franchise) such as their info and their Products, Orders, etc. Another, SQL Server database contains their financial history while a third SQL Server database contains application-specific information. The first two databases are shared by multiple applications.
Through my application, the customer may enter/upload their financial information for a given time period. When entered, I have to perform the following steps:
1.Validate the entered data against a set of static rules. For example, the data must contain a legitimate customer ID value (in the case of an upload). This requires a lookup in the DB2 database to verify that the supplied customer ID exists and is current.
2.Next I have to validate the data against a set of dynamic rules which are contained in the third (SQL Server) database. An example may be that a given value cannot exceed a certain percentage of another value.
3.Once validated, I persist the data to the second SQL Server database containing the financial data.
All the while, my code must have loosely-coupled dependencies so I may mock them in my unit tests.
As part of the analysis, I know that I have three distinct data stores to work with and about a half-dozen or so entities (at this time) that I am working with. In generic terms, I presume that I would have three DataContexts in my application, one per data store, with the entities exposed by the appropriate data context.
I could then create a separate I{repository|unit of work|service} for each entity that would be consumed by my business logic with a concrete implementation that knows which data context to use. But this seems to be a risky proposition as the number of entities increases, so does the number of individual repository|UoW|service types.
Then, take the case of my validation logic which works with multiple entities and, thereby, multiple data contexts. I'm not sure this is the most efficient way to do this.
The other requirement that I have yet to mention is on the reporting side where I will need to execute some complex queries on the data stores. As of right now, these queries will be limited to a single data store at a time, but the possibility is there that I might need to have the ability to mash data together from multiple sources.
Finally, I am considering the idea of pulling out all of the data access stuff for the first two (shared) databases into their own project and have been looking at WCF Data Services as a possible approach. This would give me the basis for a consistent approach for any application making use of this data.
How does this change your thinking?
In your case I would recommend returning IEnummerables's for your data queries for the repo. I usually aggregate calls from multiple repo's through a service class that represents the domain problem and encapsulates my business logic. To keep it clean I try keep my repros focused on the domain problem. I liken my Datacontext to a repo, and extract an interface using a T4 template to make life easier for mocking. But there is nothing stopping you using a traditional repo that encapsulates your calls. Doing it this way will allow you to switch ORM's at any stage.
EDIT: IQueryable IS NOT THE ANSWER! :-)
I have also done a lot of work in this area, and INITIALLY came to the same conclusion, however it is NOT a good solution. The point of the Repo is to abstract queries into discrete chunks of work. Exposing IQueryable is too adhoc and raises some issues later down the line. You loose your ability to scale. You loose your ability to optimize queries (Lets say I want to move to a highly optimized stored proc). You loose your ability to use IoC for the repo to switch out data access layers (switch the project from SQL to Mongo). You loose your ability to provide effective data caching in the Repo (Which is a major strength in the Repo pattern). I would recommend taking a CLOSE look as to WHY we have a Repo pattern. It isn't simply an "ORM" mapping layer. What made this really clear to me was the CQRS pattern.
Further to this allowing the ad-hoc nature of IQueryable opens you to misfitting reuse of queries. It is GENERALLY not a good idea to reuse queries, since query to query you see slight deviations, which ends up with 2 byproducts: Queries become too broad and inefficient. Queries become riddled with unmaintainable IF THEN statements to cater for the deviations.
IQueryable is easy, but opens you up to an unmaintainable mess.
Look at this SO answer. I think it shows a simplified model of what you want. IQueryable<T> is indeed our new Repository :-). DataContext and ObjectContext are our Unit of Work.
UPDATE 2:
Here is a blog post that describes the model you might be looking for.
UPDATE 3
It would be wise to hide the shared databases behind a service. This will solve several problems:
This will make the database private to the service, which makes it much easier to change the implementation when needed.
You can put the needed validation logic (for database 1) in that service and can create tests for that validation logic in that project.
Clients accessing that service can assume correctness of the service, and its validation logic.
The result of this is that your application will send data to the service to validate it. Call the service to fetch data. Query its own private database (database 3) and join the data of the three data source locally together. I've never been a fan of using cross-database or even cross-server (in your situation) database calls and letting the database join everything together. Transactions will be promoted to distributed-transactions and it's hard to predict how many data the servers will exchange.
When you abstract the shared databases behind the service, things get easier (at least from your application's point of view). Your application calls services it trusts which limits the amount of code in that application and the amount of tests. You still want to mock the calls to such a service, but that would be pretty easy. It should also solve the problem of validating over multiple data sources.
Validation is always a hard part. I'm very familiar with Validation Application block, and love it for it's flexibility. It isn't however an easy framework, but you might take a peek at what you can do with it. For instance, I've written several articles about integration with O/RM tools and how to 'embed' a context (context as in DataContext/Unit of Work) in Validation Application Block.
Please have a look at my IRepository pattern implementation using EF 4.0.
My solution has the following features:
supports connections to multiple dbs
One repository per entity
Support for execution of queries
Unit of work pattern implementation
Support for validating entities using VAB guidance
Common operations are kept at base class level. High use of OOPS techniques for code re-usability and ease of maintenance.
Given the premise:
There are competent sql programmers
(correlary - writing sql queries are not an issue)
There are competent application developers
(correlary - there is simple/strong/flexible architecture for handling connections and simple queries from code)
Why do people use linq to sql?
There is overhead added to each transaction
There is strong likelihood of performance loss for moderate-complex calculations (DBs are made for processing sets and calculations and had teams of engineers working out optimization - why mess with this?)
There is loss of flexibility (if you want to add another ui (non .NET app) or access method, you either have to put the queries back in the db or make a separate data access layer)
There is loss of security by not having a centralized control of write/update/read on db (for example, a record has changed - if you allow applications to use linq to sql to update, then you cannot prove which application changed it or what instance of an application changed it)
I keep seeing questions about linq to sql and am wondering if I'm missing something.
I keep seeing questions about linq to sql and am wondering if I'm missing something.
It's not that you're missing something. It's that you have something most shops don't have:
There are competent sql programmers
Additionally, in your shop those competent sql programmers prefer to write sql.
Here's a point by point response:
There is overhead added to each transaction
Generally true. This can be avoided by translating the queries before they are needed to run using CompiledQuery for many (but not all!) scenarios.
There is strong likelihood of performance loss for moderate-complex calculations (DBs are made for processing sets and calculations and had teams of engineers working out optimization - why mess with this?)
Either you're writing linq, which is translated to sql, and then a plan is generated from the optimizer - or your writing sql from which a plan is generated by the optimizer. In both cases you are telling the machine what you want and it is supposed to figure out how to do it. Are you suggesting that subverting the optimizer by using query hints is a good practice? Many competent sql programmers will disagree with that suggestion.
There is loss of flexibility (if you want to add another ui (non .NET app) or access method, you either have to put the queries back in the db or make a separate data access layer)
A lot of people using linq are already SOA. The linq lives in a service. The non .NET app calls the service. Bada-bing bada-boom.
There is loss of security by not having a centralized control of write/update/read on db (for example, a record has changed - if you allow applications to use linq to sql to update, then you cannot prove which application changed it or what instance of an application changed it)
This is simply not true. You prove which application is connected and issuing sql commands the same way you prove which application is connected and calling a sproc.
Let me list you a few points:
There are small software companies or mid-sized companies who develop their software in-house who might rather focus on getting many application developers than getting a freelancer DB developer or even permanently hire one.
In most cases the overhead is a non-issue either due to the amount of data to be processed or due to the low traffic. Besides, when used properly, LINQ to SQL can perform as fast as most SQL queries + the associated .net code.
Many companies just stick with the Microsoft stack and they can only enjoy the integration. Some other company develops using SOA there's just no problem. The others aren't forced to choose LINQ-to-SQL and if they make that choice is their problem how to integrate it. Nobody ever said LINQ-to-SQL is a silver bullet :)
I believe security is gained with LINQ-to-SQL because I've bumped across lots of SQL queries taking in unescaped data with string concatenation etc and explaining the whole parametrized query idea has never been easy. Besides since all queries are eventually translated into SQL, unless the tracking issue you describe would happen via a stored procedure, there're again no problems at all.
I also believe your question can be posed more generally to address all ORMs and not just LINQ-to-SQL, and still most of what I said would hold true.
The problem is that it is very rare for somewhere to have a competent SQL developer who likes writing SQL and wouldn't rather be doing something else. I would consider myself competent in SQL, I used to do all my data access layers with stored procs or parametrized queries. Trouble is that it takes ages and is dull. I'd rather be writing great applications than messing around with data access layers that essentially have a select, insert, update and delete SQL statement(or proc) repeated dozens of times for each data object.
Linq-to-SQL takes away some of the repetitive nature. It has a tool to auto generate you business objects from your database schema, and it gives you a nice integrated query language that is compile time type verified and is in your code (Stored procs are a pain to source control neatly)
I can write a DAL in Linq-to-sql several times faster than I can using plain SQL, stored procs or parametrized queries.
If you want to maintain the use of stored procs both linq-to-sql and the EF both support the use of stored procs for all their data access, you just have to set up the appropriate mappings. So, you can still use your stored procs to log details and implement security if you want. We tend to opt for using windows auth, and use that to restrict access to each table for the various users, then we have a bunch of triggers on the tables that track details for audit purposes.
Two things I will quickly note is that firstly, the entity framework seems to be getting more support from MS at the moment, and I suspect that will be considered the kind of default standard for the future in preference to linq-to-sql. Secondly, in .Net 3.5 the EF and linq-to-sql do not have very good support for n-tier disconnected apps. In both of them you kind of have to muck around with either serializing data contexts across your disconnected tiers, or manually detach and re-attach your data objects. This is much improved in the .net 4.0 though. Just something to consider depending on which version you have available to you.
Existing question/answers in the same vein/spirit:
Doesn't Linq to SQL miss the point? Aren't ORM-mappers (SubSonic, etc.) sub-optimal solutions?
LINQ-to-SQL vs stored procedures?
What's wrong with Linq to SQL?
Why do I need Stored Procedures when I have LINQ to SQL
If using LINQ to SQL is there any good reason to learn SQL queries/syntax anymore?
https://stackoverflow.com/questions/216569/are-the-days-of-the-stored-procedure-numbered
I personally believe there's no right or wrong answer. It depends on what you're developing and how you're developing it. If you need razor-sharp performance, have an overly-complex data model, etc... skip the abstraction. If you feel the abstraction speeds up your development time, like the idea of capturing all application logic in a single codebase, etc... use it.
For me, it takes a lot less time to write linq to sql code than it does to write a bunch of stored procedures. That's especially true when the design isn't finished, in that case I don't yet know how much of the work I want to do on C# objects, and how much I want to do in SQL.
So, I can skip building datasets, I don't have to click click click to add queries, basically, linq to sql means I can change my code in less time.
Also, as a big fan of Haskell, I can write lots of functional-style code with linq to sql and it just works.
I'm not saying this is an ideal solution or even a great example (it was the result of a high level constraint on the architecture, not something we necessarily would have chosen from scratch), but...
I worked on an app where the code was completely isolated from the database except through a set of exposed stored procs. The code could not "know" anything about the database schema except was was returned from the stored procs.
While this isn't that unusual and it isn't too hard to write a DAL using ADO or whatever, I decided to try out Linq to Sql, even though it wouldn't be using it for its real intended purpose and wouldn't use most of the features. Turns out it was a great decision.
I created the Linq to Sql class, dragged the stored procs from server explorer onto the right side of the designer, then... Wait, there is no then. I was pretty much done.
Linq created strongly typed methods for each stored proc. For the procs that returned rows of data, Linq automatically created a class for the items in each row and returned a List<generatedClass> for them. I wrapped the calls themselves in a lightweight public DAL class that did some verification and some automatic parameter setting and I was done. I wrote a business object class and mapped the dynamically generated Linq class objects to the business object (did this by hand, but it isn't hard to do or maintain).
The program is now immune to any schema change that doesn't affect the stored procedure signatures. If the signatures do change, we just drag off the old proc from the design and drag it back to regenerate the code. A few passes through the unit tests to make changes (which usually don't go higher than the public DAL interface) and it's done. Things upstream of the DAL use Linq to Objects techniques to select, filter, and sort data that isn't in the right format straight from the stored proc calls.
We have some excellent DBAs writing the stored procedures and an entirely different group writing the other code, so maybe it is a good example of why (and how) you can use LINQ in the scenario you describe.
Some handy features are the debugger picking up sytax errors in your query, compared to writing SQL statements as strings. Mistakes that wont get picked up until runtime.
Plus I find LINQ statements easier to read than SQL.
It may be a case of convenience triumphing over performance. If you're programming at Facebook levels of uber-performance then you might think about every clock cycle but the simple truth is that the majority of applications don't need this attention and benefit from efficiencies in code maintenance and dev time (100k contractor vs. another $erver).
That said, there's a case for outsourcing as much of the query processing from the DB box in very high scale systems, else the DB is the bottle neck and you need to shard or re-architect down the line. Costly.
I think its fair to say that LINQ will scale better/easier both in terms of servers and from many core in that your LINQ codebase will get m-core for 'free' as soon as MS release C# 4.0.
I do see your point in asking and as a non-ASP.NET dev just beginning a www project for the first time, I can't see the point of 80% of ASP.NET (themes, controls etc.) - it seems I need to learn more and code more than the HTML itself! -- again, I'm sure there's an good reason for it all.
--- I haven't got the 50 pts to comment on the post I want to so I'm doing it here ---
David B suggests that writing some SQL is all there is to getting the most out of SQL Server and that using query hints is the steering mechanism. The same task can be achieved in many different ways in SQL and many with 1000s of times the performance gain. I suggest reading Inside T-SQL Querying (Itzik Ben Gan). Over and over, Itzik shows how to rethink the query and use new commands to shrink the logical reads sometimes from thousands into less than ten.
For very simple queries, the overhead of an extra layer adds to the roundtrip cost. For somewhat more complex queries in normal 'business app' scenarios, the optimizations done by the Linq-to-SQL expression->sql translation magic can often save a lot.
As an example, I recently did a 1:1 translation of a customer-supplied 1400+ (!) line stored proc to L2S. Not only did it go from 1400 lines of SQL to 500 lines of much more readable, strongly typed, and commented code. It also started hitting the database with an average of ~1500 reads instead of ~30k reads. This is before I even started looking at db-side optimizations - that saving is something I can 100% attribute to L2S's ability to eliminate predicates that can be evaluated client-side.
simple answer, there are two approaches: create exquisite Rube Goldberg contraptions, or just get the job done in a simple way. Many devs lean towards the former.
Developers get bored easily, and would often personally enjoy doing things a harder way that seems to provide a certain intellectual beauty. Are you developing an app or writing a PhD? As my msft director used to yell in the hallways, "I don't want another research project!"
please repeat after me (min 3x)
there is no silver bullet
I will not use a technology just because its the latest thing from msft
I will not use something just to get it on my resume
Not only are their competent SQL coders, any decent app programmer, especially LOB apps, should write intermediate SQL. If you don't know any SQL and are writing LINQ to SQL, how are you going to debug your data calls? How are you going to profile them to fix bottlenecks?
We're trying out LINQ to SQL and I think there are major issues with it, such as:
There is no simple way to return the query results to another object. This in itself seems insane. Microsoft created the var anonymous datatype, and recommends using it, but there is no way to move this data out of your local method, hence the oo paradigm breaks if you have to use the data in the same function that retrieved it.
Tables are NOT objects. Study up on 3rd normal form etc. Relational databases are for storing data, not using it. I don't want to be restricted or encouraged to use my tables as objects. The data I retrieve from the database will very often be joins of multiple tables, and may include SQL casts, functions, operators, etc.
There is no performance gain, and a slight loss
Now I have way more code to worry about. There are the dbml files and still a DAL to actually write the LINQ. Yes, lots of it is machine-generated, that doesn't mean its not there, its something else that can go wrong (i.e. your dbml files, etc.).
Now that I've given the background, I will attempt to answer you actual question, why do people use LINQ To SQL:
Its the latest thing from Microsoft and I want it on my resume.
Msft has convinced managers/execs that it will decrease coding time
Developers hate SQL. (no good dev environment or debugging except manually--it would be nice to have better intellisense to a sql tool.)
I encourage people not to jump on the bandwagon just because everyone else is, learn enough to put it on your resume, be willing to use it if forced to, but try and really understand the pros and cons first.
I have been playing with some LINQ ORM (LINQ directly to SQL) and I have to admit I like its expressive powers . For small utility-like apps, It also works quite fast: dropping a SQL server on some surface and you're set to linq away.
For larger apps however, the DAL never was that big of an issue to me to setup, nor maintain, and more often than not, once it was set, all the programming was not happening there anyway...
My, honest - I am an ORM newbie - question : what is the big advantage of ORM over writing a decent DAL by hand?
(seems like a double, couldn't find it though)
UPDATE : OK its a double :-) I found it myself eventually :
ORM vs Handcoded Data Access Layer
Strong-typing
No need to write the DAL yourself => time savings
No need to write SQL code yourself =>
less error-prone
I've used Hibernate in the past to dynamically create quite complex queries. The logic involved to create the appropriate SQL would have been very time-consuming to implement, compared with the logic to build the appropriate Criteria. Additionally, Hibernate knew how to work with various different databases, so I didn't need to put any of that logic in our code. We had to test against different databases of course, and I needed to write an extension to handle "like" queries appropriately, but then it ran against SQL Server, Oracle and HSqldb (for testing) with no issues.
There's also the fact that it's more code you don't have to write, which is always a nice thing :) I can't say I've used LINQ to SQL in anything big, but where I've used it for a "quick and dirty" web-site (very small, rarely updated, little benefit from full layer abstraction) it was lovely.
I used JPA in a project, and at first I was extremely impressed. Gosh it saved me all that time writing SQL! Gradually, however, I became a bit disenchanted.
Difficulty defining tables without surrogate keys. Sometimes we need tables that don't have surrogate keys. Sometimes we want a multicolumn primary key. TopLink had difficulties with that.
Forced datastructure relationships. JPA uses annotations to describe the relationship between a field and the container or referencing class. While this may seem great at first site, what do you do when you reference the objects differently in the application? Say for example, you need just specific objects that reference specific records based on some specific criteria (and it needs to be high-performance with no unnecessary object allocation or record retrieval). The effort to modify Entity classes will almost always exceed the effort that would have existed had you never used JPA in the first place (assuming you are at all successful getting JPA to do what you want).
Caching. JPA defines the notion of caches for your objects. It must be remembered that the database has its own cache, typically optimized around minimizing disk reads. Now you're caching your data twice (ignoring the uncollected GC heap). How this can be an advantage is beyond me.
Data != Objects. For high-performance applications, the retrieval of data from the DB must be done very efficiently. Forcing object creation is not always a good thing. For example, sometimes you may want arrays of primitives. This is about 30 minutes of work for an experienced programmer working with straight JDBC.
Performance, debugging.
It is much more difficult to gauge the performance of an application with complex things going on in the (sub-optimal, autogenerated) caching subsystem, further straining project resources and budgets.
Most developers don't really understand the impedence mismatch problem that has always existed when mapping objects to tables. This fact ensures that JPA and friends will probably enjoy considerable (cough cough) success for the forseeable future.
Well, for me it is a lot about not having to reinvent/recreate the wheel each time I need to implement a new domain model. It is simply a lot more efficient to use for instance nHibernate (my ORM of choice) for creating, using and maintaining the data access layer.
You don't specify exactly how you build your DAL, but for me I used to spend quite some time doing the same stuff over and over again. I used to start with the database model and work my way up from there, creating stored procedures etc. Even if I sometimes used little tools to generate parts of the setup, it was a lot of repetitive coding.
Nowadays I start with the domain. I model it in UML, and for most of the time I'm able to generate everything from that model, including the database schema. It need a few tweaks here and there, but with my current setup I get 95% of the job with the data access done in no time at all. The time I save I can use to fine tune the parts that need tuning. I seldom need to write any SQL statements.
That's my two cents. :-)
Portability between different db vendors.
My, honest - i am an ORM newbie - question : what is the big advance of ORM over writing a decent DAL by hand?
Not all programmers are willing or even capable of writing "a decent DAL". Those who can't or get scared from the mere thought of it, find LINQ or any other ORM a blessing.
I personally use LINQ to manipulate collections in the code because of its expressiveness. It offers a very compact and transparent way to perform some common tasks on collections directly in code.
LINQ will stop being useful to you when you will want to create very specific and optimized queries by hand. Then you are likely to get a mixture of LINQ queries intermingled with custom stored procedures wired into it. Because of this considerations, I decided against LINQ to SQL in my current project (since I have a decent (imho) DAL layer). But I'm sure LINW will do just fine for simple sites like maybe your blog (or SO for that matter).
With LINQ/ORM there may also be a consideration of lagging for high traffic sites (since each incoming query will have to be compiled all over again). Though I have to admit I do not see any performance issues on SO.
You can also consider waiting for the Entity Framework v2. It should be more powerful than LINQ (and hopefully not that bad as v1 (according to some people)).
Transparent persistence - changes get saved (and cascaded) without you having to call Save(). At first glance this seems like a nightmare, but once you get used to working with it rather than against it, your domain code can be freed of persistence concerns almost completely. I don't know of any ORM other than Hibernate / NHibernate that does this, though there might be some...
The best way to answer the question is to understand exactly what libraries like Hibernate are actually accomplishing on your behalf. Most of the time abstractions exist for a reason, often to make certain problems less complex, or in the case Hibernate is almost a DSL for expression certain persistance concepts in a simple terse manner.
One can easily change the fetch strategy for collections by changing an annotation rather than writing up lots of code.
Hibernate and Linq are proven and tested by many, there is little chance you can achieve this quality without lots of work.
Hibernate addresses many features that would take you months and years to code.
Also, while the JPA documentation says that composite keys are supported, it can get very (very) tricky quickly. You can easily spend hours (days?) trying to get something quite simple working. If JPA really makes things simpler then developers should be freed from thinking too much about these details. It doesn't, and we are left with having to understand two levels of abstraction, the ORM (JPA) and JDBC. For my current project I'm using a very simple implementation that uses a package protected static get "constructor" that takes a ResultSet and returns an Object. This is about 4 lines of code per class, plus one line of code for each field. It's simple, high-performance, and quite effective, and I retain total control. If I need to access objects differently I can add another method that reads different fields (leaving the others null, for example). I don't require a spec that tells me "how ORMs must (!) be done". If I require caching for that class, I can implement it precisely as required.
I have used Linq, I found it very useful. I saves a lot of your time writing data access code. But for large applications you need more than DAL, for them you can easily extent classes created by it. Believe me, it really improves your productivity.