I need your help to one strategic question in my programing for database.
I work in Delphi 2010 and SQL Server 2012. Components Sdac for access to database. I quite often use a stored procedure for very many often absolutely different tasks. Fortunately SQL Server allows different result sets from one stored procedure.
I have a parameter #Wmode int (and several others too) and each value of wmode is separate task. Some of this are heavy other not, some have quite a lot of code some very few.
I use this approach because I try to put as much logic of application in code of stored procedures because at least for me its much easier to make any modifications, to test, debug, to set logs or to execute separately of my application.
Tell me please how much this approach can affect a performance this requests to database and how i can improve it. Is it better for me to use such approach only to seldomly executing parts of application. Can you give some links read on this subject.
I have a ~2,300 line package, which is split into many procedures & function. It is running slower than I would like. Many years ago on a previous release of Oracle (9i or 11g) I had a similar problem and I was able to build a hierarchical structure which contained everything that was executed in the procedure/package an how much time was spent on each item.
I cannot seem to find a tutorial/blog that shows how to accomplish this. It is probably done with the DBMS_STATS package, but I find Oracle's documentation unsuitable for task oriented problem solving. It may be great if you want to learn everything there is to know about a subject, but generally all I need to know is how to solve the issue I am currently working on.
At any rate can someone point me to how I can get the runtime statistics of a run of an Oracle Procedure?
There are two options:
dbms_profiler - records times on each statement executed. docs
dbms_hpof - similar, but collects statistics on a hierarchy of pl/sql calls docs
Either method will require some setup using SYS (dba) access. Setup Instuctions for dbms_profiler
Did you tried dbms_utility.get_time method as described in THIS post.
Link to the original question which is almost the same as your question.
https://www.quora.com/How-can-I-log-the-execution-time-of-a-stored-procedures-in-a-table-in-Oracle-database
Have you applied test driven development to purely sql scripts? if so what has been your experience. Is it worth it? What are the rewards? disadvantages ? etc.
I have played around a bit, to be honest I would rather generate my regular DB code. I was this awhile ago and thought it was interesting. http://sourceforge.net/apps/trac/tsqlunit/
Most (all?) of my database "scripts" are generated, not hand-written. And, I avoid stored procedures and views. I basically treat my database as a file. The testing and logic stays in the application layer (where it belongs, IMO).
This approach works pretty well for me and the kinds of applications that I develop. It might not work so well in other situations.
For me the answer to your question is "not applicable".
SQL was the vehicle for one of my very first TDD collaborations. This was in a setting where I was an application developer (C++, I think, but it's been a while) and we had a DBA with responsibility for all queries. I wouldn't choose to go that route again, but that's another story. The time came when I needed a new query, so I wrote up some test data and expected results and sent it to the DBA; he wrote the script and thanked me for making the requirements so clear and precise.
TDD as it's usually practiced doesn't fit well with SQL (or maybe the other way 'round), but it's not really all that hard to adapt the practice to work well with the language. One-button testing may be a little harder to bring into the mix, but it's rarely hard to run a query.
I'm using a version of SQL Server 2005 that does not support the profiler, trying to figure out how best to compare the performance of two stored procedures. I've run the execution plan for each, but it's not clear to me which of the provided metrics I should be focusing on. Do I go through and add up the various costs? What's the best approach?
Thanks in advance.
Look at this article: Measuring SQL Performance
If you don't want to register to free account, here is a solution 1:
DECLARE #start datetime, #stop datetime
SET #start = GETDATE()
EXEC your_sp
SET #stop = GETDATE()
2nd:
SET STATISTICS TIME ON
EXEC your_sp
3rd:
SET STATISTICS IO ON
EXEC your_sp
Btw, this site has some nice articles. I'd recommend to register. It's free.
The question is what are you optimizing for? Is it for speed or resources used?
If speed, then in the query analyzer I would look at the execution between several runs, make changes and time them again.
If it is resources then I would look through the execution plan. In that case I would start with the worse offenders and work my way down the list. Adding them up will tell you the over all performance, but most cases it is an item or 2 that is the bottle neck.
if you are using something like
SET SHOWPLAN_ALL ON
look at the TotalSubtreeCost column value for the row with the EXE YourProcedureName
this might help:
http://technet.microsoft.com/en-us/library/ms180765.aspx
Like most questions, the answer depends... In the final analysis, the only measure that matters is end-user perception, which can be affected by many things,including not only the stored procedure, but network performance, usage patterns (is the sProc being called 20x/day, or 1000x/ second?), etc., - and the sProc may not be the determining factor.
But if the stored procedure is the "piece if the puzzle" that is having the major adverse impact on the end-user perception of some function, then, you have to look at elapsed time to run the stored procedure. But this itself can be affected by numerous underlying metrics, and to do anything about it you need to analyse them all to determine which of them is the major or overriding contributer to the overall stored proc performance.
You could always rig a test harness to call your stored procedures and measure the call times. Unfortunately you're not going to get the details about exactly which parts of the Stored Procedure are causing the slow-down.
You could always run the Stored Procedure by hand in Query Analyzer and measure the results that way as well. The .NET harness just automates the process for you.
The simple low-brow solution is to run them with print statements printing the execution time over the various parts. This won't help if the performance problem is more subtle and found in production only, but if you can reproduce it in your test environment, you should be fine.
One handy technique if you are trying to compare the performance of two procs or statements is to select both blocks of sql in query analyzer and run the query plan. The plan will tell you the cost percentage of each block relative to one another. This is not full proof. I have seen it tell me one was cheaper when it was clearly more expensive when actually ran, but for the most part it is a nice, quick trick.
Given the premise:
There are competent sql programmers
(correlary - writing sql queries are not an issue)
There are competent application developers
(correlary - there is simple/strong/flexible architecture for handling connections and simple queries from code)
Why do people use linq to sql?
There is overhead added to each transaction
There is strong likelihood of performance loss for moderate-complex calculations (DBs are made for processing sets and calculations and had teams of engineers working out optimization - why mess with this?)
There is loss of flexibility (if you want to add another ui (non .NET app) or access method, you either have to put the queries back in the db or make a separate data access layer)
There is loss of security by not having a centralized control of write/update/read on db (for example, a record has changed - if you allow applications to use linq to sql to update, then you cannot prove which application changed it or what instance of an application changed it)
I keep seeing questions about linq to sql and am wondering if I'm missing something.
I keep seeing questions about linq to sql and am wondering if I'm missing something.
It's not that you're missing something. It's that you have something most shops don't have:
There are competent sql programmers
Additionally, in your shop those competent sql programmers prefer to write sql.
Here's a point by point response:
There is overhead added to each transaction
Generally true. This can be avoided by translating the queries before they are needed to run using CompiledQuery for many (but not all!) scenarios.
There is strong likelihood of performance loss for moderate-complex calculations (DBs are made for processing sets and calculations and had teams of engineers working out optimization - why mess with this?)
Either you're writing linq, which is translated to sql, and then a plan is generated from the optimizer - or your writing sql from which a plan is generated by the optimizer. In both cases you are telling the machine what you want and it is supposed to figure out how to do it. Are you suggesting that subverting the optimizer by using query hints is a good practice? Many competent sql programmers will disagree with that suggestion.
There is loss of flexibility (if you want to add another ui (non .NET app) or access method, you either have to put the queries back in the db or make a separate data access layer)
A lot of people using linq are already SOA. The linq lives in a service. The non .NET app calls the service. Bada-bing bada-boom.
There is loss of security by not having a centralized control of write/update/read on db (for example, a record has changed - if you allow applications to use linq to sql to update, then you cannot prove which application changed it or what instance of an application changed it)
This is simply not true. You prove which application is connected and issuing sql commands the same way you prove which application is connected and calling a sproc.
Let me list you a few points:
There are small software companies or mid-sized companies who develop their software in-house who might rather focus on getting many application developers than getting a freelancer DB developer or even permanently hire one.
In most cases the overhead is a non-issue either due to the amount of data to be processed or due to the low traffic. Besides, when used properly, LINQ to SQL can perform as fast as most SQL queries + the associated .net code.
Many companies just stick with the Microsoft stack and they can only enjoy the integration. Some other company develops using SOA there's just no problem. The others aren't forced to choose LINQ-to-SQL and if they make that choice is their problem how to integrate it. Nobody ever said LINQ-to-SQL is a silver bullet :)
I believe security is gained with LINQ-to-SQL because I've bumped across lots of SQL queries taking in unescaped data with string concatenation etc and explaining the whole parametrized query idea has never been easy. Besides since all queries are eventually translated into SQL, unless the tracking issue you describe would happen via a stored procedure, there're again no problems at all.
I also believe your question can be posed more generally to address all ORMs and not just LINQ-to-SQL, and still most of what I said would hold true.
The problem is that it is very rare for somewhere to have a competent SQL developer who likes writing SQL and wouldn't rather be doing something else. I would consider myself competent in SQL, I used to do all my data access layers with stored procs or parametrized queries. Trouble is that it takes ages and is dull. I'd rather be writing great applications than messing around with data access layers that essentially have a select, insert, update and delete SQL statement(or proc) repeated dozens of times for each data object.
Linq-to-SQL takes away some of the repetitive nature. It has a tool to auto generate you business objects from your database schema, and it gives you a nice integrated query language that is compile time type verified and is in your code (Stored procs are a pain to source control neatly)
I can write a DAL in Linq-to-sql several times faster than I can using plain SQL, stored procs or parametrized queries.
If you want to maintain the use of stored procs both linq-to-sql and the EF both support the use of stored procs for all their data access, you just have to set up the appropriate mappings. So, you can still use your stored procs to log details and implement security if you want. We tend to opt for using windows auth, and use that to restrict access to each table for the various users, then we have a bunch of triggers on the tables that track details for audit purposes.
Two things I will quickly note is that firstly, the entity framework seems to be getting more support from MS at the moment, and I suspect that will be considered the kind of default standard for the future in preference to linq-to-sql. Secondly, in .Net 3.5 the EF and linq-to-sql do not have very good support for n-tier disconnected apps. In both of them you kind of have to muck around with either serializing data contexts across your disconnected tiers, or manually detach and re-attach your data objects. This is much improved in the .net 4.0 though. Just something to consider depending on which version you have available to you.
Existing question/answers in the same vein/spirit:
Doesn't Linq to SQL miss the point? Aren't ORM-mappers (SubSonic, etc.) sub-optimal solutions?
LINQ-to-SQL vs stored procedures?
What's wrong with Linq to SQL?
Why do I need Stored Procedures when I have LINQ to SQL
If using LINQ to SQL is there any good reason to learn SQL queries/syntax anymore?
https://stackoverflow.com/questions/216569/are-the-days-of-the-stored-procedure-numbered
I personally believe there's no right or wrong answer. It depends on what you're developing and how you're developing it. If you need razor-sharp performance, have an overly-complex data model, etc... skip the abstraction. If you feel the abstraction speeds up your development time, like the idea of capturing all application logic in a single codebase, etc... use it.
For me, it takes a lot less time to write linq to sql code than it does to write a bunch of stored procedures. That's especially true when the design isn't finished, in that case I don't yet know how much of the work I want to do on C# objects, and how much I want to do in SQL.
So, I can skip building datasets, I don't have to click click click to add queries, basically, linq to sql means I can change my code in less time.
Also, as a big fan of Haskell, I can write lots of functional-style code with linq to sql and it just works.
I'm not saying this is an ideal solution or even a great example (it was the result of a high level constraint on the architecture, not something we necessarily would have chosen from scratch), but...
I worked on an app where the code was completely isolated from the database except through a set of exposed stored procs. The code could not "know" anything about the database schema except was was returned from the stored procs.
While this isn't that unusual and it isn't too hard to write a DAL using ADO or whatever, I decided to try out Linq to Sql, even though it wouldn't be using it for its real intended purpose and wouldn't use most of the features. Turns out it was a great decision.
I created the Linq to Sql class, dragged the stored procs from server explorer onto the right side of the designer, then... Wait, there is no then. I was pretty much done.
Linq created strongly typed methods for each stored proc. For the procs that returned rows of data, Linq automatically created a class for the items in each row and returned a List<generatedClass> for them. I wrapped the calls themselves in a lightweight public DAL class that did some verification and some automatic parameter setting and I was done. I wrote a business object class and mapped the dynamically generated Linq class objects to the business object (did this by hand, but it isn't hard to do or maintain).
The program is now immune to any schema change that doesn't affect the stored procedure signatures. If the signatures do change, we just drag off the old proc from the design and drag it back to regenerate the code. A few passes through the unit tests to make changes (which usually don't go higher than the public DAL interface) and it's done. Things upstream of the DAL use Linq to Objects techniques to select, filter, and sort data that isn't in the right format straight from the stored proc calls.
We have some excellent DBAs writing the stored procedures and an entirely different group writing the other code, so maybe it is a good example of why (and how) you can use LINQ in the scenario you describe.
Some handy features are the debugger picking up sytax errors in your query, compared to writing SQL statements as strings. Mistakes that wont get picked up until runtime.
Plus I find LINQ statements easier to read than SQL.
It may be a case of convenience triumphing over performance. If you're programming at Facebook levels of uber-performance then you might think about every clock cycle but the simple truth is that the majority of applications don't need this attention and benefit from efficiencies in code maintenance and dev time (100k contractor vs. another $erver).
That said, there's a case for outsourcing as much of the query processing from the DB box in very high scale systems, else the DB is the bottle neck and you need to shard or re-architect down the line. Costly.
I think its fair to say that LINQ will scale better/easier both in terms of servers and from many core in that your LINQ codebase will get m-core for 'free' as soon as MS release C# 4.0.
I do see your point in asking and as a non-ASP.NET dev just beginning a www project for the first time, I can't see the point of 80% of ASP.NET (themes, controls etc.) - it seems I need to learn more and code more than the HTML itself! -- again, I'm sure there's an good reason for it all.
--- I haven't got the 50 pts to comment on the post I want to so I'm doing it here ---
David B suggests that writing some SQL is all there is to getting the most out of SQL Server and that using query hints is the steering mechanism. The same task can be achieved in many different ways in SQL and many with 1000s of times the performance gain. I suggest reading Inside T-SQL Querying (Itzik Ben Gan). Over and over, Itzik shows how to rethink the query and use new commands to shrink the logical reads sometimes from thousands into less than ten.
For very simple queries, the overhead of an extra layer adds to the roundtrip cost. For somewhat more complex queries in normal 'business app' scenarios, the optimizations done by the Linq-to-SQL expression->sql translation magic can often save a lot.
As an example, I recently did a 1:1 translation of a customer-supplied 1400+ (!) line stored proc to L2S. Not only did it go from 1400 lines of SQL to 500 lines of much more readable, strongly typed, and commented code. It also started hitting the database with an average of ~1500 reads instead of ~30k reads. This is before I even started looking at db-side optimizations - that saving is something I can 100% attribute to L2S's ability to eliminate predicates that can be evaluated client-side.
simple answer, there are two approaches: create exquisite Rube Goldberg contraptions, or just get the job done in a simple way. Many devs lean towards the former.
Developers get bored easily, and would often personally enjoy doing things a harder way that seems to provide a certain intellectual beauty. Are you developing an app or writing a PhD? As my msft director used to yell in the hallways, "I don't want another research project!"
please repeat after me (min 3x)
there is no silver bullet
I will not use a technology just because its the latest thing from msft
I will not use something just to get it on my resume
Not only are their competent SQL coders, any decent app programmer, especially LOB apps, should write intermediate SQL. If you don't know any SQL and are writing LINQ to SQL, how are you going to debug your data calls? How are you going to profile them to fix bottlenecks?
We're trying out LINQ to SQL and I think there are major issues with it, such as:
There is no simple way to return the query results to another object. This in itself seems insane. Microsoft created the var anonymous datatype, and recommends using it, but there is no way to move this data out of your local method, hence the oo paradigm breaks if you have to use the data in the same function that retrieved it.
Tables are NOT objects. Study up on 3rd normal form etc. Relational databases are for storing data, not using it. I don't want to be restricted or encouraged to use my tables as objects. The data I retrieve from the database will very often be joins of multiple tables, and may include SQL casts, functions, operators, etc.
There is no performance gain, and a slight loss
Now I have way more code to worry about. There are the dbml files and still a DAL to actually write the LINQ. Yes, lots of it is machine-generated, that doesn't mean its not there, its something else that can go wrong (i.e. your dbml files, etc.).
Now that I've given the background, I will attempt to answer you actual question, why do people use LINQ To SQL:
Its the latest thing from Microsoft and I want it on my resume.
Msft has convinced managers/execs that it will decrease coding time
Developers hate SQL. (no good dev environment or debugging except manually--it would be nice to have better intellisense to a sql tool.)
I encourage people not to jump on the bandwagon just because everyone else is, learn enough to put it on your resume, be willing to use it if forced to, but try and really understand the pros and cons first.