When we execute SQL queries using PreparedStatement (as described here http://tutorials.jenkov.com/jdbc/preparedstatement.html), what exactly does the executeQuery() method do, if we use for example a database like SQL Server or Postgres? Does it convert the SQL query directly into a set of database operations, or does it make a network call to a database server that translates the SQL query to the database operations?
This is more generally a question about how databases like SQL Server work. I'm just wondering if they're running on separate servers than the ones calling executeQuery().
The implementations details vary per database system, but in general JDBC drivers for RDBMSes that uses SQL as their native query language will work as follows:
When Connection.prepareStatement(...) is executed, the query is sent to the database server for compilation
On PreparedStatement.executeQuery(), the driver sends an execute command together with the collected parameter values, and the database server executes the statement compiled earlier using those parameter values.
In other words, the driver is not concerned with low-level operations on the database server, but will just send a 'compile' and 'execute-with-parameters' command to the database server, and the database server takes care of the low-level operations.
And to be clear, not all drivers work this way. For example, the MySQL Connector/J by default 'compiles' the statement locally (determines the number of parameters), and on execute it will inline the parameter values (with proper escaping) into the statement, and then send a SQL string with literal values instead of parameters for execution on the database server. However, in that case, the database server is still responsible for determining and performing the necessary low-level operations.
On the other hand, it is entirely possible that a driver for some NoSQL database (or other type of datastore) will translate a SQL query into low-level operations on that datastore.
Related
Is there any way I can execute more than 1 query using a NamedParameterJdbcTemplate in one single call?
The simple answer: You can't. JDBC statements should execute only one query / statement at a time.
The longer answer: You usually can't, because either the driver or the database server doesn't support it, and formally the JDBC specification doesn't allow for it (if you read a bit between the lines in both the specification and the Javadoc). However, some driver (ie MySQL Connector/J) do allow you to execute multiple queries in one go (though you might need to specify a connection property to enable it, not sure).
I need to compare data from two databases (both of them are DB2) which are on different servers with no existing connection between them. Because both of the db's are used in production I don't want to overload them, therefore I will create a new db (probably MySQL) on my local machine, extract data from both of the DB2's, insert into MySQL and do the comparison locally.
I would like to do this in Java, so my question is how to do this task as effectively as possible, without overloading the production databases. I've done some research and came up with the following points:
limit the number of columns that I will use in my initial SELECT statement
tune the fetch size of the ResultSet object (the default for IBM DB2 JCC drivers seems to be 64)
use PreparedStatement object to pre-compile the SQL
Is there anything else that I can do, or any other suggestions?
Thank you
DB2 for Linux UNIX, and Windows includes the EXPORT utility as part of its runtime client. This utility can be pointed at a DB2 database on z/OS to quickly drain a table (or query result set) into a flatfile on your client machine. You can choose whether the flatfile is delimited, fixed width, or DB2's proprietary IXF format. Your z/OS DBA should be able to help you configure the client on your workstation and bind the necessary packages into the z/OS databases as required by the EXPORT utility.
Once you have the flatfiles on your client, you can compare them however you like.
Sounds like a great job for map reduce (hadoop). One job with two mappers, one for each DB and a reducer to do the compares. It can scale to as many processors as you need, or just run on a single machine.
I have an Oracle 9 database from which my Delphi 2006 application reads data into a TSimpleDataSet using a SQL statement like this one (in reality it is more complex, of course):
select * from myschema.mytable where ID in (1, 2, 4)
My applications starts up and executes this query quite often during the course of the day, each time with different values in the in clause.
My DBAs have notified me that this is creating execessive load on the database server, as the query is re-parsed on every run. They suggested to use bind variables instead of building the SQL statement on the client.
I am familiar with using parameterized queries in Delphi, but from the article linked to above I get the feeling that is not exactly what bind variables are. Also, I would need theses prepared statements to work across different runs of the application.
Is there a way to prepare a statement containing an in clause once in the database and then have it executed with different parameters passed in from a TSimpleDataSet so it won't need to be reparsed every time my application is run?
My answer is not directly related to Delphi, but this problem in general. Your problem is that of the variable-sized in-list. Tom Kyte of Oracle has some recommendations which you can use. Essentially, you are creating too many unique queries, causing the database to do a bunch of hard-parsing. This will spike the CPU consumption (and DBA blood pressures) unnecessarily.
By making your query static, it can get by with a soft-parse or perhaps no parse at all! The DB can then cache the execution plan, the DBAs can deal with a more "stable" SQL, and overall performance should be improved.
In the Oracle JDBC driver, there is an option to cache prepared statements. My understanding of this is that the prepared statements are precompiled by the driver, then cached, which improves performance for cached prepared statements.
My question is, does this mean that the database never has to compile those prepared statements? Does the JDBC driver send some precompiled representation, or is there still some kind of parsing/compilation that happens in the database itself?
When you use the implicit statement cache (or the Oracle Extension for the explicit Statement Cache) the Oracle Driver will cache a prepared- or callable statement after(!) the close() for re-use with the physical connection.
So what happens is: if a prepared Statement is used, and the physical connection has never seen it, it sends the SQL to the DB. Depending if the DB has seen the statement before or not, it will do a hard parse or a soft parse. So typically if you have a 10 connection pool, you will see 10 parses, one of it beein a hard parse.
After the statement is closed on a connection the Oracle driver will put the handle to the parsed statement (shared cursor) into a LRU cache. The next time you use prepareStatement on that connection it finds this cached handle to use and does not need to send the SQL at all. This results in a execution with NO PARSE.
If you have more (different) prepared statements used on a physical connection than the cache is in size the longest unused open shared cursor is closed. Which results in another soft parse the next time the statement is used again - because SQL needs to be sent to the server again.
This is basically the same function as some data sources for middleware have implemented more generically (for example prepared-statement-cache in JBoss). Use only one of both to avoid double caching.
You can find the details here:
http://docs.oracle.com/cd/E11882_01/java.112/e16548/stmtcach.htm#g1079466
Also check out the Oracle Unified Connection Pool (UCP) which supports this and interacts with FAN.
I think that this answers your question: (sorry it is powerpoint but it defines how the prepared statement is sent to Oracle, how Oracle stores it in the Shared SQL pool, processes it, etc). The main performance gain you are getting from Prepared statements is that on the 1+nth run you are avoiding hard parses of the sql statement.
http://www.google.com/url?sa=t&source=web&cd=2&ved=0CBoQFjAB&url=http%3A%2F%2Fchrisgatesconsulting.com%2FpreparedStatements.ppt&rct=j&q=java%20oracle%20sql%20prepared%20statements&ei=z0iaTJ3tJs2InQeClPwf&usg=AFQjCNG9Icy6hmlFUWHj2ruUsux7mM4Nag&cad=rja
Oracle (or db of choice) will store the prepared statement, java just send's it the same statement that the db will choose from (this is limited resources however, after x time of no query the shared sql will be purged esp. of non-common queries) and then a re-parse will be required -- whether or not it is cached in your java application.
How oracle CACHE a Query (SQL), Query execution contains the following steps.
1. PARSE
2. Execute
3. Fetch
in First step oracle checks if query exists in CACHE (Shared Pool) or not (It will be exists if query is identical and based on LRU), if exists then PARSING will be skipped and execution will start.
So to make query performance intensive we must use bind variables and use the identical sql queries.
But in Parsing oracle verifies authentication also (User Access), if multiple users are using the same query, how oracle skip/use parsing?
The parsing of a query is not tied to a user, it is dependent on the query. Note that an exact character for character match is required. Blanks and comments in a query will cause it to miss the Shared Pool match.
The parse-tree is then used in the generation of an execution plan. If the same schema is used in the new query as the matched query then the existing execution plan is used.
You can test this by creating multiple schema, one with a small amount of data and one with a large
amount. Then analyze all the tables. Take a look at the execution plans for the same query with vastly different amounts of data. This will show the different execution plans for the same query.
Now run the query a large amount of times and check the amount of time that it takes for the first
and then subsequent executions. Use Oracle Trace and look in the left hand pain for the "Re-Parse"
frequency. This can also be gleaned from some of the dictionary tables.
Take a look at The Oracle documentation on using Oracle Trace
First step oracle checks if query exists in CACHE (Shared Pool) or not (It will be exists if query is identical and based on LRU), if exists then PARSING will be skipped and execution will start. So to make query performance intensive we must use bind variables and use the identical sql queries.
This is actual process when you execute a query on Oracle:
Parsing Steps
Syntax Check
Semantic Analysis
Has the query been executed in some other session?
Hard Parse
Parse
Optimize
Generate the plan for the query.
If the answer to #1.3 is yes - Oracle skips the hard parse portion, and uses the existing query plan.
For more info:
* AskTom: Difference between soft parse and hard parse
* Bind variables - The key to application performance
Usual practice in Oracle is to create stored procedures with definer rights which means that the queries are executed with privileges of their definer, despite of who calls them. That's why cache works well.
If you create a procedure or package with invoker rights (authid current_user), then the queries will be parsed for each invoker separately.
See Invoker Rights Versus Definer Rights for details.