I'm developing a large scale website and when I was checking linq queries with SQL Profiler I found that there is about 30 login/logout actions. I want to decrease these actions by using an open connection for all DataContexts but I don't know how to do it. Do you have any suggestion?
That is pretty bad idea - you are creating large scalable website so let Linq-to-sql handle connection itself.
Linq-to-sql internally handles connection opening and releasing in effective way for it usage. It uses default ADO.NET connection pooling so the connection is correctly reused and not opened for every single context.
Using single connection for all context is exactly what makes your application non scalable and not working. Single connection allows only single transaction so once two requests want to make concurrent changes and use their own transaction your application will crash.
Do not share contexts and do not share connections - let ADO.NET handle connection pooling and create new context for every single request or you can expect serious issues.
Related
need some software architecture insight on this. Which of the following is more efficient in terms of resource (cpu, memory, database)?
Having a single database connection in one flow? (Close connection only after everything is done, including business logic)
Having multiple database connections in one flow? (Open then close the database connection immediately after the query is executed)
By business logic, this is where data returned from the query is sanitized, or manipulated according to business rules.
Attaching here is the diagram for visual representation.
UPDATE:
Programming language: PHP (Laravel for web app, Lumen for API)
Database: MySQL
Host: AWS
opening new connection between your runtime and your database needs your OS to create new socket ( if runtime and database are on the same system this socket probably is linux socket, else this socket is tcp/udp socket)
this socket creation has overhead itself.
so I don't suggest to open and close connections after each database usage.
but there are specific conditions you want do that.
for example your database has a limited number of concurrent connections and you have thousands of long running processes using this connections, maybe in this situation you can use second approach.
In my project, developers use a single instance of Connection instead of a connection pool on an Oracle 12c.
Using a pool is a common practice and Oracle itself documents it: http://docs.oracle.com/database/121/JJUCP/get_started.htm#JJUCP8120.
But JDBC 4.2 specification says:
13.1.1 Creating Statements
Each Connection object can create multiple Statement objects that may be used concurrently by the program.
Why using a pool of connections instead of a single connection, if it's possible to use statements to manage concurrency?
The Oracle Database Dev Team strongly discourages using a single Connection in multiple threads. That almost always causes problems. As a general rule we will not consider any problem report that does this.
A Connection can have multiple Statements and/or ResultSets open at one time but only one can execute at a time. Connections are strictly single threaded and blocking. We try to prevent multiple threads from accessing a Connection simultaneously but there are a few odd cases where it is possible. These are all but guaranteed to cause problems. (It is not practical to fix or prevent these cases mostly for performance reasons. Just don't share a single Connection across multiple threads.)
If a client connects to the database via a dedicated server connection then that database session will only serve that client . If the client connects to the database via shared server connection, then a given database session may serve multiple clients over its lifetime.
This is documented here.
Also, at any one point in time, a session can only execute one thing at a time. If that wasn't the case, then running things in parallel wouldn't spawn multiple other sessions!
A single connection cannot execute several statements concurrently.
Yes one connection can execute more that one statement. It will be the programmer to chose connection pooling setting or multiple statements when executing over more than one thread. Most databases in the market can handle multiple statements in one connection.
We're in the process of rewriting a web application in Java, coming from PHP. I think, but I'm not really sure, that we might run into problems in regard to connection pooling. The application in itself is multitenant, and is a combination of "Separate database" and "Separate schema".
For every Postgres database server instance, there can be more than 1 database (named schemax_XXX) holding more than 1 schema (where the schema is a tenant). On signup, one of two things can happen:
A new tenant schema is created in the highest numbered schema_XXX database.
The signup process sees that a database has been fully allocated and creates a new schemas_XXX+1 database. In this new database, the tenant schema is created.
All tenants are known via a central registry (also a Postgres database). When a session is established the registry will resolve the host, database and schema of the tenant and a database session is established for that HTTP request.
Now, the problem I think I'm seeing here is twofold:
A JDBC connection pool is defined when the application starts. With that I mean that all databases (host+database) are known at startup. This conflicts with the signup process.
When I'm writing this we have ~20 database servers with ~1000 databases (for a total sum of ~100k (tenant) schemas. Given those numbers, I would need 20*1000 data sources for every instance of the application. I'm assuming that all pools are also, at one time or another, also started. I'm not sure how much resources a pool allocates, but it must be a non trivial amount for 20k pools.
So, is it feasable to even assume that a connection pool can be used for this?
For the first problem, I guess that a pool with support for JMX can be used, and that we create a new datasource when and if a new schemas_XXX database is created. The larger issue is that of the huge amount of pools. For this, I guess, some sort of pool manager should be used that can terminate a pool that have no open connections (and on demand also start a pool). I have not found anything that supports this.
What options do I have? Or should I just bite the bullet and fall back to an out of process connection pool such as PgBouncer and establish a plain JDBC connection per request, similar to how we're handling it now with PHP?
A few things:
A Connection pool need not be instantiated only at application start-up. You can create or destroy them whenever you want;
You obviously don't want to eagerly create one Connection pool per database or schema to be open at all times. You'd need to keep at least 20000 or 100000 Connections open if you did, a nonstarter even before you get to the non-Connection resources used by the DataSource;
If, as is likely, requests for Connections for a particular tenant tend to cluster, you might consider lazily, dynamically instantiating pools, and destroying them after some timeout if they've not handled a request for a while.
Good luck!
My Java application is processing a lot of information parsed from XML. For some methods, I need to validate some info in an SQL Database on another machine. I am using JDBC, currently for each validation, I call a DB Handling method that opens a connection and returns the result set to validate. I am not sure if I am taking the best design option. This seems very dummy and expensive to me. I am wondering if there are better practices.
Is there is a proper way to have the connection open through the whole application run time so no time is wasted (opening connection) for every validation iteration (I might have hundreds of thousands).
Should I build another application that retrieves all needed tables and convert them to XMLs that are saved on my machine. Later, my application parses them normally and have better access and performance
I am open to any better suggestion
Of course you can use one db connection for whole life of application. Use Singleton pattern. If your program work long and do not use database for a longer time it may loose db connection (some kind of timeout on network equipment etc.). For such cases you can use db pool. Such pool should manage longer time of inactivity or give you separate db connections for separate threads.
I think your solution with converting data into XML is not good. Why do you want to convert it into XML? How often is this data changed? How big is the database you want to copy? How do you want to synchronize your local copy with database? I think that local copy in XML files adds too many problems. If data is small than maybe you can read it at start of your program, save it in some data structures and use to verify other data? It can even be SQLite or other small database. But I would go this way only if singleton or db pool performance is really weak.
My application have performance issues, so i started to investigate this from the root: "The connection with the database".
The best practices says: "Open a connection, use it and close is as soon as possible", but i dont know the overhead that this causes, so the question is:
1 -"Open, Use, Close connections as soon as possible is the best aproach using ODP.NET?"
2 - Is there a way and how to use connection pooling with ODP.NET?
I thinking about to create a List to store some connections strings and create a logic to choose the "best" connection every time i need. Is this the best way to do it?
Here is a slide deck containing Oracle's recommended best practices:
http://www.oracle.com/technetwork/topics/dotnet/ow2011-bp-performance-deploy-dotnet-518050.pdf
You automatically get a connection pool when you create an OracleConnection. For most middle tier applications you will want to take advantage of that. You will also want to tune your pool for a realistic workload by turning on Performance Counters in the registry.
Please see the ODP.NET online help for details on connection pooling. Pool settings are added to the connection string.
Another issue people run into a lot with OracleConnections is that the garbage collector does not realize how truly resource intensive they are and does not clean them up promptly. This is compounded by the fact that ODP.NET is not fully managed and so some resources are hidden from the garbage collector. Hence the best practice is to Close() AND Dispose() all Oracle ODP.NET objects (including OracleConnection) to force them to be cleaned up.
This particular issue will be mitigated in Oracle's fully managed provider (a beta will be out shortly)
(EDIT: ODP.NET, Managed Driver is now available.)
Christian Shay
Oracle
The ODP.NET is a data provider for ADO.NET.
The best practice for ADO.Net is Open, Get Data (to memory), close, use in memory data.
For example using a OracleDataReader to load data in a DataTable in memory and close connection.
[]'s
For a single transaction this is best but for multiple transaction where you commit at the end this might not be the best solution. You need to keep the connection open until the transaction either committed or rolled back. How do you manage that and also how do you check the connection still exist in that case?(ie network failure) There is ConnectionState.Broken property which does not work at this point.