I have a legacy vb6 monolithic application which reads/writes/deletes records from Access database using ADO library functions. There are many connections which are made in multiple functions to a single database:
Despite the fact that the data in the DB is not too large, the Access MDB file grows rapidly leading to hit the 2 GB mark. The only way out for me then is to stop the application, compact and repair the DB and then restart the app. Does creating too many connections from the application cause this issue? What is the way to resolve this?
As per #plutonix, data manipulation operations expand the DB. What do you call "not too large". An MDB >100MB is very large!
It sounds like it's time to scale out your application to use Linked Tables and store the data in an SQL Server instance (Express is free). This leaves only your application logic in MS Access. Doing this requires minimal effort and you get to keep almost all of your code without touching it.
https://support.office.com/en-sg/article/Import-or-link-to-SQL-Server-data-a5a3b4eb-57b9-45a0-b732-77bc6089b84e
Related
need some software architecture insight on this. Which of the following is more efficient in terms of resource (cpu, memory, database)?
Having a single database connection in one flow? (Close connection only after everything is done, including business logic)
Having multiple database connections in one flow? (Open then close the database connection immediately after the query is executed)
By business logic, this is where data returned from the query is sanitized, or manipulated according to business rules.
Attaching here is the diagram for visual representation.
UPDATE:
Programming language: PHP (Laravel for web app, Lumen for API)
Database: MySQL
Host: AWS
opening new connection between your runtime and your database needs your OS to create new socket ( if runtime and database are on the same system this socket probably is linux socket, else this socket is tcp/udp socket)
this socket creation has overhead itself.
so I don't suggest to open and close connections after each database usage.
but there are specific conditions you want do that.
for example your database has a limited number of concurrent connections and you have thousands of long running processes using this connections, maybe in this situation you can use second approach.
Is there any problem on using many open connections at the same time from different threads?
From what I've read it's thread safe by default, but, can this be hurting performance rather than improving it?
Having multiple connection is not a problem, the only thing to keep in mind is that SQLite does not support concurrency of multiple write transactions. From the SQlite site:
SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, this is not a problem. Writer queue up. Each application does its database work quickly and moves on, and no lock lasts for more than a few dozen milliseconds. But there are some applications that require more concurrency, and those applications may need to seek a different solution.
SQLite is an "untypical" database management system: in practice it is a library that offers SQL as language to access a simple "database-in-a-file", and a few other functionalities of DBMSs. For instance, it has no real concurrency control (it uses the Operating Systems functions to lock the db file).
So, if you need concurrent insertions into a database, you should use something else, for instance PostgreSQL.
The documentation say:
A connection can only be used from within the thread that created it.
Moving connections between threads or creating queries from a
different thread is not supported.
In addition, the third party libraries used by the QSqlDrivers can
impose further restrictions on using the SQL Module in a multithreaded
program. Consult the manual of your database client for more
information.
It is mean you have to create connection to database which will be linking with parent thread. At docs of QSqlDatabase class you can see description:
The QSqlDatabase class represents a connection to a database.
The QSqlDatabase class provides an interface for accessing a database
through a connection. An instance of QSqlDatabase represents the
connection. The connection provides access to the database via one of
the supported database drivers, which are derived from QSqlDriver.
Create a connection (i.e., an instance of QSqlDatabase) by calling one
of the static addDatabase() functions, where you specify the driver or
type of driver to use (i.e., what kind of database will you access?)
and a connection name.
Using static addDatabase() function is way to create connection.
But as Renzo said SQLite does not support multiple write transactions at the same time. So you need some mechanisms(wrapper) for synchronizing threads like task queue using low-level mutex or something like that. More information you can see at docs.
My Java application is processing a lot of information parsed from XML. For some methods, I need to validate some info in an SQL Database on another machine. I am using JDBC, currently for each validation, I call a DB Handling method that opens a connection and returns the result set to validate. I am not sure if I am taking the best design option. This seems very dummy and expensive to me. I am wondering if there are better practices.
Is there is a proper way to have the connection open through the whole application run time so no time is wasted (opening connection) for every validation iteration (I might have hundreds of thousands).
Should I build another application that retrieves all needed tables and convert them to XMLs that are saved on my machine. Later, my application parses them normally and have better access and performance
I am open to any better suggestion
Of course you can use one db connection for whole life of application. Use Singleton pattern. If your program work long and do not use database for a longer time it may loose db connection (some kind of timeout on network equipment etc.). For such cases you can use db pool. Such pool should manage longer time of inactivity or give you separate db connections for separate threads.
I think your solution with converting data into XML is not good. Why do you want to convert it into XML? How often is this data changed? How big is the database you want to copy? How do you want to synchronize your local copy with database? I think that local copy in XML files adds too many problems. If data is small than maybe you can read it at start of your program, save it in some data structures and use to verify other data? It can even be SQLite or other small database. But I would go this way only if singleton or db pool performance is really weak.
I'm developing a large scale website and when I was checking linq queries with SQL Profiler I found that there is about 30 login/logout actions. I want to decrease these actions by using an open connection for all DataContexts but I don't know how to do it. Do you have any suggestion?
That is pretty bad idea - you are creating large scalable website so let Linq-to-sql handle connection itself.
Linq-to-sql internally handles connection opening and releasing in effective way for it usage. It uses default ADO.NET connection pooling so the connection is correctly reused and not opened for every single context.
Using single connection for all context is exactly what makes your application non scalable and not working. Single connection allows only single transaction so once two requests want to make concurrent changes and use their own transaction your application will crash.
Do not share contexts and do not share connections - let ADO.NET handle connection pooling and create new context for every single request or you can expect serious issues.
We have a JPA -> Hibernate -> Oracle setup, where we are only able to crank up to 22 transactions per seconds (two reads and one write per transaction). The CPU and disk and network are not bottlenecking.
Is there something I am missing? I wonder if there could be some sort of oracle imposed limit that the DBA's have applied?
Network is not the problem, as when I do raw reads on the table, i can do 2000 reads per second. The problem is clearly writes.
CPU is not the problem on the app server, the CPU is basically idling.
Disk is not the problem on the app server, the data is completely loaded into memory before the processing starts
Might be worth comparing performance with a different client technology (or even just a simple test using SQL*Plus) to see if you can beat this performance anyway - it may simply be an under-resourced or misconfigured database.
I'd also compare the results for SQLPlus running directly on the d/b server, to it running locally on whatever machine your Java code is running on (where it is communicating over SQLNet). This would confirm if the problem is below your Java tier.
To be honest there are so many layers between your JPA code and the database itself, diagnosing the cause is going to be fun . . . I recall one mysterious d/b performance problem resolved itself as a misconfigured network card - the DBAs were rightly insistent that the database wasn't showing any bottlenecks.
It sounds like the application is doing a transaction in a bit less than 0.05 seconds. If the SELECT and UPDATE statements are extracted from the app and run them by themselves, using SQL*Plus or some other tool, how long do they take, and if you add up the times for the statements do they come pretty near to 0.05? Where does the data come from that is used in the queries, and which eventually gets used in the UPDATE? It's entirely possible that the slowdown is not the database but somewhere else in the app, such a the data acquisition phase. Perhaps something like a profiler could be used to find out where the app is spending its time.
Share and enjoy.