I am optimizing Postgres with Ruby on Rails. For last few days I am finding that my site is loading slowly. Application is using different queries with join of 3-4 tables to fetch the data.
Could you help me what I need to do to improve the performance of the application at database level?
Have a look here. You need to capture and look at all the activity and go from there. The link below provides a open source tool to do that.
http://dalibo.github.io/pgbadger/
Related
We are developing an app in dart wherein we need to fetch around more than 50K rows at once (doing it when app loads) and then data will be used in other sections of app for further calculations. We are using Firebase Realtime database and we are facing some serious performance issues.
Its currently taking somewhere around 40 seconds to load 50K rows(currently using free database version, not sure if that would the reason), but we have also observed that when multiple users uses the app, it starts to take around 1 minute 20 sec to load 50K rows and Peak goes to 100%.
Can you please suggest how can we improve performance in firebase realtime database ?
If I break the data in two collection but keep it in same JSON file, would that help ?
Can it be because we are using currently free database version for testing ?
We have tried creating indexes in "Rules" section on 1 Key field but that did not help much. Is there any way we can improve this ?
Can it be because we are using currently free database version for testing?
All Firebase Realtime Database instances run on the same infrastructure. There is no difference based on the plan your project is on.
Can you please suggest how can we improve performance in firebase realtime database?
The best way to improve performance is to only load data that you're going to show to the user right away. In any client-side application it's unlikely that the user will look at 50K items, let alone look at them straight when the application starts.
If you need 50K items to show the initial data to the user, that typically means that you're aggregating that data in some way, and showing that aggregate to the user. Consider doing that aggregation when you write the data to the database, and store the aggregation result in the database. Then you can load just that result in each client, instead of having each client do its own aggregation.
For more data modeling tips, read NoSQL data modeling and watch Firebase for SQL developers. I'd also recommend watching Getting to know Cloud Firestore, which is for Cloud Firestore, but contains many great tips that apply to all NoSQL databases.
I'm interested in the best practice to query the performance state of the mongo cluster (on mongohq) using a ruby script.
I would like to build some ruby script that checks if the mongo is idle (or near idle) and if so, start to do some work (lots of queries and updates) on it.
I suggest instead of writing this yourself to have a look at MMS. MongoHQ supports this for their dedicated database plans. See https://mms.10gen.com/docs/faq for information.
If you really want to do this yourself, you need to call the serverStatus command.
Dru:
Also, there are some additional tools that MongoHQ can make available to you. Please drop the team a note and they can give you some things to try out. But yes, as was stated above, MMS is a good solution as well.
Jason
MongoHQ
I am very interested in using monetdb as a datamart, holding some huge data tables for querying and reporting
However, after some searching, I am unable to find any online posts / blogs regarding their use of Monetdb in any kind of production capacity.
Also, there seems to be little or next to no activity online regarding Monetdb.
Is this a bad sign for the future of Monetdb ?
I am very interested in using monetdb as a datamart, holding some huge data tables for >querying and reporting
My boss is also interested in MonetDB and I had the same reaction as you. No one is writing about MonetDB... is no one using MonetDB?
Regardless, I have been running performance tests on datasets of 500,000 to 1,000,000 records comparing MonetDB (column-oriented dbms) vs. MySQL (row-oriented dbms) and MonetDB beats MySQL in all regards- even in bulk inserts... which hypothetically it should not be as good at.
I can't speculate as to what all this means for MonetDB's future, but while it's around you might want to check it out because it performs well.
(I run Windows 7 and am communicating with each database using PHP)
I react a bit late to this post, but I'd like to add my voice to the ones using MonetDB in a production environment. We use it as the back-end of Spinque, a framework for designing complex search solutions. I've been using MonetDB for about 10 years, but only in the past 3 years in a production environment. Clearly, it has pros and cons and bugs like all other products, but it is being developed and improved very actively (I don't understand the low-activity signs that you refer to). If you want a DB that allows you to be ahead of the market standards, it's a good choice. Otherwise, just go for MS SQL ;)
I've been evaluating it lately for a client so I've had some time with it. My impression at this point is that it is just finishing "growing up" from being an academic experimental playground. It clearly has yet to be really discovered, though it does have some rough edges which might hinder certain applications.
As I write, I'm in the process of trying to load over 100 million rows into an instance (at 27mil presently). So far, it performs startlingly well in some areas (aggregates), but is oddly sluggish in others (most joins I've tried so far); that said, I've not yet run the recommended sampling process yet and I'm forcing it to live in just a single service with 32GB RAM.
I've found a few little glitches and one thing that caused a full service crash (obscure and reported), but I'm thinking that for many applications MonetDB could be just the ticket. Columnar storage (rather than NoSQL) seems to be the future IMO.
I'll update this if I find anything particularly interesting.
MonetDB is first and for all a research system, but has progressed far beyond the level of the average research prototype. It is the (only) relational column-store platform in open source that I know of that supports full SQL. I have used it myself at CWI in many research projects that are not core DB research, but do need advanced DB technology.
You can see on the user's mailing list that deployments happen in many different organisations. As Roberto Cornacchia stated in a different answer, it is the backend of all Spinque deployments and we are happy MonetDB users. MonetDB is also used at a variety of non-profit projects like open streetmap and open kvk.
More and more commercial parties deploy MonetDB for analytics. (They do not always like to advertise that their analyses depend on an open source system.) Recently, MonetDB Solutions has started to provide dedicated commercial support for these deployments.
We have been using MonetDB in our business. We analyse very large data sets with many millions of rows. Traditional methods of data warehousing on SQL databases became so slow. The problem we were facing was that the data was only going to get bigger! The only way forward was to go columnar.
The results have been amazing. When you have very few joins it is staggeringly quick. Even with joins on the data sets we are looking at it is still frightening how fast it comes back.
Having seen some of the commercial partnerships I think MonetDB is going to boom over the next few years. I believe some of the major BI suppliers are using Monet under their hood to perform the large data work.
A question of ColdFusion ORM
We are using ColdFusion 9 for the past 6 months and while we've used some of the new features, ORM is something we've avoided because we usually work on the same very large website. Over the years we've used Apache OBJ but then we moved back to CF and used our own DAO objects generated from tables to handle basic CRUD. These objects are basic and need to be regenerated manually for schema changes and do not model table relationships at all. To supplement these we have a set of gateway classes for multi-table queries. While all this is very bespoke, these DAOs and gateways do, however, give us great control over the SQL we execute, for example using locking hints and optimiser hints. Our site is busy but our database is very efficient.
So much for the history lesson: the point of this is that we have a new site upcoming soon that will be written from scratch to handle financial transactions. We would normally use our aforementioned DAO objects to handle CRUD and then the usual set of gateway objects for multi-table joins but I thought we might investigate using baked-in CF ORM...
So with that in mind, I'd love to hear of the lessons, tips and tricks others might have to share in regard to using ORM on a busy financial site. For example:
What's the best way to flush your SQL ? What tips do you have for transactional ORM? How do you setup development and live sites for ORM? What about HQL? When should we just do the SQL ourselves by hand?!
Thanks in advance!
Short answer, YES! go for ORM!!!
What's the best way to flush your SQL ?
ormflush()
What tips do you have for transactional ORM?
<cftransaction> or transaction {} in CFScript works fine, and even work across DSN's, much improved in 9.0.1! watch http://tv.adobe.com/watch/max-2010-develop/coldfusion-undocumented/
How do you setup development and live sites for ORM?
same as without ORM, but you may look into this.ormSettings.dbCreate if u mean the DB creation part
What about HQL?
What about it? :) Use it when u want an array of objects, but nothing stopping you from using the good old <cfquery>
When should we just do the SQL ourselves by hand?!
When you feel like HQL doesn't do what you want.
Check out: Things to watch out for in ColdFusion 9 with CF-ORM however keep in mind that some points are no longer true for 9.0.1
I've seen a couple of SQLite performance questions here on Stackoverflow, but the focus was on websites, and I'm considering using this DB in a client-server scenario:
I expect 1-10 clients for one server for now, could go up to 50 or more in the future.
slightly more reads than writes
the DB would sit behind a server process (i.e: not using direct DB access through a network)
Would using SQLite make the app less responsive as opposed to using PostgreSQL? My intuition tells me that it should be ok for these loads, but maybe someone has some practical experience with this kind of scenario.
I did use SQLite for a major client/server product used with ~10 concurrent users and I deeply regret that decision. In my opinion - PostgreSQL is much more suitable for client/server scenarios than SQLite due to its fine locking granularity.
You simply can't get very far when the entire database is locked whenever someone needs to write something ..
I like SQLite very much (I even wrote a commercial utility for comparing SQLite databases - SQLite Compare but I don't think it fits the bill when you have client/server scenarios.
Even SQLite's author says that it should be used as a replacement for custom file formats and not as a full blown database server. I wish I took his advice more seriously..
You didn't mention what operating system and Postgres versions you are using. However, before considering change of database engine, try to do some logging and benchmarking your current database with typical usage, then optimize "heaviest" questions. And maybe your backend processing load makes DB question time irrelevant? As SQLite is a file-based DBMS, concurrent access from multiple processes will degrade performance when client number grows up (edited after comment)
Following question may be helpful: How Scalable is SQLite?
I would confirm to S.Lott's answer.
I dont know how SQLite performs in comparison to PostgreSQL, since I don't know any newer meassurements, but my own experience with SQLite in a rather similar environment is rather good.
The only thing that might cause troubles in my view is that you have rather many writes. But it all depends on the total number per second I would say.
Also your setting to have one server process is optimal for SQLite in my opinion -- so you circumvent its weakness in multi-tasking.