We have an application running that makes and intensive usage of EF version 4.1. Right now an update is not scheduled, but this is not the point.
We are going under a process of huge optimization as the workload over the Sql Server database is very high. I successfully succeeded (after never succeeding) in profiling the sent queries (we are under Sql Azure) and I discovered that a lot of awful queries were sent, because of stupid usage of Includes. We removed all the avoidable "Includes", by substituing them with direct queries on the sets (i.e. 10 queries against 1 before) and the total workload on Sql Server has dramatically reduced.
But then I discovered that the global code is running slower. How may this be possible? I started tracing and I discovered that even if a query was processed in less that 50ms, EF takes more than 1 second to materialize the entities. That's awful. It's 2 order degrees more than a normal sql query through ADO.Net.
I tried by disabling the Entity Tracking on the set and querying the set with AsNoTracking() but nothing seems changed.
I never had the possibility to analyze the total split time between query processing and object materialization, but I never ever thought that that was the impact. Is there any more that I can try?
--
Update 2015/06/15 - Performance Analysis
I ran 7 different set of tests, and each test has been run 10 times. I tracked the total time to execute the .NET code using the old version (1 huge query with a lot of Includes) and the new version (1 small query for each old Include).
I would say that the problem is, as Jeroen Vannevel said in the comment, the latency of the multiple request / response.
Here are the summed results. Please consider that when I write "item" I mean a complex entity, composed by at least 10 different entities taken from 10 different DB tables.
GET of 1 item from the DB
OLD method: 3.5 seconds average
NEW method: 276 milliseconds average
GET of 10 items
OLD method: 3.6 seconds average
NEW method: 340 milliseconds average (+23%)
GET of 50 items
OLD method: 4.0 seconds average
NEW method: 440 milliseconds average (+30%)
GET of 100 items
OLD method: 4.5 seconds average
NEW method: 595 milliseconds average (+35%)
GET of 250 items
OLD method: 5.7 seconds average
NEW method: 948 milliseconds average (+60%)
GET of 500 items
OLD method: 5.9 seconds average
NEW method: 1.8 seconds average (almost +100%)
GET of 1000 items
OLD method: 6.9 seconds average
NEW method: 5.1 seconds average (over +180%)
So it seems that, while the OLD method is very slow from the very first execution, because of the heavy query, it increments "slowly" as I get more and more items from the DB.
I would say that as soon as the packet grows up, the time to transfer it to the calling machine is higher than DB time.
May this really be the reason?
Thanks
Marco
Related
I'm running a job using Spring Batch 4.2.0 with postgres (11.2) as backend. It's all wrapped in a spring boot app. I've 5 steps and each runs using a simple partitioning strategy to divide data by id ranges and reads data into each partition (which are processed by separate threads). I've about 18M rows in the table, each step reads, changes few fields and writes back. Each step reads all 18M rows and writes back. The issue I'm facing is, the queries that run to pull data into each thread scans data by id range like,
select field_1, field_2, field_66 from table where id >= 1 and id < 10000.
In this case each thread processes 10_000 rows at a time. When there's no traffic the query takes less than a second to read all 10,000 rows. But when the job runs there's about 70 threads reading all that data in. It goes progressively slower to almost a minute and a half, any ideas where to start troubleshooting this?
I do see autovacuum running in the backgroun for almost the whole duration of job. It definitely has enough memory to hold all that data in memory (about 6GB max heap). Postgres has sufficient shared_buffers 2GB, max_wal_size 2GB but not sure if that in itself is sufficient. Another thing I see is loads of COMMIT queries hanging around when checking through pg_stat_activity. Usually as much as number of partitions. So, instead of 70 connections being used by 70 partitions there are 140 conections used up with 70 of them running COMMIT. As time progresses these COMMITs get progressively slower too.
You are probably hitting https://github.com/spring-projects/spring-batch/issues/3634.
This issue has been fixed and will be part of version 4.2.3 planned to be released this week.
I have a Postgres 9.6 installation and I am running into this weird case where - if I run a same query having multiple joins after 10 to 15 mins, there is increase in the value of query cost in the order of few hundreds and its keep on increasing.
I do understand what vacuuming and analyse does, but I am worried about the query cost which starts increases within few minutes of performing vacuum and analyse. I am afraid this might lead do future performance bottlenecks.
PS: I have two table out of which one is heavily written (about 5 million records ) and other is heavily updated (70 K records with postGIS this table mostly have updates on lat lon & geom column)
Does this means I should have auto vacuum run every few hours?
Make Autovaccum aggressive; but if you think autovaccum is using up resources(by looking at cpu ]and IO usage) you could tweak-- autovacuum_vacuum_cost_delay and autovacuum_vacuum_threshold paramters at table level
I have a program that make many queries to Google Search Analytics server. My program does the queries one after the other sequentially, so each instant, only one query will be in process.
Google has advised about a throughput limit of 2000 queries per each 100 seconds at most so to configure my system to be the more efficient it could be I have two ideas on mind:
Known that 2000 queries per 100 seconds is one query per each 0.05 seconds, i have separated my queries by sleeping the process, but only if any query take less than 0.05 seconds, so the time the process will sleep in that case is the remaining time to complete the 0.05 second interval. If the query takes 0.05s or more I trigger the following without waiting.
The second idea is more easy to implement but I think it will be less efficient: i will trigger the queries taking note of the time when the process start so if i reach 2000 queries before 100 seconds, I will wait the remaining time sleeping.
So far I don't know how to measure which one is the best.
Which is your opinion about the two options? Any of them is better and why? Any additional option I haven't figured out? (specially if it's better than mine)
Actually what you need to consider is that its 2000 requests per 100 seconds. But you could do all 2000 requests in 10 seconds and still be on the good side of the quota.
I am curious as to why you are worried about it though. If you get one of the following errors
403 userRateLimitExceeded
403 rateLimitExceeded
429 RESOURCE_EXHAUSTED
Google just recommends that you implement Exponential backoff which consists of making your request getting the error sleeping for a bit and trying again. (do this up to eight times). Google will not penalize you for getting these errors they just ask that you wait a bit before trying again.
If you want to go crazy you can do something like what i did in my C# application I created a request queue that i use to track how much time has gone since i created the last 100 request. I call it Google APIs Flood Buster.
Basically i have a queue where i log each requests as i make it before i make a new request i check how long it has gone since i started. Yes this requires moving the items around the queue a bit. If there has gone more then 90 seconds then i sleep (100 - time since ) this has reduced my errors a great deal. Its not perfect but that's because google is not perfect with regard to tracking your quota. they are normally off by a little.
I'm querying my database whit ES 2.3.1 and I've been measuring the times responses, but I got three different times.
First I measured the time of the first query on database. It takes about 9 seconds.
Second time I measured, I closed the ES, cleared the RAM and cache and query again. It takes about 1,2 seconds.
The third time I queried without cleaning caches and it takes 97 ms.
Can anyone explain way it happens?
The last measure I know that its faster because the data already queried is on cache. I think the first time takes more time because the data have to be pulled on cache.
For me, when I clear the cache and RAM, the time of the second measuring had to be equal the first measure, but no. Can someone explain me way?
We have a business requirement to show a cost summary for all our projects in a single table.
In order to tabulate these costs we have to query through all the client tasks, regions, job roles, pay rates, cost tables, deliverables, efforts, and hour records (client tasks are in the same table and tasks and regions are in the same table and deliverables, effort, and hours are stored as monthly totals).
Since I have to query all of this before I go for-looping through everything it hits a large number of scripting lines very quickly. Computationally it's like O(m * n * o * p) and some of our projects have all four variables that go up very quickly. My estimates for how to do this have ranged from 90 million lines of code to 600 billion.
Using batch apex we could break this up by task regions into 200 batches, but that would reduce the computational profile to (600 B / 200 ) = 3 billion lines of code (well above the salesforce limit.
We have been playing around with using informatica to do these massive calculations, but we have several problems including (1) our end users can not wait more than five or so minutes, but just transferring the data (90% of all records if all the projects got updated at once) would take 15 minutes over informatica or the web api (2) we have noticed these massive calculations need to happen in several places (changing a deliverable forecast value, creating an initial forecast, etc).
Is there a governor limit work around that will meet our requirements here (massive volume of data with response in 5 or so minutes? Is force.com a good platform for us to use here?
This is the way I've been doing it for a similar calculation:
An ERD would help, but have you considered doing this in smaller pieces and with reports in salesforce instead of custom code?
By smaller pieces I mean, use roll-up summary fields to get some totals higher in your tree of objects.
Or use apex triggers so as hours are entered the cost * hours is calculated and placed onto the time record, and then rolled-up to the deliverables.
Basically get your values calculated at the time the data is entered instead of having to run your calculations every time.
Then you can simply run a report that says show me all my projects and their total cost or total time because those total costs/times are stored/calculated already.
Roll-up summaries only work with master-detail
Triggers work anytime, but you'll want to account for insert, update as well as delete and undelete! Aggregate Functions will be your friend assuming that the trigger context has fewer than 50,000 records to aggregate - which I'd hope it does b/c if there are more than 50,000 time entries for a single deliverable that's a BIG deliverable :)
Hope that helps a bit?