According to this question (and the link posted in the comments, DataGrip should show query execution time in the status bar as of version 2017.1. However, this only works for me with some queries.
E.g., on a short query, I see the following, as expected
But on some other queries, I see something like this instead
I can't really figure out when the second one happens and when it doesn't. The best guess I have is that it's only for long-running queries. But this is exactly when I would want it to show the execution time, so that would be very frustrating.
Queries took longer than 20 secs give more information in output and status bar gives only the last row of it.
Anyway, looks bad, so there's a ticket:
https://youtrack.jetbrains.com/issue/DBE-7164
Related
I'm open to posting the code in this section to work through the optimization but its a bit length and complex, so instead I'm hoping that somebody can assist me with a few debugging questions I have. My goal is to find out what is causing my Apex CPU Time Limit Exceeded issue.
When using the Debug Log in its basic or normal layout I receive the message
Maximum CPU Time: 15062 out of 10,000 ** Close to Limit
I've optimized and re-wrote various loops and queries several times now and in each case this number concludes around there which leads me to believe it is lying to me and that my actual usage far exceeds that number. So on my journey I switched the Log Panels of the Developer Console to Analysis in hopes of isolating exactly what loop, method, or area of the code is giving me a headache.
This leads me to my main question and problem.
Execution Tree, Performance Tree & Executed Units
All show me that my durations UNDER the 10,000ms allowance. My largest consumption is 3,556.19ms which is being used by a wrapper class I created and consumed in the constructor method where there is a fair amount of logic that is constructing a fairly complicated wrapper class that spans over 5-7 custom objects. Still even with those 3,000ms the remainder of the process shows at negligible times bringing my total around 4,000ms. Again my question is.... Why am I unable to see or find what is consuming all my time?
Incorrect Iteration Data
In addition to this, on the Performance tree there is a column of data that shows the number of iterations for each method. I know that my Production Org has 81 objects that would essentially call the constructor for my custom wrapper object. I.E. my Constructor SHOULD be called 81 times, but instead it is called 32 times. So my other question is can I rely on the iteration data in the column? Or because it was iterating so many times does it stop counting at a certain point? Its possible that one of my objects is corrupted or causing an infinite loop somehow, but I don't want to dig through all the data in search of that conclusion if its a known issue that the iteration data is not accurate anyway.
System.Debug in the Production org
The Last question is why my System.Debug() lines are not displaying in my Developer Console on the production org. I've added serveral breadcrumbs throughout the code that would help me isolate just which objects are making it through and which are not, however, I cannot in any layout view system.debug messages outside of my Sandbox.
Sorry for the wealth of questions but I did want to give an honest effort to better understand the debugging process in Salesforce. If this is a lost cause I'm happy to start sharing some code as well but hopefully some debugging tips can get me to the solution.
It's likely your debug log got truncated, see "Each debug log must be 20 MB or smaller. If it exceeds this amount, you won’t see everything you need." in https://trailhead.salesforce.com/en/content/learn/modules/apex_basics_dotnet/debugging_diagnostics
Download the log and search for text similar to "skipped 123456 bytes of detailed log" to confirm, some system.debug statements will just not show up.
You might have to fine-tune the log levels (don't log validation rules and workflows? don't log every single variable assignment with "FINE" level etc). You might have to set all flags to NONE, then track only 1 particular class/trigger that you suspect (see https://help.salesforce.com/articleView?id=code_debug_log_classes.htm&type=5 and https://salesforce.stackexchange.com/questions/214380/how-are-we-supposed-to-use-debug-logs-for-a-specific-apex-class-only)
If it's truncated it's possible analysis tools give up (I had mixed luck with console to be honest, sometimes https://apextimeline.herokuapp.com/ is great to give overview - but it'll also fail to parse a 20 MB log...
When all else fails you can load up the log into Notepad++ (or any editor of your choice), find lines related to method entry/method exit (you might need a regular expression search), take these filtered lines tor excel, play with "text to columns" and just look at timing manually, see if there's a record that causes the spike. Because it could be #10 that's the problem, the fact it exhausts limits on #32 of 81 doesn't mean much. Search like [METHOD_ENTRY|METHOD_EXIT]MyTriggerHandler.onBeforeUpdate could be a good start. But first thing is to make sure log is not truncated.
When trying to run a fairly basic query using the driver provided by Simba, I'm running into issues where the "nanosecond" value is negative, causing IllegalArgumentException.
When writing a simple query that returns a Timestamp value, what comes back is an epoch value that is initially stored in a Double. Going through and debugging for example, I can see that the value coming back from the query is "1.498571169792E9". This corresponds to a timestamp of "Tuesday, June 27, 2017 1:46:09.792 PM" according to epochconverter.com, which is exactly what it should be.
Continuing to step through the code, we eventually try to use BQCoreUtils.convertUnixToTimestamp(). Now, while I've tried to disassemble the class (thanks IntelliJ!), I can't quite figure out what's going on. It eventually tries to create a new TimestampTz() which is an extension of java.sql.Timestamp, but the value getting passed for nanos is negative. This of course prompts Java to throw an IllegalArgumentException, but I can't figure out what I need to do to avoid this.
I also don't know if there's a simpler explanation for what's going on. Ultimately though, it appears that there's a driver bug. BQCoreUtils.convertUnixToTimestamp doesn't properly safeguard against the nanos calculation being non-negative.
The dumb question then is: has anyone else experience issues querying BigQuery where Timestamp values are triggering exceptions?
Update: Since this is happening in Timestamp created by JDBC driver, it does appear to be a bug in JDBC driver itself. Please file it under https://issuetracker.google.com/issues?q=componentid:187149.
Original:
The timestamp resolution in BigQuery is microseconds, and it looks like the value you are providing is in seconds, so you should multiply it by 1000000.
With reference to Google issue tracker,
This should be resolved with versions newer than 1.1.1 of the drivers, which also addressed other timestamp-related issues.
If any issue persists, please report at Google issue tracker they will re-open to examine.
I am developing Ruby on Rails application which uses Thinking Sphinx. Unfortunately, from time to time (few times per month) search tends to return bad results (many documents missing). Reindexing helps, but this is not a solution for production.
I am experiencing bad results even when I am typing simple queries into rails console (like ThinkingSphinx.search 'skalee'). Sphinx search tool returns proper results, so indexing apparently works properly.
When I type in ThinkingSphinx.search('skalee').results[:words] I see proper numbers of hits (for example, term found in 30 documents) but ThinkingSphinx.search('skalee').results[:matches] contains, let's say, 2 documents. The numbers in results[:words] are equal to those I am getting with search.
I am using delayed delta but this problem is appearing even when I am not running ts:dd.
Just happened to stumble across this:
http://freelancing-god.github.com/ts/en/common_issues.html#deltas
Perhaps your user permissions are off?
Thinking Sphinx (or Delayed Delta, I don't remember well) adds special, internal attribute (sphinx_deleted or something like this) to all models. It is used to filter out destroyed records. Unfortunately, it works bad from time to time. After modifying the gem (getting rid of this attribute) everything works fine. Of course I need to wait till full reindexing (which I perform every night) to remove destroyed records from indexes, but this is minor disadvantage in my case. Alternatively, I could use Sphinx's kill list feature to filter out removed entries.
I have a 400 line sql query which is throwing exception withing 30 seconds
ORA-03113: end-of-file on communication channel
Below are things to note:
I have set the timeout as 10 mins
There is one last condition when removed resolves this error.
This error came only recently when I analyzed indexes.
The troubling condition is like this:
AND UPPER (someMultiJoin.someColumn) LIKE UPPER ('%90936%')
So my assumption is that the query is getting terminated from the server side apparently because its identified as a resource hog.
Is my assumption appropriate ? How should I go about to fix this problem ?
EDIT: I tried to get the explain plan of faulty query but the explain plan query also gives me an ORA-03113 error. I understand that my query is not very performant but why should that be a reason for ORA-03113 error. I am trying to run the query from toad and there are no alert log or trace generated, my db version is
Oracle9i Enterprise Edition Release 9.2.0.7.0 - Production
One possible cause of this error is a thread crash on the server side. Check whether the Oracle server has generated any trace files, or logged any errors in its alert log.
You say that removing one condition from the query causes the problem to go away. How long does the query take to run without that condition? Have you checked the execution plans for both versions of the query to see if adding that condition is causing some inefficient plan to be chosen?
I've had similar connection dropping issues with certain variations on a query. In my case connections dropped when using rownum under certain circumstances. It turned out to be a bug that had a workaround by adjusting a certain Oracle Database configuration setting. We went with a workaround until a patch could be installed. I wish I could remember more specifics or find an old email on this but I don't know that the specifics would help address your issue. I'm posting this just to say that you've probably encountered a bug and if you have access to Oracle's support site (support.oracle.com) you'll likely find that others have reported it.
Edit:
I had a quick look at Oracle support. There are more than 1000 bugs related to ORA-03113 but I found one that may apply:
Bug 5015257: QUERY FAILS WITH ORA-3113 AND COREDUMP WHEN QUERY_REWRITE_ENABLED='TRUE'
To summarize:
Identified in 9.2.0.6.0 and fixed in 10.2.0.1
Running a particular query
(not identified) causes ORA-03113
Running explain on query does the
same
There is a core file in
$ORACLE_HOME/dbs
Workaround is to set
QUERY_REWRITE_ENABLED to false: alter
system set query_rewrite_enabled =
FALSE;
Another possibility:
Bug 3659827: ORA-3113 FROM LONG RUNNING QUERY
9.2.0.5.0 through 10.2.0.0
Problem: Customer has long running query that consistently produces ORA-3113 errros.
On customers system they receive core.log files but do not receive any errors
in the alert.log. On test system I used I receivded ORA-7445 errors.
Workaround: set "_complex_view_merging"=false at session level or instance level.
You can safely remove the "UPPER" on both parts if you are using the like with numbers (that are not case sensitive), this can reduce the query time to check the like sentence
AND UPPER (someMultiJoin.someColumn) LIKE UPPER ('%90936%')
Is equals to:
AND someMultiJoin.someColumn LIKE '%90936%'
Numbers are not affected by UPPER (and % is independent of character casing).
From the information so far it looks like an back-end crash, as Dave Costa suggested some time ago. Were you able to check the server logs?
Can you get the plan with set autotrace traceonly explain? Does it happen from SQL*Plus locally, or only with a remote connection? Certainly sounds like an ORA-600 on the back-end could be the culprit, particularly if it's at parse time. The successful run taking longer than the failing one seems to rule out a network problem. I suspect it's failing quite quickly but the client is taking up to 30 seconds to give up on the dead connection, or the server is taking that long to write trace and core files.
Which probably leaves you the option of patching (if you can find a relevant fix for the specific ORA-600 on Metalink) or upgrading the DB; or rewriting the query to avoid it. You may get some ideas for how to do that from Metalink if it's a known bug. If you're lucky it might be as simple as a hint, if the extra condition is having an unexpected impact on the plan. Is someMultiJoin.someColumn part of an index that's used in the successful version? It's possible the UPPER is confusing it and you could persuade it back on to the successful plan by hinting it to use the index anyway, but that's obviously rather speculative.
It means you have been disconnected. This not likely to be due to being a resource hog.
I have seen where the connection to the DB is running over a NAT and because there is no traffic it closes the tunnel and thus drops the connection. Generally if you use connection pooling you won't get this.
As #Daniel said, the network connection to the server is being broken. You might take a look at End-of-file on communication channel to see if it offers any useful suggestions.
Share and enjoy.
This is often a bug in the Cost Based Optimizer with complex queries.
What you can try to do is to change the execution plan. E.g. use WITH to pull some subquerys out. Or use the SELECT /*+ RULE */ hint to prevent Oracle from using the CBO. Also dropping the statistics helps, because Oracle then uses another execution plan.
If you can update the database, make a test installation of 9.2.0.8 and see if the error is gone there.
Sometimes it helps to make a dump of the schema, drop everything in it and import the dump again.
I was having the same error, in my case what was causing it was the length of the query.
By reducing said length, I had no more problems.
I'm in a situation where I have many connections that are "IDLE in transaction". I
spotted them myself. I could kill them, but that wouldn't prevent them
from happening again. In fact, it happens regularily.
Is there a way to get the list of SQL statements that were previously
executed as part of a given transaction?
If I could get that, it would be a hell lot easier to figure out the
misbehaving client.
There is some work being done right now on the pgsql-hackers mailing list toward adding exactly this capability, under the title "display previous query string of idle-in-transaction". Where that looks to be going is that pg_stat_activity will have a new column named something like "last_query" that includes the info you want.
Until that's done and available in probably the next release, the suggestion from depesz is probably as good as you're going to get here--unless you want to start grabbing early patches working on this feature as they trickle out.
Basically - you have to turn on all statements logging, with time of execution. best way to achieve it is to use log_min_duration_statement with value of 0, and using log_line_prefix such that is it will include information required to match lines coming form the same backend.
I generally use log_line_prefix = '%m %u#%d %p %r '.
Afterward you can write some tool to help you hunt idle-in-transaction, or you can use mine.