I am trying to use quarkus healthcheck, and I use it like this:
quarkus:
datasource:
metrics:
enabled: true
jdbc:
enable-metrics: true
maria-db-datasource:
db-kind: mariadb
jdbc:
url: ${DB_URL:jdbc:mariadb://localhost:3306/feed}
enable-metrics: true
validation-query-sql: "INSERT INTO feed(ID, FEED_EVENT_ID, FEED_PROVIDER, CREATED_TIMESTAMP, CONTENT) VALUES (-1000, -1000, 'irrelevant', NOW(), 'irrelevant')"
In reality, I am not doing an insert, but I wanted to see that it reports DOWN. To my surprise, running curl http://localhost:8080/q/health/ a few times reveals no problem, and the service reports UP. I have to issue curl a few good times, to see it reporting DOWN with a duplicate exception.
How come? is there a setting that invalidates the previous result? Is there a cache somewhere? I've tried to debug, but nothing so far.
Thank you
Related
i'm using elk stack of version 5.5 in ubuntu 16.0
My logs are getting broken and not writting properly into elastic which is resulting in json.erros
like below
Error decoding JSON: invalid character 'e' in literal null (expecting 'u')"
getting json.errors very frequent and those logs are not reading or writting properly into elasticsearch ?
and this is happening for every 5 to 10 mins. please help me solve it.
screenshot of broken logs in kibana
My sample log is :
{"log":"2019-10-01 07:18:26:854*[DEBUG]*cluster2-nio-worker-0*Connection*userEventTriggered*Connection[cassandraclient/10.3.254.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat\n","stream":"stdout","time":"2019-10-01T07:18:26.85462769Z"}
Since you have stated that the json logs are not pretty printed I assume that the multiline-settings of your input configuration are causing the problems.
In my opinion you don't need any multiline settings when you have logs in json format and they are not pretty printed, meaning the whole json object (= log event) is written in one line.
You have already specified
json.message_key: log
This solely should get the job done.
So to sum it up:
Remove the multiline settings and try again. Your configuration should look like this:
filebeat.inputs:
- type: log
paths:
- "/var/log/containers/*.log"
tags: ["kube-logs"]
symlinks: true
json.message_key: log
json.keys_under_root: true
json.add_error_key: true
I am using the standard HikariCP implementation in my SpringBoot 2.0.1 application. But after a while I get the same error over and over
HikariPool-1 – Connection is not available, request timed out after 30000ms
I first checked the code if there are any not closed connection or missing Transactional annotations but I did not find anything. I also tried then to increase the pool and decrease the time out in my application.yml but it seems that this does not have any effect.
The weird thing is that HikariCP seems to create only 4 pooled connection and I am not able to override these properties.
HikariPool-1 - Timeout failure stats (total=4, active=4, idle=0, waiting=100)
This is my application.yml file
spring:
jpa:
hibernate:
ddl-auto: update
datasource:
hikari:
maximum-pool-size: 64
connection-timeout: 5000
Your application is not able to get the connection within the 30s which is the default connection timeout for HikariCP. There can be reasons for that such as:
The Pool size is small, and the requests take too long to execute (more than 30 s).
Possible explanation is that for e.g your pool size is 1 and your one request/query takes approximately 500ms to execute. suppose you have opened for example approximately 200 concurrent query requests, then there might be a chance that some requests will be timed out. (one request has waited for the 30s but a connection was not available for it).
Disclaimer: Numbers are taken randomly to explain and is not tested.
To solve this you might want to increase the connectionTimeout or increase the pool size. ( Also check for the memory requirements).
In application.properties set the values
spring.datasource.hikari.minimumIdle=10
spring.datasource.hikari.maximumPoolSize=10
spring.datasource.hikari.idleTimeout=120000
spring.datasource.hikari.maxLifetime=500000
Official Hikari CP documentation link
There can be another reason for this is that your connection is not getting closed properly use try-with-resources to close your connection
e.g
try(Connection c = dataSource.getConnection()) {
//ToDo: Add your logic statement here
log.debug("[d] Tenant '{}' added.", tenantId);
}
I'm having problems getting hystrix timeouts to work. I've created an example project to show this here: https://github.com/stianlagstad/spring-boot-timeout-demo.
In bootstrap.yml I'm setting a timeout like this:
hystrix:
command:
default:
execution.isolation.thread.timeoutInMilliseconds: 60000
circuitBreaker:
enabled: true
sleepWindowInMilliseconds: 300000
fallback.enabled: false
# My client
MyFeignClient#getPost:
execution.isolation.thread.timeoutInMilliseconds: 1
I expect the result of this to be that hystrix commands should timeout after 60 seconds, except for getPost in MyFeignClient which should timeout after 1 millisecond. I'm not seeing that, though. The getPost method returns an answer every time, and I'm pretty sure it takes longer than one millisecond.
I've also tried to set the timeout manually in a test using ConfigurationManager, but that doesn't seem to work either: https://github.com/stianlagstad/spring-boot-timeout-demo/blob/master/src/test/java/com/example/TimeoutDemoApplicationTests.java
How can I make the timeouts I'm setting take effect?
You need to fix your properties in two places.
First, add the below property. From dalston release, feign's hystrix support is optional. You already have hystrix on your classpath, so all you need to do is just adding the below property.
feign:
hystrix:
enabled: true
Second, you specified wrong HystrixCommandKey for your feign. You need to change your HystrixCommandKey like below.
MyFeignClient#getPost():
You need parentheses after #getPost.
is there a way to find queries in mongodb that are not using Indexes or are SLOW? In MySQL that is possible with the following settings inside configuration file:
log-queries-not-using-indexes = 1
log_slow_queries = /tmp/slowmysql.log
The equivalent approach in MongoDB would be to use the query profiler to track and diagnose slow queries.
With profiling enabled for a database, slow operations are written to the system.profile capped collection (which by default is 1Mb in size). You can adjust the threshold for slow operations (by default 100ms) using the slowms parameter.
First, you must set up your profiling, specifying what the log level that you want. The 3 options are:
0 - logger off
1 - log slow queries
2 - log all queries
You do this by running your mongod deamon with the --profile options:
mongod --profile 2 --slowms 20
With this, the logs will be written to the system.profile collection, on which you can perform queries as follows:
find all logs in some collection, ordering by ascending timestamp:
db.system.profile.find( { ns:/<db>.<collection>/ } ).sort( { ts: 1 } );
looking for logs of queries with more than 5 milliseconds:
db.system.profile.find( {millis : { $gt : 5 } } ).sort( { ts: 1} );
You can use the following two mongod options. The first option fails queries not using index (V 2.4 only), the second records queries slower than some ms threshold (default is 100ms)
--notablescan
Forbids operations that require a table scan.
--slowms <value>
Defines the value of “slow,” for the --profile option. The database logs all slow queries to the log, even when the profiler is not turned on. When the database profiler is on, mongod the profiler writes to the system.profile collection. See the profile command for more information on the database profiler.
You can use the command line tool mongotail to read the log from the profiler within a console and with a more readable format.
First activate the profiler and set the threshold in milliseconds for the profile to consider an operation to be slow. In the following example the threshold is set to 10 milliseconds for a database named "sales":
$ mongotail sales -l 1
Profiling level set to level 1
$ mongotail sales -s 10
Threshold profiling set to 10 milliseconds
Then, to see in "real time" the slow queries, with some extra information like the time each query took, or how many registries it need to "walk" to find a particular result:
$ mongotail sales -f -m millis nscanned docsExamined
2016-08-11 15:09:10.930 QUERY [ops] : {"deleted": {"$exists": false}, "prod_id": "367133"}. 8 returned. nscanned: 344502. millis: 12
2016-08-11 15:09:10.981 QUERY [ops] : {"deleted": {"$exists": false}, "prod_id": "367440"}. 6 returned. nscanned: 345444. millis: 12
....
In case somebody ends up here from Google on this old question, I found that explain really helped me fix specific queries that I could see were causing COLLSCANs from the logs.
Example:
db.collection.find().explain()
This will let you know if the query is using a COLLSCAN (Basic Cursor) or an index (BTree), among other things.
https://docs.mongodb.com/manual/reference/method/cursor.explain/
While you can obviously use Profiler a very neat feature of Mongo DB due to which I actually fall in love with it is Mongo DB MMS.
Takes less than 60 seconds and can manage from anywhere. I am sure you will Love it.
https://mms.mongodb.com/
I get the following error under a certain scenario
When a different thread is populating a lot of users via the bulk upload operation and I was trying to view the list of all users on a different web page. The list query, throws the following timeout error. Is there a way to set this timeout so that I can avoid this timeout error.
Env: h2 (latest), Hibernate 3.3.x
Caused by: org.h2.jdbc.JdbcSQLException: Timeout trying to lock table "USER"; SQL statement:
[50200-144]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
at org.h2.message.DbException.get(DbException.java:167)
at org.h2.message.DbException.get(DbException.java:144)
at org.h2.table.RegularTable.doLock(RegularTable.java:482)
at org.h2.table.RegularTable.lock(RegularTable.java:416)
at org.h2.table.TableFilter.lock(TableFilter.java:139)
at org.h2.command.dml.Select.queryWithoutCache(Select.java:571)
at org.h2.command.dml.Query.query(Query.java:257)
at org.h2.command.dml.Query.query(Query.java:227)
at org.h2.command.CommandContainer.query(CommandContainer.java:78)
at org.h2.command.Command.executeQuery(Command.java:132)
at org.h2.server.TcpServerThread.process(TcpServerThread.java:278)
at org.h2.server.TcpServerThread.run(TcpServerThread.java:137)
at java.lang.Thread.run(Thread.java:619)
at org.h2.engine.SessionRemote.done(SessionRemote.java:543)
at org.h2.command.CommandRemote.executeQuery(CommandRemote.java:152)
at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:96)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:208)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1808)
at org.hibernate.loader.Loader.doQuery(Loader.java:697)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2228)
... 125 more
Yes, you can change the lock timeout. The default is relatively low: 1 second (1000 ms).
In many cases the problem is that another connection has locked the table, and using multi-version concurrency also solves the problem (append ;MVCC=true to the database URL).
EDIT: MVCC=true param is no longer supported, because since h2 1.4.200 it's always true for a MVStore engine, which is a default engine.
I faced quite the same problem and using the parameter "MVCC=true", it solved it. You can find more explanations about this parameter in the H2 documentation here : http://www.h2database.com/html/advanced.html#mvcc
I'd like to suggest that if you are getting this error, then perhaps you should not be using a transaction on your bulk database operation. Consider instead doing a transaction on each individual update: does it make sense to think of an entire bulk import as a transaction? Probably not. If it does, then yes, MVCC=true or a bigger lock timeout is a reasonable solution.
However, I think for most cases, you are seeing this error because you are trying to perform a very long transaction - in other words you are not aware that you are performing a really long transaction. This was certainly the case for myself and I simply took more care on how I was writing records (either using no transactions or using smaller transactions) and the lock timeout issue was resolved.
For those having this issue with integration tests (i.e. server is accessing the h2 db and an integration test is accessing the db before calling the server, to prepare the test), adding a 'commit' to the script executed before the test makes sure that the data are in the database before calling the server (without MVCC=true - which I find is a bit 'weird' if it is not enabled by default).
I had MVCC=true in my connection string but still was getting error above. I had added ;DEFAULT_LOCK_TIMEOUT=10000;LOCK_MODE=0 and problem was solved
I got this issue with the PlayFramework
JPAQueryException occured : Error while executing query from
models.Page where name = ?: Timeout trying to lock table "PAGE"
It ended being an infinite loop of sorts because I had a
#Before
without an unless which caused the function to repeatedly call itself
#Before(unless="getUser")
Working with DBUnit, H2 and Hibernate - same error, MVCC=true helped, but I would still get the error for any tests following deletion of data. What fixed these cases was wrapping the actual deletion code inside a transaction:
Transaction tx = session.beginTransaction();
...delete stuff
tx.commit();
From a 2020 user, see reference
Basically, the reference says:
Sets the lock timeout (in milliseconds) for the current session. The default value for this setting is 1000 (one second).
This command does not commit a transaction, and rollback does not affect it. This setting can be appended to the database URL: jdbc:h2:./test;LOCK_TIMEOUT=10000