I've got a website build with Spring and jpa (by hibernate). I've got a bug and I don't know how to identify the line where the bug appears.
I can't debug it on my ide because it's a live version (all runs fine in local).
I've got log which says:
o
rg.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)#012#011
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:625)#012#011
at com.mycompany.server.rpc.UserService$$EnhancerByCGLIB$$64ed2d4f.createAccount(<generated>)#012#011
at com.mycompany.server.rpc.ServiceRPCImpl.createAccount(ServiceRPCImpl.java:309)
My problem is the third line. As the UserService object is handled by Spring, it becomes a proxy and I can't know the line of the bug.
Do you know how to solve the problem ?
Thanks
Is it possible for you to change from cglib to jdk proxy? (Spring AOP proxy reference)
Basically: if you access your beans as interfaces, you can use jdk proxies (spring default mechanism), thereby leaving the underlying object intact and gaining access to line numbers in stack traces.
I would say that not being able to reproduce this locally is a significant restraint. I would try to set up your local environment or a test server to reproduce the problem, using JMeter or other load test software to simulate load of concurrent user accesses. Once this is done, your tweak/compile/test cycle becomes a lot shorter, and you can make experimental changes without fear of disrupting service on your production server. It may seem like a lot of effort, but the work will pay dividends not just for this bug, but for bugs you may encounter in future.
It sounds like it could be a threading bug, especially since spring by default uses singleton scope. With that in mind, look into creating multithreaded integration tests for the service that is failing. Once you have reproduced the bug through load testing, you can verify that it's a threading bug by making your main service method synchronized, preventing concurrent use. If the bug disappears, it is most likely a concurrency bug.
As to finding the line of the bug - there is no line to look for since the code is generated. The best you can do is to add defensive checks in all beans that are being used in the advice around the UserService. (E.g. check for null values due to missing injections.) The init-method attribute on beans is useful for performing checks that the bean has been fully constructed and all required collaborators have been set.
If you cannot reproduce the issue in local environment, then may be it is environment / network related issue. I would first recreate the issue in test environment ( which is closer to production environment and not just own local machine ) to debug the bug.
You may also use Fiddler to debug network related issues for a live version.
Related
Spring lists SO as the only place to ask questions on their community page, which is why I ask this rather generic question here. It may not be the best fit for SO, but, according to Spring's community overview page, there's no other adequate place to ask such questions.
I have a spring boot application built on spring cloud gateway (version 2) which also uses an embedded hazelcast cluster. It runs in multiple instances, which communicate via hazelcast. Everything works fine, except under heavy load. If one instance fails, restarting it is no longer possible.
When the instance is restarted while the cluster of instances is under heavy load, it will start creating and wiring beans, up to some point, after which it will not do anything spring-related anymore. Hazelcast-generated messages are visible in the log (with root log level DEBUG), past that point, but nothing generated by spring or the application itself.
In order to restart that one instance that failed, I need to stop the load generation, wait some 10-15 minutes, then restart the failed instance. Then the new/restarted instance starts up rather quickly, with no problems at all.
The load consists of http requests which get proxied to another application, and is of such nature that it generates a lot of read accesses to hazelcast's distributed storage, but very few writes.
My problem: I have no idea how to debug this. Since the http endpoint never becomes available, there's no way I can query metrics or other actuator information.
So my question is: what tools or mechanisms can I employ to debug this problem? I.e. how can I find out exactly how the boot sequence under heavy load of the other instances of the hazelcast cluster differs from the boot sequence when there is no load at all in the cluster? Once I have this information, the problem is narrowed down enough for me to investigate it further on my own.
I didn't find a way to debug the problem, but had an idea of what might cause it, tried it, and it was a fix.
My application was running as a Kubernetes deployment. A few beans inside the application were relying on a usable CP subsystem during their initialization. Spring's bean initialization process is by necessity sequential and blocking, to account for inter-bean dependencies.
I hypothesized that under heavy load, for whatever reason, the initialization of those beans was blocking forever. As a first experiment, I made that initialization code async, so that Spring can finish bean wiring, even if, until that async part finished too, the instance was unable to perform usable work, to see if that was the problem, at least.
To my surprise, that fully fixed the problem. This way, Spring finished bean wiring, the HZ-dependant initialization also finished rather quickly, when executed async, even under high load, and the instance became usable soon after being started.
I didn't have the time to dig deeper to find out what the precise failure mechanism was. What I believe might have been the problem is the interaction between HZ and K8s. K8s-based discovery works using a K8S service. A pod/instance isn't added to the service until it becomes healthy. If a bean inside the application prevents initialization, the instance is never added to the service. As such, discovery never finds the new/restarted instance. I don't know what effect this might have on the HZ cluster's inner workings.
Spring boot 2.1.5
Project Reactor 3.2.9
I am setting up a bunch of rest reactive APIs using the above-mentioned frameworks and I am running into an annoying problem with MDC (mapped diagnostic context). My applications are in JAVA.
MDC relies on thread locals to store the current query's mapped context to put in the logs. That system, obviously, is not perfect and contradicts the reactive pattern since the different steps of your execution will be executed through different threads.
I have run into the same problem with the Play Reactive framework but found a workaround there by copying the mapped context transparently from one actor to another.
For spring and reactor, I could not find a satisfying solution yet.
Some random examples found on the internet:
First - It works but forces you to use a bunch of utility methods
Same thing
Second - It tries to copy the context during the onNext publisher event but seems to lose some features on the way of doing that. The signal context, for example, is lost.
I am in need of a proper solution to deal with this:
A library which would make the link between MDC and reactor?
A way to tweak reactor/spring to achieve it transparently?
Any advice?
"I could not find a satisfying solution yet."
Working with contexts is the only solution for the moment. Since as you said threadlocals goes against everything that has to do with reactive programming. Using thread local as a storage point during a request is a resource heavy way of solving things and in my opinion poor design. Unless logging frameworks themselves come up with a better solution to the problem we developers must pass the data through the context to accommodate for the logging frameworks blocking nature.
Reactive programming is a paradigm shift in the programming world. Other things like database drivers, that use threadlocal to rollback transactions are also in big trouble. the JDBC database driver spec is defined as blocking in nature, and atm. there has been attempts by spring and the R2DBC project to define a new JDBC driver spec that is inherently non/blocking. This means that all vendors must rewrite ther database driver implementations from scratch.
Reactive program is so new that lots of libraries need to rewrite entire codebases. The logging frameworks as we know it needs to be rewritten from the ground up which is a huge task. And the context in reactive is actually something that should not even be in reactive programming, it was implemented just to accommodate for MDC problems.
It's actually a lot of overhead needing to pass data from thread to thread.
So what can we do?
push on logging frameworks, and/or help logging frameworks to rewrite their codebase
Accept that there is no "tweak" that will magically fix this
use the context and the way suggested in the blogposts
Project reactor context
I have a large spring project, using xml configuration. I'm looking for a quick way to verify changes to the xml configuration.
I can load the whole project locally - the problem is this takes more than 5 minutes, loads a huge amount of data.
My XML editor catches XML formatting errors.
I'm looking for something intermediate - to catch obvious problems like references to beans that aren't defined, or calling constructors with the wrong arguments. Is there a quick way to do this, without having to actually invoke all the constructors and bring up the whole environment?
I'm building with Maven and editing with Eclipse, although my question isn't specific to either.
Since you already use Eclipse, you could try Spring Tool Suite (comes either standalone or as an add-on). It's essentially Eclipse with extra Spring-specific features, like Beans Validator. I'm not sure how thorough the validation is, but it should catch most configuration problems.
It's maintained by SpringSource so its integration with Spring "just works" and it's guaranteed not be more or less in sync with Spring Framework's release cycle.
Beanoh :
http://beanoh.org/overview.html#Verify
this project does exactly what I'm looking for. Verify obvious problems with spring config, but without the overhead of initializing everything.
You can use a Spring testing support to integration test your Spring configuration. However if the loading of the context is taking 5 mins, then the tests will also take the same amount of time. Spring does cache the context so if you have multiple tests using the same set of Spring contexts, then once cached the tests should be very quick.
I can suggest a few ways to more efficiently test your configuration:
Organize your project in modules, with each module being responsible for its own Spring configuration - this way, each module can be independently developed and tested.
If you have a modular structure, the testing can be more localized by mocking out the dependent modules, again this is for speed.
So this is a rather "big" question, but what I'm trying to accomplish is the following:
I have a Spring application, MVC, JDBC (MySQL) and JSP running on tomcat.
My objective is to test the entire "stack" using a proper method.
What I have so far is Junit using Selenium to simulate an actual user interacting with the application (requires a dummy account for that), and performing different validations such as, see if element is present in the page, see if the database has a specific value or if a value matches the database.
1st concern is that this is actually using the database so it's hard to test certain scenarios. I would really like to be able to mock the database. Have it emulate specific account configs, data states etc
2nd concern is that given the fact that I use what is in the database, and data is continuously changing, it is hard to predict behavior, and therefore properly asserting
I looked at Spring Test but it allows for testing outside a servlet container, so no JSP and no Javascript testing possible.
I saw DBUtils documentation but not sure if it will help me in this case
So, to my fellow developers, I would like to ask for tips to:
Run selenium tests on top of a mocked database
Allow different configs per test
Keep compatibility with Maven/Gradle
I have started with an ordered autowire feature to support this kind of stubbing.
It's basically an idea that i took over from the Seam framework i was working with in the past but i couldnt find yet a similar thing in spring.
The idea is to have a precedence annotation (fw, app,mock,...) that will be used to resolve the current implementation of an autowired bean. This is easy already in xml but not with java config.
So we have our normal repository beans in with app precedence and a test package stubbing these classes with mock precedence.
If both are in the classpath spring would normally fail with a duplicate bean found exception. In our case the extended beanfactory simply takes the bean with the highest precedence.
Im not sure if the order annotation of spring could be used directly but i prefered to have "well defined" precedence scopes anyway, so it will be clear for our developers what this is about.
! While this is a nice approach to stub so beans for testing i would not use it to replace a database definition but rather go with an inmemory database like hsql, like some previous answers mentionned already. !
I am considering to utilize Expression-Based Access Control from Spring Security 3.0.
The documentation says: You can access any of the method arguments by name as expression variables, provided your code has debug information compiled in.
That means that I have to have debug info left in my production wars and jars to properly use Expression-Based Access Control. For me it seems not very good idea at all.
Please tell me your opinions on this issue, so I can summarize your expirience to deside where I go for it or not.
Thank you in advance!
Max
It is a little strange, but this isn't tied to Spring Security. Spring Web MVC uses it too; e.g., to discover #RequestParam and #PathVariable default values.
In my experience people typically leave debug information in their builds (even production builds) to support troubleshooting (debug level logging is a different story), so Spring takes advantage of this. But it's fair to say that Spring is violating the principle of least surprise here, meaning that one wouldn't expect turning debug info off to turn a working app into a broken app.
Please see Spring security annotations with EL — requires debug information compiled in?