Facing a strange issue for sometime in Spring Session with Pivotal GemFire integration.
We have multiple HTTP requests, which eventually does set/get of session attributes in a varying order based on several conditions.
At some given point...
(T) session.getAttribute(sessionKeyN); // (T) is template object
... is retrieving null. We have cross verified that no session.setAttribute(..) is invoked in between two session.getAttribute(..) calls out of which one misses the object.
We enabled trace logging in GemFire client. There we see a mismatch in hashmaps being read/written. Sharing the logs:
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.DataSerializer: **Writing HashMap with 8 elements**: {LOGIN_DATE_TIME=09-24-2018 02:46:08 AM, TERMINAL=terminal.Terminal#5aa33970, org.springframework.session.FindByIndexNameSessionRepository.PRINCIPAL_NAME_INDEX_NAME=testuser, LOG_TRANSACTION_ID=20180924_451_5_4,20180912045104000005, USER=user.User#70b4b11b, TRANSACTION_HEADER=TransactionHeader#4144221c, XXXX_LOGIN=true, SPRING_SECURITY_CONTEXT=org.springframework.security.core.context.SecurityContextImpl#65bb8fc1: Authentication: org.springframework.security.authentication.UsernamePasswordAuthenticationToken#65bb8fc1: Principal: testuser; Credentials: [PROTECTED]; Authenticated: true; Details: null;}
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.internal.**InternalDataSerializer: basicWriteObject**: {LOGIN_DATE_TIME=09-24-2018 02:46:08 AM, TERMINAL=terminal.Terminal#5aa33970, org.springframework.session.FindByIndexNameSessionRepository.PRINCIPAL_NAME_INDEX_NAME=testuser, LOG_TRANSACTION_ID=20180924_451_5_4,20180912045104000005, USER=user.User#70b4b11b, TRANSACTION_HEADER=TransactionHeader#4144221c, XXXX_LOGIN=true, SPRING_SECURITY_CONTEXT=org.springframework.security.core.context.SecurityContextImpl#65bb8fc1: Authentication: org.springframework.security.authentication.UsernamePasswordAuthenticationToken#65bb8fc1: Principal: testuser; Credentials: [PROTECTED]; Authenticated: true; Details: null;}
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.DataSerializer: Writing STRING_BYTES of len=8
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.DataSerializer: Writing String "testuser"
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.DataSerializer: Writing STRING_BYTES of len=36
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.DataSerializer: Writing String "5c4948d9-7438-4dff-badc-fdc0f9997781"
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.internal.InternalDataSerializer: basicWriteObject: { #type = org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSession, id = 5c4948d9-7438-4dff-badc-fdc0f9997781, creationTime = 2018-09-24T09:44:23.180Z, lastAccessedTime = 2018-09-24T09:46:14.909Z, maxInactiveInterval = PT30M, principalName = testuser }
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.internal.InternalDataSerializer: DataSerializer Serializing an instance of org.apache.geode.cache.Operation
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.internal.InternalDataSerializer: basicWriteObject: UPDATE
>2018-09-24T02:46:15:342-0700 [20180924_451_5_4,20180912045104000005] DEBUG org.apache.geode.cache.client.internal.PutOp: PutOpImpl constructing message with operation=UPDATE
>2018-09-24T02:46:15:144-0700 [20180924_451_5_4,20180912045104000005] DEBUG org.apache.geode.internal.cache.LocalRegion: invoking listeners: [org.springframework.session.data.gemfire.GemFireOperationsSessionRepository#4471a4f]
>2018-09-24T02:46:15:144-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.internal.cache.LocalRegion: dispatchListenerEvent event=EntryEventImpl[op=LOCAL_LOAD_CREATE;region=/XXXX2wl;key=5c4948d9-7438-4dff-badc-fdc0f9997781;oldValue=null;newValue={ #type = org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSession, id = 5c4948d9-7438-4dff-badc-fdc0f9997781, creationTime = 2018-09-24T09:44:23.180Z, lastAccessedTime = 2018-09-24T09:46:15.079Z, maxInactiveInterval = PT30M, principalName = testuser };callbackArg=null;originRemote=false;originMember=tstplXXXX0004(ClientConfigXXXX2Application:28299:loner):35884:0c27e20a:ClientConfigXXXX2Application;callbacksInvoked;version={v20; rv161; mbr=10.5.230.71(server_devplgemf0066:123628)<v23>:1024; time=1537782375131; remote};isFromServer]
>2018-09-24T02:46:15:144-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.internal.cache.versions.VersionTag: deserializing class org.apache.geode.internal.cache.versions.VMVersionTag with flags 0x4
>2018-09-24T02:46:15:144-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.internal.InternalDataSerializer: basicReadObject: header=1
>2018-09-24T02:46:15:144-0700 [20180924_451_5_4,20180912045104000005] TRACE org.apache.geode.DataSerializer: **Read HashMap with 9 elements**: {LOGIN_DATE_TIME=09-24-2018 02:46:08 AM, TERMINAL=terminal.Terminal#5a2aa051, **CUSTOMER_SEARCH_RESPONSE=CustomerInfo#600fa25f**, org.springframework.session.FindByIndexNameSessionRepository.PRINCIPAL_NAME_INDEX_NAME=testuser, LOG_TRANSACTION_ID=20180924_451_5_4,20180912045104000005, USER=user.User#7178708f, TRANSACTION_HEADER=TransactionHeader#30215dcd, XXXX_LOGIN=true, SPRING_SECURITY_CONTEXT=org.springframework.security.core.context.SecurityContextImpl#65bb8fc1: Authentication: org.springframework.security.authentication.UsernamePasswordAuthenticationToken#65bb8fc1: Principal: testuser; Credentials: [PROTECTED]; Authenticated: true; Details: null;}
Attribute CUSTOMER_SEARCH_RESPONSE is missing even though no session.setAttribute(..) for it was invoked.
This is not WRT to one attribute and also not consistent. A re-run might not show this issue.
After working with another Pivotal (GemFire) customer on a similar problem (also using Spring Session and Pivotal GemFire (SSDG) to manage HTTP session state in a highly concurrent Web application/environment), we uncovered the underlying issue(s) and ultimately found BUGS in Pivotal GemFire!
In nutshell, these bugs lead to Lost Updates due to Race Conditions that are exasperated in a highly concurrent (multi-user) Web environment, where multiple HTTP requests might possibly be accessing and modifying the same HTTP session under load. And, the greater the concurrency (users) and the greater the load (number of HTTP requests to the same HTTP session), the more apparent this problem becomes.
In fact, I have written several Integration Tests illustrating this problem.
First, I wrote a Load Integration Test (MultiThreadedClientProxyRegionSessionIntegrationTests). This class spawns 180 Threads (users) performing 10,000 concurrent requests to the same underlying Session. The Session object, though not exactly the same, is modeled after SSDG's GemFireSession object representation.
Second, I wrote another Integration Test (TwoThreadsClientProxyRegionSessionIntegrationTests) that reliably and repeatedly reproduces the problem.
Both of these test classes were written purely with the GemFire API, thereby illustrating that Pivotal GemFire is the problem, not SSDG.
I have written similar test using Spring Session Data GemFire both in my example, and now, also included in the SSDG test suite (along with many other MultiThread/Concurrency based integration tests), itself, ensuring Spring Session (for Pivotal GemFire) will never encounter this problem again, and if it does, I'll know about it sooner than later.
In short, the 2 underlying Pivotal GemFire bugs are:
GEODE-6152
GEODE-6032
The workaround is as follows:
First, you must configure your Spring Session, GemFire cache client application with:
A client PROXY Region to manage the Session state (default)
Set copy-on-read set to true.
And, you must use GemFire DataSerialization, by setting the sessionSerializerBeanName appropriately:
#SpringBootApplication
#ClientCacheApplication(copyOnRead = true, subscriptionEnabled = true)
#EnableGemFireHttpSession(
clientRegionShortcut = ClientRegionShortcut.PROXY,
sessionSerializerBeanName = GemFireHttpSessionConfiguration.SESSION_DATA_SERIALIZER_BEAN_NAME
)
class MySpringBootSpringSessionDataGemFireApplication {
...
}
See here, for example.
You will also need to upgrade to Spring Session for Pivotal GemFire 2.1.2.RELEASE (to be released soon), since I made several important, recent enhancements, such as:
Issue #12 - Prevent SessionRepository.save(Session) on non-dirty Sessions.
Issue #9 - Add server-side configuration support for GemFire/Geode DataSerialization when SSDG is not used to configure Spring Session on the servers.
Issue #17 - Consider support for customizable IsDirty application domain object checking.
Using GemFire DataSerialization with Deltas will not prevent, but greatly reduces the possibility of lost updates and other race condition intrinsically inherit in a Web environment, particularly since a Servlet container (e.g. Tomcat) is multi-threaded, processing each HTTP request in a separate thread.
While SSDG goes to great effort to ensure the HTTP session representation (i.e. GemFireSession) is thread-safe, you must also ensure that any object you put in the HTTP session is also thread-safe since it can and most likely will be accessed by more than 1 thread in a highly-concurrent Web application, especially 1 where more than a single HTTP request can access the same HTTP session (by session ID) at a time.
Anyway, food for thought.
When the configuration above is used, everything works as expected, and when not, lost updates can and will occur dues to the GemFire BUGS!
In fact, my load test revealed that out of 10,000 Session updates, where ~9800 Session attributes get added, only ~1100 successfully are, that is a whopping ~89% loss of data!!!
However, when the configuration above is applied, all data is accounted for correctly.
Hope this helps!
Related
I have a spring boot app which uses spring data and hikaricp for db connection pooling. I noticed the following behaviour that looks strange to me:
I have one method which is not transactional and in that method several db queries are executed using spring data repositories
public void testMethod(final Long firstRepositoryId, final Long secondRepositoryId) {
final DomainObject result = firstRepository.findById(firstRepositoryId);
// here there's some code that is processing the result without db queries
secondRepository.findById(secondRepositoryId);
// some more logic here without additional db queries
}
So as expected when there's no transaction on the method then the spring data methods opens a transaction for executing the query and complete it after the methods returns. I have enabled transaction logging so there's the following log output:
2021-06-03 15:34:30.961 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Getting transaction for [com.test.FirstRepository.findById]
2021-06-03 15:34:30.966 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Completing transaction for [com.test.FirstRepository.findById]
2021-06-03 15:34:30.967 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Getting transaction for [com.test.SecondRepository.findById]
2021-06-03 15:34:30.972 TRACE c681f76a-5d7e-41d5-9e50-fb6f96169681 --- [tp659271212-291] o.s.t.i.TransactionInterceptor : Completing transaction for [com.test.SecondRepository.findById]
Everything seems to be exactly how I expects to be. The thing I can't understand is the hikari behaviour. This method is invoked within a http request. A connection is taken from hikari cp right after the execution of the firstRepository.findById but this connection is returned in the pool only after the http controller returns response. What I expect is that a connection is taken after a transaction is opened and returned back after the transaction is completed. Is there something that I miss or maybe I have some wrong configuration?
P.S. I'm monitoring the active hikari connections through the spring boot actuator prometheus data. And to be able to reproduce the behavior I explained above I'm suspending the connection thread with several debug breakpoints.
I found out what causes this behaviour- it's Spring functionality to maintain the hibernate session in view in order to be able to retrieve lazy loaded data in the view. In order to disable this you need the following property:
spring.jpa.open-in-view=false
Here's another SO post where is explained what it does:
What is this spring.jpa.open-in-view=true property in Spring Boot?
Most of my application is secured with method level security (AspectJ, but it doesn't matter) and now that I am trying to call some code from within application itself (not controllers, but e.g. EventListener) I can't help to wonder if Spring Security provides some out-of-box way of giving Authentication object to the application itself, otherwise I cannot get past my method security since application has null security objects (Authentication in SecurityContext, if it even exists - depends on situation, You might have to init it first).
Sure I can do something like this (just before running relevant code):
UserDetails ud = User.builder()
.username("APPLICATION")
.password("APPLICATION")
.roles("APPLICATION")
.build();
Authentication auth = new UsernamePasswordAuthenticationToken(ud, ud.getPassword(), ud.getAuthorities());
SecurityContextHolder.getContext()
.setAuthentication(auth);
But is it safe to do this in deployment (security-wise)?
Is there any guarantee on which thread will own this SecurityContext? What about other threads and their tasks?
Once set, can it stay there? Will it for the rest of app's run (can be days/months), context could be reloaded, etc. I lack deep Spring knowledge to know what happens Thread-wise inside Spring.
I am working on a SpringBoot application. I am running integration tests on REST APIs using SpringBooTest and TestNG. It is a stateless application. Sessions are not stored.
Context
I have filter that explicitly sets the authentication object into SecurityContext like below
Authentication authentication = new UsernamePasswordAuthenticationToken(contextUser, "", null);
SecurityContextHolder.getContext().setAuthentication(authentication);
// Continue to app.
After this point, I am not using any authentication manager as the token is already validated. I am not explicitly clearing the security context any where.
Problem
When I run a couple of tests from SpringBootTest, the second test still has access to security context set in the first test. When I ran the test in debug mode, I can see SecurityContextHolder.clearContext(); getting called from SecurityContextPersistenceFilter which means context should be cleared.
How do I solve this issue? I would think security context should be always cleared after the request is complete and should not be available for the next test.
I had to set a FilterChainProxy like below for the filters to get executed in Spring Integration test, which would take care of clearing the context.
restLoginMockMvc = MockMvcBuilders.standaloneSetup(loginResource).setControllerAdvice(exceptionTranslator)
.apply(springSecurity(springSecurityFilterChain)).build();
All,
I've hit a roadblock implementing OpenId due to some unexpected behavior storing session information.
The basic outline is as follows
// In OpenId Initialize Servlet
request.getSession().setAttribute("isLinkingRequired", true);
response.sendRedirect(openIdUrl);
// In Open ID Verify Servlet
boolean linkCheck = request.getSession().getAttribute("isLinkingRequired") != null
The sample provided to me by Intuit is written with Spring. In the Spring app, linkCheck evaluates to true. The previously stored session information is there.
My app is written with JSF (in particular, Oracle's ADF framework). In my case linkCheck evaluates to false. All previous stored session attributes are gone!
This has caused me a boatload of problems. Can anyone explain this odd behaviour?
I'd like to know if the Spring framework's HttpSessionMutexListener listener is still relevant in today's application servers/web containers (say 2.5+ servlet spec servers like Tomcat 6 or Tomcat 7) for locking the session object in clustered environments (ie. among different JVMs), or do they only resolve the concurrency issue in clustered environments for 2.3 (or previous) servlet spec containers and now it is unnecessary?
I think you're granting more power to Spring's session mutex than it deserves. It's simply a session attribute stored under a public name, WebUtils.SESSION_MUTEX_ATTRIBUTE, that's intended to be used in the expression for the synchronized statement. I'm not really sure how it could be used for "locking the session object in clustered environments". Here's a usage snippet from Spring's own code:
HttpSession session = request.getSession(false);
if (session != null) {
Object mutex = WebUtils.getSessionMutex(session);
synchronized (mutex) {
return handleRequestInternal(request, response);
}
}
A reference to the mutex object in one JVM will not be available to another JVM, so acquiring its lock will not have any impact on code running in another JVM. However, the servlet spec does contain the following:
Within an application marked as distributable, all requests that are
part of a session must be handled by one JVM at a time.
This requirement has been around since at least 2.3 and may cause a distributed app to behave as if the Spring mutex were doing something when, in fact, it's the container that is forcing requests to be handled by one JVM at time.
As an aside, this reminds me of a post I made to concurrency-interest a few years back that refers to Spring's session mutex:
JTaP article on stateful web apps
Update based on comment:
Assume that JVM-1 and JVM-2 make up two nodes in a cluster. Also assume that request-1 and request-2 participate in the same session. If request-1 is being handled in JVM-1, then request-2 cannot be handled in JVM-2 until request-1 has completed. However, request-2 can be handled concurrently by JVM-1.
For the case where the requests are handled in different JVMs, the implication is that any session changes caused by the first request (JVM-1) will be visible to the second request (JVM-2).