rails session being reset intermittently - session

Question

I know that this topic has been discussed a lot, but I believe that I've found a new variation of it: I have a Rails 4 application, which was upgraded from Rails 3, and which has the rails_ujs and csrf_meta_tags setup correctly.
Once the root_url is loaded in the browser, there is a javascript that fires a GET and a PUT, each one to its respective controller API in the application. By the moment those 2 API calls are fired, the session SHOULD have the _csrf_token there. And this is true, most of the times. Keep reading.
The problem is that, SOMETIMES, not always, we see some InvalidAuthenticityToken exceptions happening for the PUT request. (yes, I'm using protect_from_forgery :with => :exception on our API base controllers).
Analyzing the dump from exception_notification, I can see that the CSRF_TOKEN is setup correctly in the request header, but the most intriguing thing is that the session has only the session_id on it. Everything else is gone, including the _csrf_token.
Remember: this is happening intermittently! So I believe that it must be some kind of race condition.
This app is hosted on Heroku and running on Unicorn. I'm unable to reproduce the problem in my local environment. I've also read a lot of Rails code on github trying to understand the flows in which it resets the session, but I could not find the answer, since all CSRF protection is setup correctly and the problem happens intermittently.
It's also worth mentioning that we did not setup config.secret_key_base yet. But as this problem is happening intermittently, I don't think that this may be the root cause of it.
Also, I believe its worth mentioning that we have two controller hierarchies:
(1) all 'normal' application requests go through controllers which inherits from ApplicationController
(2) all API requests go through controllers which inherits from Api::BaseController, which inherits from ActionController::Base directly
I believe this controllers scheme is the most common for everybody...
The API endpoint for the GET request is rendering a json response. The API endpoint for the PUT request is returning head :ok.
Well, I would love if some Ruby on Rails expert could help on this.

Answer 1

Are you using the standard cookie-based session store? The cookie-based session store, last i looked, most definitely was subject to race-conditions, especially around AJAX requests -- and the race conditions were kind of an inherent part of the design of a cookie-based session store, with no real way to fix it.
This post from 2011 describes a cookie store race condition that also involves the authenticity token, and may be similar to yours. Their solution was turning off the CSRF protection, for certain actions anyway, which doesn't sound like a great solution to me.
This post from 2014 outlines race conditions with cookie session store, and suggests ActiveRecord or other server-side store as a solution. (As i write this, that URL is 404'ing, but available in google cache).
As you can see from the example, the session cookie is updated on every request, regardless of if the session was modified or not. Depending on when the response gets back to the client last, thats the cookie that will be used in the next call. For example, if in our previous example, if get_current_result’s response was slower than get_quiz, then our cookie would have the correct data and the next call to update_response would of work fine! So sometimes it will work and sometimes not all depending on the internet gods. This type of race condition is no fun to deal with.
The implications of this is that using cookie storage for sessions when you are doing multiple ajax call is just not safe. All information saved in the session might be wrong or nonexistent next time you check. So whats the solution?
...
A better solution would be to use a server side session store like active record or memcache. Doing so prevents the session data from being reliant on client side cookies. Session data no longer has to be passed between the client and the server which means no more potential race conditions when two ajax are simultaneously made!
I can't say for sure if you are running into that problem, but it would be worth a try to switch to the ActiveRecord cookie store, and see if your problem goes away.
Even the activerecord-based session store has at times in the past been subject to race conditions, I am not sure if the current implementation is, but they are at least conceivably solvable, whereas the race conditions in cookie-based store are fundamental.
The ActiveRecord store is probably subject to analagous race conditions to the cookie store actually -- IF you have more than one app process running (or a multi-threaded app server), so concurrent request handling is still possible, a very similar race condition is probably still possible. Although it should probably be even rarer than the race condition with the cookie store, and it is theoretically solvable, although perhaps with some domain-specific logic, unlike the race condition with the cookie store that is pretty much unsolvable if you are doing any async ajax.

Answer 2

For better format, write this as an answer, maybe it should be a comment though.
I met a similar problem, the root reason is my app called current_user before protect_from_forgery executed, this is current_user implementation in Devise:
def current_#{mapping}
#current_#{mapping} ||= warden.authenticate(scope: :#{mapping})
end
And Devise have a feature config.clean_up_csrf_token_on_authentication = true.
So the problem is, csrf token has been reset after current_user get invoked. because it calls warden.authenticate. Then when running protect_from_forgery. a csrf error has been raised. your session will be reset or an exception raised.
Hope this can help.