Race conditions in Rails sessions and how to fix them

This article originally appeared upon on texperts.com

rails We’ve finally managed to track down and fix a bug in our system which has been bothering us for a while. It turns out to be related to a race condition in Ruby on Rails’ session management code. We would like to share our analysis of the problem, and our solution, with you.

Session management in Rails

Session management is one of the (many!) things which “just works” in Rails.

Until it doesn’t.

The session in a Rails app is a hash. In between actions, it’s stored persistently as a marshaled string. A number of different persistent stores are supported, but whichever you use, roughly speaking each action in a Rails controller does the following:

  1. Load the current session, or create a new one if necessary
  2. Run the code of the action
  3. Save the session, including any changes made while processing the action

What could possibly go wrong?

Concurrent requests

What could go wrong is concurrency. Imagine that the user makes two requests at the same time (let’s call them request1 and request2). What’s going to happen is something like this:

  1. The action servicing request1 (lets call it action1) loads the current session
  2. The action servicing request2 (lets call it action2) loads the current session
  3. Action1 makes some changes to its copy of the session
  4. Action2 makes some changes to its copy of the session
  5. Action2 completes and saves its changed session data
  6. Action1 completes and saves its changed session data, destroying the changes made by action2 in the process

Depending on the precise order in which things happen, maybe the changes made to the session while serving request1 “win”, maybe those made during request2. But whichever, if two requests are made concurrently with a single session, somebody’s data is going to be lost.

What makes this even worse is that Rails saves the session even if all the action ever does is read from it. So in our example above, action2′s changes to the session would be destroyed even if action1 made no changes to the session at all.

Is this really a problem?

Short answer: yes.

Longer answer: it depends on your application. In a “traditional” non-AJAX application although this kind of thing might happen in theory (if the user has two browser windows open on your application and refreshes them both at the same time, for example), in practice its not going to happen very often.

In an application making use of AJAX, on the other hand, it’s increasingly common for pages to be constructed from multiple requests, and for these requests to “overlap”. And as with most race conditions, things will probably work just fine most of the time – but occasionally they will break in difficult to understand, difficult to reproduce and difficult to debug ways.

An example

The following isn’t a very realistic example, but it’s simple and demonstrates the problem. Imagine that you have a controller containing the following:

def long
  session[:foo] = "bar"
  sleep 15
  render :nothing => true
end

def short
  session[:short] = 1
  render :nothing => true
end

def status
  render :text => "session[:short] = #{session[:short]}"
end

To see the problem, you will need to run this on a webserver which allows more than one action to be handled simultaneously (i.e. not webrick). A couple of mongrels will do just fine.

In one browser window, run the long action. Before it completes, run the short action in another browser window and then the status action. You should see that the :short key contains the value 1. Run the status action again after the long action has completed, however, and you’ll see that the :short key goes back to being empty.

So, what to do?

The first thing to note is that this kind of thing is a fundamental issue with concurrent actions. Nothing we do is going to help if two different actions want to make conflicting changes to the session. But that doesn’t mean that we can’t improve things considerably.

One improvement we could make would be to avoid saving the session if an action makes no changes to it. That would help, but we can do better. The solution we’ve chosen is to modify Rails’ session handling so that each action performs the following steps:

  1. Load the current session, or create a new one if necessary
  2. Save a copy of the unmodified session for future reference
  3. Run the code of the action
  4. Compare the modified session with the copy saved previously to determine what has changed
  5. If the session has changed:
    1. Lock the session
    2. Reload the session
    3. Apply the changes made to this session and save it
    4. Unlock the session

This approach means that two actions which modify different keys within the session hash won’t interfere with each other at all. Of course this doesn’t come for free; we’re doing quite a bit more work than the standard Rails code (although we should recover a little performance by not saving the session unless it’s changed). Depending on how much data you store in your session, this may or may not be acceptable. Speaking for ourselves, we’re prepared to trade a little performance off against correctness!

We’ve put together a plugin (based on SqlSessionStore) which implements these changes and is available from here. Please feel free to use it, and let us know how you get on!

Links

This problem is not unique to Rails. A discussion of the same problem and a (slighly different) PHP-based solution can be found here.

Credits

All of the work described above has been carried out by Frederick Cheung, as part of his work here at 82ASK.

17 Responses to “Race conditions in Rails sessions and how to fix them”


  1. 1 Jimmy P May 17, 2007 at 2:10 pm

    What about session creation? What if two async xhr calls are made at about the same time, where the user has no pre-existing session and the request handlers both try to create the session? I don’t use rails, but am facing that problem with java/tomcat. Could you run into the same problem with rails? If not, how does rails work around it?

  2. 2 fred May 21, 2007 at 10:28 am

    That’s not a case that we handle. Apart from anything else, in rails those 2 requests would be trying to create sessions with different session ids). I don’t think it’s likely though, most of the time those xhr requests will be triggered from some other page, loading of such a page would create the session (unless of course you had disabled session handling for all requests except those xhr ones)

  3. 3 Patrick DiLeonardo September 19, 2007 at 3:23 am

    I think I found an easily fixed weakness in this code as it pertains to actions that call session.update or session.restore multiple times.

    I’ve changed the code to address these issues and would be delighted to share it back to you for inclusion in your distribution.

    Thanks very much for providing this code and your very helpful writeup above.

    Please email if interested in more detail (test case that shows problem and code).

    Sincerely,

    Patrick M. DiLeonardo

  4. 4 Frederick Cheung September 19, 2007 at 7:05 am

    Awesome, I’d very much like to see it. I’ve sent you a mail

  5. 5 Björn Andreasson February 27, 2008 at 8:00 am

    Frederick Cheung and Patrick DiLeonardo: Is Patrick’s upgraded code implemented here http://svn1.hosted-projects.com/fcheung/smart_session_store/trunk/ ?

  6. 6 Frederick Cheung March 5, 2008 at 4:27 pm

    No it isn’t. What Patrick was trying to address updating the session multiple times from within a single action, which for us at least isn’t a valid use case.

  7. 7 Ralph March 26, 2008 at 12:49 pm

    This should be pushed in rails code as the default implementation, or at least default behaviour. Here is why:
    - session store should just work, it is the very basement of stateful web
    - race conditions are often awfully difficult to spot
    - a lot of web developers that are not engineer could just not cope with such problems

    Please push your work, or at least your ideas, into rails.

  8. 8 Paul Butcher March 26, 2008 at 1:06 pm

    We submitted this to the Rails Trac site when we originally wrote the article (over a year ago). Here’s the ticket:

    http://dev.rubyonrails.org/ticket/8256

    And also posted it to the Rails core mailing list:

    http://www.ruby-forum.com/topic/106919

    As you know, what makes it into the core and what doesn’t largely depends upon what does and what does not receive support from the community. As you can see, we didn’t get a great deal of response.

    I agree with you that it would be better if Rails core included this fix, but the trick is gaining the attention of other Rails developers.

    Can I suggest that if you feel strongly about this, it might be worth raising it on the Rails mailing list? If someone other than the original authors raises it, it will demonstrate wider support and other people may add their voice?

    PS – one additional point. The default session store in the current version of Rails is the cookie store, in which our fix is completely impossible :-(

  9. 9 SACK March 13, 2009 at 2:21 pm

    Concerning the change request made by Patrick DiLeonard:

    We ran into the same problem. Our application needs to update the session several times during a single request by calling

    session.update(). Unfortunatelly – as Frederick Cheung already wrote – this case isn’t considered by the SmartSessionStore.

    Thus it may happen that not all updates made to the session are stored to the database.

    An example:
    - We make a request and the session is acquired from database
    - Say there’s the key “Name” in the session data with the value “Max Mustermann”
    - Then we change the name to “Lisa Lustig” and call session.update()
    - smartSessionStore realizes that the value has changed and thus does an update to the database
    - Then our code does funky stuff and realizes that “Lisa Lustig” is not valid and thus reverts the name back to “Max

    Mustermann”. Again we call session.update().
    - SmartSessionStore compares the initial value of key “name” (which was “Max Mustermann”) with the new value (which is also

    “Max Mustermann”). Because the 2 values are equal SmartSessionStore thinks thtat there were no changes and doesn’t perform an

    update to the database. Although the actual value stored in the datatabse is “Lisa Lustig”.

    To fox this problem we implemented some changes in class smart_session_store.rb in method save_session:


    def save_session
    if @original_marshalized_data
    @original_data ||= unmarshalize @original_marshalized_data
    else
    @original_data = {}
    end
    @data ||= {}

    deleted_keys = @original_data.keys - @data.keys
    changed_keys = []
    @data.each {|k,v| changed_keys << k if Marshal.dump( @original_data[k]) != Marshal.dump( v)}

    return if changed_keys.empty? && deleted_keys.empty?

    SqlSession.transaction do
    fresh_session = @@session_class.find_session(@session.session_id, true)
    if fresh_session && fresh_session.data != @original_marshalized_data && fresh_data = unmarshalize(fresh_session.data)
    deleted_keys.each {|k| fresh_data.delete k}
    changed_keys.each {|k| fresh_data[k] = @data[k]}
    @data = fresh_data
    @session = fresh_session
    end
    # Bugfix BEGIN
    @original_marshalized_data = marshalize(@data) # NEW LINE
    @session.update_session(@original_marshalized_data) # ORIGINAL: @session.update_session(marshalize(@data))
    # Bugfix END
    end
    end

    The lines between “# Bugfix BEGIN” and “# Bugfix END” are the ones we changed. We simply update the instance variable

    @original_marshalized_data, so that the next time you call session.update() in the same request the method is capable of

    evaluation which keys have changed and which not and perform an update if necessary.

    I tested the code a bit and it seemed to work properly. I would appreciate if anyone who is more experienced with the

    SmartSessionStore can review this code a bit.

    Thankx.

  10. 10 Frederick Cheung March 13, 2009 at 2:26 pm

    Looks sane to me (although I personally don’t think of this as a valid use case)

  11. 11 SACK March 13, 2009 at 2:59 pm

    Thanks for the short review. The use case is indeed a bit weird. The actual use case in out application is as follows:
    - We have an ajax style application. Depending on the user’s actions there may be lots of concurrent requests.
    - Nonetheless there are some services which must be called synchronous. This of course must be ensured by the frontend in javascript. But for security reasons we also ensure this with special before and after filters on the server.
    The before filter sets a special flag in the session indicating that the service is currently running. The after filter removes this flag from the session.
    If the second service is called it first checks whether the first service is still running. If so we throw an exception.

    Maybe there are better techniques to implement such a lock.

  12. 12 SACK March 25, 2009 at 9:39 am

    The above code snippet solving the problem when the session is saved multiple times during a single request is wrong. So here’s the correct and tested version of it. See comments in code for details.


    require 'active_record'
    require 'cgi'
    require 'cgi/session'
    require 'base64'
    require 'pp'
    # +SmartSessionStore+ is a session store that strives to correctly handle session storage in the face of multiple
    # concurrent actions accessing the session. It is derived from Stephen Kaes' +SqlSessionStore+, a stripped down,
    # optimized for speed version of class +ActiveRecordStore+.

    class SmartSessionStore

    # The class to be used for creating, retrieving and updating sessions.
    # Defaults to SmartSessionStore::Session, which is derived from +ActiveRecord::Base+.
    #
    # In order to achieve acceptable performance you should implement
    # your own session class, similar to the one provided for Myqsl.
    #
    # Only functions +find_session+, +create_session+,
    # +update_session+ and +destroy+ are required. See file +mysql_session.rb+.

    cattr_accessor :session_class
    @@session_class = SqlSession

    # Create a new SmartSessionStore instance.
    #
    # +session+ is the session for which this instance is being created.
    #
    # +option+ is currently ignored as no options are recognized.

    def initialize(session, option=nil)
    if @session = @@session_class.find_session(session.session_id)
    # Is this really necessary? Because CGI:Session will call restore method anyway.
    self.data = unmarshalize(@session.data)
    else
    @session = @@session_class.create_session(session.session_id, marshalize({}))
    self.data = {}
    end
    end

    # Update the database and disassociate the session object
    def close
    if @session
    save_session
    @session = nil
    end
    end

    # Delete the current session, disassociate and destroy session object
    def delete
    if @session
    @session.destroy
    @session = nil
    end
    end

    # Restore session data from the session object.
    # The data hash returned by this restore method is stored in the CGI::Session and is used
    # by the application when session properties are set and get within
    # an ActionController::Base instance whenever the brackets operator is used.
    # e.g. session[:username] = "Peter".
    # This means that this session store must not create a new Hash
    # except here, because otherwise the CGI:Session and this session store
    # use different objects.
    def restore
    if @session
    self.data = unmarshalize(@session.data)
    end
    end

    # Save session data in the session object
    def update
    if @session
    save_session
    end
    end

    private

    def data= data
    @data = data
    if @session && @session.data
    @original_marshalized_data = @session.data
    else
    @original_marshalized_data = marshalize( {})
    end
    @data
    end

    def unmarshalize(data)
    Marshal.load(Base64.decode64(data))
    end

    def marshalize(data)
    Base64.encode64(Marshal.dump(data))
    end

    # Bugfix by Flexoptix was added in this method:
    # If save is called more than once in a single request (it is already called automatically
    # at the end of request processing by the rails framework and it can be called manually through
    # "session.update()", the session may not be stored into the database, because smart_session thinks
    # nothing has changed.
    def save_session
    # fo_logger = Log4r::Logger.new(self.class.name.to_s)
    # fo_logger.level = Log4r::DEBUG
    # fo_logger.add('fo_logger_output')

    if @original_marshalized_data
    # Bugfix by Flexoptix BEGIN
    # we have to get the information every time not only if @original_data is empty.
    # this is because @original_marshalized_data is update at the end of this method.
    # So if save_session is called more than once than this is necessary.
    # ORIGINAL @original_data ||= unmarshalize @original_marshalized_data
    @original_data = unmarshalize @original_marshalized_data
    # Bugfix by Flexoptix END
    else
    @original_data = {}
    end
    @data ||= {}

    # fo_logger.info("#{ENV['SERVER_IDENTIFICATION']} - session date before merge: #{@data.object_id} #{@data.to_a().join(', ')}")
    # fo_logger.info("#{ENV['SERVER_IDENTIFICATION']} - comparing with original: #{@original_data.object_id} #{@original_data.to_a().join(', ')}")

    # find out which keys have been deleted
    deleted_keys = @original_data.keys - @data.keys
    changed_keys = []

    # find out which keys have been change
    @data.each {|k,v| changed_keys << k if Marshal.dump( @original_data[k]) != Marshal.dump( v)}

    if changed_keys.empty? && deleted_keys.empty?
    # fo_logger.info("#{ENV['SERVER_IDENTIFICATION']} - no session update")
    return
    end

    SqlSession.transaction do
    fresh_session = @@session_class.find_session(@session.session_id, true)
    # Important for testing:
    # This code block is only execute when the session record in database has changed meanwhile
    # by another concurrent request. So the manual save problem (which has been fixed)
    # only popped up when the there were more than one instance of mongrel.
    if fresh_session && fresh_session.data != @original_marshalized_data && fresh_data = unmarshalize(fresh_session.data)
    # fo_logger.info("#{ENV['SERVER_IDENTIFICATION']} - current data: #{fresh_data.to_a().join(', ')}")
    deleted_keys.each {|k| fresh_data.delete k}
    changed_keys.each {|k| fresh_data[k] = @data[k]}

    # Bugfix by Flexoptix BEGIN
    # ORIGINAL @data = fresh_data
    # we must use the existing hash which was returned to the CGI::Session when
    # the restore method was called.
    # If we just would use the = operator the @data field
    # would point to a new Hash in memory and CGI::Session would not
    # know anything about it.
    @data.clear()
    @data.merge!(fresh_data)
    # Bugfix by Flexoptix END

    @session = fresh_session
    end
    # Bugfix by Flexoptix BEGIN
    # ORIGINAL: @session.update_session(@original_marshalized_data)
    @original_marshalized_data = marshalize(@data)
    @session.update_session(marshalize(@data)) # @original_marshalized_data
    # Bugfix by Flexoptix END

    # fo_logger.info("#{ENV['SERVER_IDENTIFICATION']} - session data after merge: #{@data.object_id} #{@data.to_a().join(', ')}")
    end

    end
    end

    __END__

    # This software is released under the MIT license
    # Copyright (c) 2007 Frederick Cheung
    # Copyright (c) 2005,2006 Stefan Kaes

    # Permission is hereby granted, free of charge, to any person obtaining
    # a copy of this software and associated documentation files (the
    # "Software"), to deal in the Software without restriction, including
    # without limitation the rights to use, copy, modify, merge, publish,
    # distribute, sublicense, and/or sell copies of the Software, and to
    # permit persons to whom the Software is furnished to do so, subject to
    # the following conditions:

    # The above copyright notice and this permission notice shall be
    # included in all copies or substantial portions of the Software.

    # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
    # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
    # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
    # NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
    # LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
    # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
    # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

  13. 13 Frederick Cheung March 25, 2009 at 9:53 am

    The code for this is also on github ( github.com/fcheung ), I’d encourage you to fork the plugin and post your changes there rather than putting them in a blog comment where they probably won’t get much visability.

  14. 14 Ben January 12, 2012 at 4:09 am

    I know this is an old post, but hoping for some updates as it’s still referenced around the web:
    1. Has this issue been resolved in any way in rails core? (I believe it hasn’t)
    2. If not, is there a current ticket number available?

  15. 15 paul January 12, 2012 at 7:37 pm

    I’m afraid I can’t be much help, Ben – I’m not aware of this issue being resolved in Rails, but I’ve not been seriously involved in Rails development for a couple of years now, so my knowledge is not as up to date as it might be. I don’t know if there’s a more up to date ticket.


  1. 1 Nodeta » Blog Archive » Avoiding Rails session race conditions - now with PostgreSQL Trackback on March 27, 2009 at 7:04 pm
  2. 2 Rails4, AngularJS, CSRF and Devise | technpol Trackback on April 17, 2014 at 1:41 pm

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





Follow

Get every new post delivered to your Inbox.

Join 188 other followers

%d bloggers like this: