Blog Archives

Handling race conditions with Rails’ pessimistic locking

Posted in Code, Ruby on Rails, Tips and Tricks

I recently worked on a Rails application where race conditions were causing issues in the background tasks operating on my ActiveRecord objects. In this article I will explain how I used pessimistic locking to get around these issues.

Defining an example

Firstly, consider a sample Rails application with the following features/rules:

  1. Administrator can see a list of clients;
  2. Administrator can visit the client’s profile page and send an SMS to the client;
  3. Administrator cannot send more than one SMS to each client per day;

Let’s keep the implementation as simple as possible. Consider a Client model with an attribute last_sms_sent_on and the following class (which can be used by a controller or a Sidekiq task):

class ClientSMSSender
  def self.perform(client, message)
    client.transaction do
      if client.last_sms_sent_on.blank? || !client.last_sms_sent_on.today?
        client.last_sms_sent_on = Date.today
        client.save
        SMSGateway.send(client.phone_number, message)
      end
    end
  end
end

Analysing the race condition issue

Imagine that there are some external issues and SMSGateway.send hangs for an average of 30 seconds. In this meantime another administrator makes a new request to send an SMS to the client. Due to race conditions we’ll end up sending more than one message to the client on the same day.

This is what will happen:

  1. Administrator A makes a request (Request A) to send an SMS to the client
  2. Client has not received any messages today
  3. Request A is hanging due to external issues
  4. Administrator B make a new request (Request B) to send the same SMS to the client
  5. Client has not received any messages today (the Request A still hanging)
  6. Request B is hanging due to external issues as well
  7. Request A finishes and client.last_sms_sent_on is updated
  8. Request B still holding the previous state of the client object
  9. Request B finishes, send the SMS again and re-update client.last_sms_sent_on

Work around

In order to work around that issue, you can make use of the method with_lock from ActiveRecord::Locking::Pessimistic.

Have a look at the code below:

class ClientSMSSender
  def self.perform(client, message)
    client.with_lock do
      if client.last_sms_sent_on.blank? || !client.last_sms_sent_on.today?
        client.last_sms_sent_on = Date.today
        client.save
        SMSGateway.send(client.phone_number, message)
      end
    end
  end
end

Under the hood, with_lock:

  1. opens up a database transaction
  2. reloads the record (in order to obtain the last state of the record)
  3. requests exclusive access to the record from the database

When using with_lock: 

  1. Administrator A makes a request (Request A) to send an SMS to the client
  2. Request A locks the client record in database
  3. Client has not received any messages today
  4. Request A is hanging due to external issues
  5. Administrator B make a new request (Request B) to send the same SMS to the client
  6. As the client is currently locked by Request A, the Request B hangs until the database releases the record
  7. Request A finishes and client.last_sms_sent_on is updated
  8. Database releases the client record
  9. Request B (was hanging and waiting for database) now starts the execution
  10. Request B locks the client record
  11. Request B reloads the client record in order to obtain the latest state of the object
  12. Client has already received a message today
  13. Request B finishes without sending a new SMS.

If you’re still confused about that, here’s an easy way to see how with_lock works from your Rails console:

  1. Grab any existing Rails project and open up two Rails consoles
  2. In the first console execute the following:
    u = User.first
    u.with_lock do
      u.email = "test@thefrontiergroup.com.au"
      u.save
      sleep 40 # emulates a very slow external process
    end
    
  3. In the second console execute:
    u = User.first
    u.email = "test2@thefrontiergroup.com.au"
    u.save
    

You’ll notice in the second console that the execution of u.save will hang until the first console finishes the whole process.  You should be careful not to lock your entire app unnecessarily, otherwise you’re likely to introduce a new bottleneck.

Conclusion

The method with_lock is handy, but use it sparingly. Sticking with_lock everywhere might bring you good business logic consistency, but it can come at the expense of performance.

Search Posts

Featured Posts

Categories

Archives

View more archives