Reliable Concurrency with Rails

Most of Rails applications will benefit from using multi-thread mode, it allows server to process multiple requests simultaneously. While one thread waiting for IO (http requests, database, disk, cache) other thread can do some processing.

I learned some lessons and want to share them with you.

Release DB Connection Before Sending Network Requests

Let’s say we use Puma server with 16 threads, but you don’t want it to keep 16 DB connections open. Probably our application send some requests to other services, and some day that services may become slow to response. This may increase number of borrowed db connections, and may cause application downtime. To protect ourself, we should return connection back to pool before starting IO-related task:

ActiveRecord::Base.connection.close  
# perform some IO waiting work, such as HTTP requests
user.update(foo: "bar") # will borrow new connection from pool  

Or we can borrow connection only when necessary:

ActiveRecord::Base.connection_pool.with_connection do  
  User.first.update(foo: "bar")
end  
# do something long and heavy

Monitor Database Pool Status

For last few years I was making /status pages on every application or service I’m working. I try to make it as rule for me and my team at my work. It serve just json (usually pretty formatted) with connection status to all other services and databases, also some internal information of app. We found it very useful, and google is doing something like this for years too. It may looks like this:

{
  "services": {
    "backend": { "status": "ok", "time": 0.004 },
    "queue_worker": {"status": "ok", "time": 0.909 }
  },
  "databases": {
    "postgres": { "status": "ok", "time": 0.002},
    "redis": { "status": "ok", "time": 0.001},
    "influx": { "status": "ok", "time": 0.012}
  },
  "app": {
    "puma_stats": {
      "backlog": 0,
      "running": 1,
      "min_threads": 0,
      "max_threads": 16
    },
    "dbpool_stats": {
      "size": 15,
      "connections": 5,
      "busy": 0,
      "dead": 0,
      "idle": 5,
      "waiting": 0,
      "checkout_timeout": 5
    },
    "memory": {
      "bytes": 297746432,
      "human": "283.953 MB"
    }
  },
  "version": "master-2017_04_11-11_23-40b6d7b8"
}

It’s useful for developers to see how application is doing, for DevOps to verify new VMs, for security engineers to make sure network policies not killing application, for monitoring team to confirm alarms. It literally saved us hours of debugging.

ActiveRecord stats provided by ActiveRecord::Base.connection_pool.stat (rails ≥ 5.1), or using this hack.

Puma stats:

if puma_instance = Puma::Launcher.running_instance  
  return JSON.parse(puma_instance.stats).merge(
    min_threads: puma_instance.options[:min_threads],
    max_threads: puma_instance.options[:max_threads]
  )
end  

Use Threads Carefully

Our application may need to use threads for parallel execution or running some code after sending response. We can do something like Thread.new { ... } but exceptions will be ignored and DB connection will not return back to pool. Safe way to handle it: copy thread variables (i18n, timezone, etc…), release db connections, report errors. It should look something like this:

parent_t = Thread.current  
Thread.new do  
  begin
    # copy thread variables
    Thread.current[:time_zone] = parent_t[:time_zone]
    Thread.current[:i18n_config] = parent_t[:i18n_config]
    ActiveRecord::Base.connection_pool.with_connection do
      yield
    end
  rescue => error
    # handle your error here
  end
end  

I made some helper module for this.

Find Un-returned Connections

If we do something with your database on boot, such as checking columns, tables, extensions, etc... then we should explicitly return connection. Using ActiveRecord::Base.connection.close or ActiveRecord::Base.connection_pool.with_connection {}. To find such places we can use connection callbacks on connection pool:

ActiveRecord::ConnectionAdapters::AbstractAdapter.set_callback :checkout, :after, -> (conn) {  
  puts "Connection acquired #{conn}"
  begin
    raise "Test error"
  rescue => error
    puts error.backtrace[1..-1]
  end
}
ActiveRecord::ConnectionAdapters::AbstractAdapter.set_callback :checkin, :after, -> (conn) {  
  puts "Connection returned #{conn}"
}

Configure NoSql and Cache Connection Pools

Most of database and cache server libraries provide connection pool features now. We need to set correct amount of maximum connections. I would recommend to make it little bigger then application number of threads. Puma server, with default settings, runs with 16 threads, then it will be safe to set max connection pool size to 17-20. (give couple more connections then app's pool because our code can have connection leaks)

Mongoid has connection pool feature, for redis and memcached we can use gem connection_pool.

Use Persistent HTTPS Connection

Making HTTPS requests takes longer time then just HTTP, it's because of SSL negotiation process (see benchmarks). But persistent connection for HTTPS isn't easy, every external service may have different Keep-Alive settings.

If we sending only GET requests (only fetching data, not making changes), then if we get Closed socket exception, it will be safe just re-open connection and send request. If our request making changes and we got network error, then we don't know if remote server received our request or not. In this situation sending request again may cause double changes. For example send SMS to customer twice.

We can use connection pool pattern for HTTPS connections too. I think it will be safe to assume that all servers will keep connection at least for 5 seconds. And for HTTP/2 we can use ping command to keep connection open.

Author: Pavel Evstigneev

Artikel - Artikel Terkait