Memory Leak in ActiveMessaging

ActiveMessaging is great. It allows you to easily hook up to ActiveMQ to offload all your batch processing needs. Only problem, is it eats memory like crazy. Just hookup a simple queue with a publisher and consumer, write a few hundred thousand tickets, and watch the consumer eat all your available memory (will quickly eat a couple hundred megs and go on to use more than a GB).

In the gateway.rb, there is a dispatch method that routes the message to the appropriate processor:

def dispatch(message)
@@guard.synchronize {
begin
  prepare_application
  _dispatch(message)
rescue Object => exc
  ActiveMessaging.logger.error "Dispatch exception: #{exc}"
  ActiveMessaging.logger.error exc.backtrace.join("\n\t")
  raise exc
ensure
  reset_application
end
}
end

If you comment out the prepare_application and reset_application the memory consumption stops. You can chew through millions of tickets and stay at a steady usage. Only problem is that now, ActiveRecord will not keep its MySQL connection fresh, aka you will get a MySQL::Error: Mysql has gone away

These methods seem to wedge deep in rails' dispatch foo. Somewhere in there, it is likely doing validation on the connection. So, the trick will probably be to override the process!(message) method of the base processor class, and rescue MySQL::Error and call ActiveRecord::Base.verify_active_connections! and retry.

I will update this once I can validate it to see if this fixes the stale connection issue and if I run into any other issues, or if any kind commenter leaves the answer.