Ruby Batch Processing: ActiveMQ and ActiveMessinging

Lets say you have some long running background tasks that are triggered by user actions. For example, you may want to allow users to upload a list of bookmark URLs that you will create thumbnails for.

Heres the sequence of events we are looking for.


user posts list -> rails controller -> create ticket(s) -> queue -> (offline) processor(s) do work


A message queue is a good solution for this type of setup if you want to have a number of processors that you can simply start more of to scale. ActiveMQ and ActiveMessaging using STOMP make a simple ruby/rails solution.

ActiveMQ
ActiveMessaging (A13g)

Create our processor

class ThumbnailProcessor < ApplicationProcessor
 # - using ActiveMQ STOMP extension prefetchSize 
 #   to only take 1 ticket at a time
 # - set ack to client or else prefetchSize won't 
 #   do any good
 subscribes_to :thumbnail, 
   { :activemq:prefetchSize=>1, :ack=>'client'}

 def on_message(msg)
   # stub for long running code that 
   # creates thumbnail from a url
   create_thumbnail(msg)
 end
end

Create our controller

class ThumbnailController < ApplicationController
 publishes_to :thumbnail

 def create
   # create a ticket for each url in list
   params[:urls].split.each do |url| 
     publish :thumbnail, url
   end
 end
end

Configure ActiveMessaging (config/messaging.rb)

ActiveMessaging::Gateway.define do |s|
 s.destination :thumbnail, '/queue/thumbnail'
end

Now spin up as many processors as you need (you can always start more later)

./script/poller start
./script/poller start
./script/poller start

and you are ready to roll!