Thursday, July 24, 2008

Ruby Batch Processing: ActiveMQ and ActiveMessinging

Lets say you have some long running background tasks that are triggered by user actions. For example, you may want to allow users to upload a list of bookmark URLs that you will create thumbnails for.

Heres the sequence of events we are looking for.

user posts list -> rails controller -> create ticket(s) -> queue -> (offline) processor(s) do work

A message queue is a good solution for this type of setup if you want to have a number of processors that you can simply start more of to scale. ActiveMQ and ActiveMessaging using STOMP make a simple ruby/rails solution.

ActiveMessaging (A13g)

Create our processor

class ThumbnailProcessor < ApplicationProcessor
# - using ActiveMQ STOMP extension prefetchSize
# to only take 1 ticket at a time
# - set ack to client or else prefetchSize won't
# do any good
subscribes_to :thumbnail,
{ :activemq:prefetchSize=>1, :ack=>'client'}

def on_message(msg)
# stub for long running code that
# creates thumbnail from a url

Create our controller

class ThumbnailController < ApplicationController
publishes_to :thumbnail

def create
# create a ticket for each url in list
params[:urls].split.each do |url|
publish :thumbnail, url

Configure ActiveMessaging (config/messaging.rb)

ActiveMessaging::Gateway.define do |s|
s.destination :thumbnail, '/queue/thumbnail'

Now spin up as many processors as you need (you can always start more later)

./script/poller start
./script/poller start
./script/poller start

and you are ready to roll!