Drupal 7 queues using beanstalkd

Queues are a great way to process large amounts of data without holding a users browser hostage, and even better if you can process that data in real-time using a queue worker like beanstalkd instead of cron, which Drupal uses by default.

Installing Dependencies

Before we can start using beanstalkd for our queues, we first need to install a few dependencies

• beanstalkd
• beanstalkd Drupal module
• pheanstalk

beanstalkd is a system package, so installation will depend on which system you’re on. Using ubuntu, the command is

sudo apt-get install beanstalkd

Note that the ‘d’ on the end is required. There is also a ‘beanstalk’ package in Ubuntu which is not what we need.

Once installed, make sure it’s running

sudo service beanstalkd start

Installing the drupal module is straight forward with Drush

drush en -y beanstalkd-7.x-2.x-dev

We’re using the 7.2 dev branch here because I was not able to get the 7.1 branch working due to it needing old versions of libraries.

At the time of writing, the 7.2 branch also required the composer-manager module for installing pheanstalk. I had a few problems getting this all working, but once I got everything installed my beanstalkd module directory looking like this

Once all of that’s installed, it’s time to move on to the module.

The Module

There are three main functions that your module needs

1. A function to queue an item
2. A function to claim an item
3. A hook to tell Drupal how to handle queued items

Your item queueing function needs to do two things, first is to make sure that the queue exists and if it does not, create it

// Get a queue (of the default type) called 'mycustom_queue'.
$queue = DrupalQueue::get('mycustom_queue');
// There is no harm in trying to recreate existing.
$queue->createQueue();

And then we can add an item to the queue. For example, you could add a node ID and title

// Queue the item.
$queue->createItem(array(
    'title' => $node->title,
    'nid' => $node->nid,
));

You could also store the entire node in the queue if you wanted. When the item comes off of the queue, Drupal would convert it back to a node object automatically.

And that’s all we need to do. Here’s the full code

function {module name}_queue_item($node) {
    // Get a queue called 'mycustom_queue'.
    $queue = DrupalQueue::get('mycustom_queue');
    // There is no harm in trying to recreate existing.
    $queue->createQueue();

    // Queue the string.
    $queue->createItem(array(
        'company' => $node->title,
        'nid' => $node->nid,
    ));
}

And to add an item to the queue, all you need to do is call the function from somewhere else

{module name}_queue_item($node);

Now you can create a function to claim the item

function {module name}_claim_phonetic_item($item_data) {
    // $item_data['nid']
    // $item_data['title']
}

If there are errors thrown within the claiming function, the item will be placed back on the queue for reprocessing, otherwise it will be pulled off the queue and then the next item will be processed.

Now we use hook_cron_queue_info to tell Drupal how to process an item

function {module name}_cron_queue_info() {
    $queues = array();
    $queues['mycustom_queue'] = array(
        'worker callback' => '{module name}_claim_item',
    );
    return $queues;
}

Then all that’s left is getting up and running with the queue itself.

The Queue

Now that we have the framework laid out we can tell Drupal to use beanstalkd for our queue, and then finally run the queue.

In sites/default/settings.php add the following line

$conf['queue_class_custom_queue'] = 'BeanstalkdQueue';

This will tell Drupal that the “custom_queue” queue should be processed by the BeanstalkdQueue class from the beanstalkd module.

To run the queue itself, we’ll use the runqueue.sh file included with the beanstalkd module

/path/to/drupal/sites/all/modules/beanstalkd/runqueue.sh -q=custom_queue -r=/path/to/drupal -s=all

The -q argument specifies which queue we’re interested in processing, the -r argument tells the script where the root of the Drupal install is located, and the -s argument is the site (i.e. /sites/all, /sites/default, etc).

Once you confirm the queue is working, you can set that up to run in a screen or supervisor so that it’s always running.

Wrap Up

Getting set up with beanstalkd queues isn’t very intuitive, but then the default DB queue isn’t either. But once you upgrade to real-time queue I don’t think you’ll ever want to go back.