:

Almost every application needs background jobs for mailers, regular clean-ups, or any other time-consuming operation that doesn’t require a user to be present.

Several gems support job queues and background processing in the Rails world - Delayed Job and Sidekiq being the two most popular ones. In this post, we will take a detailed look at Delayed Job and Sidekiq, including how they fare against each other. Let’s go!

A Quick Introduction to Delayed Job

Delayed Job is a direct extraction from Shopify and uses a table to maintain all background jobs. It follows a very simple pattern. Any Ruby object that responds to a perform method can be enqueued in the jobs table.

In addition, if you don’t need to maintain special job objects (although this is highly recommended for testability and clear separation of long-running operations), it also allows you to call .delay.method(params) on any Ruby object. It will process the method in the background.

The Delayed Job README does a great job of explaining all common usage patterns.

Many teams choose Delayed Job because it is simple and uses their already existing database. They don’t need to spend/maintain other resources.

However, it will still take up space in your database table. If you have too many jobs queued at the same time, you might need more disk space to accommodate them all.

A Quick Introduction to Sidekiq

Sidekiq, on the other hand, uses Redis as its data store to maintain all job metadata. This comes with the obvious benefit of being much faster than the regular database systems Delayed Jobs uses. In addition to this, each Sidekiq process spawns multiple threads to process the jobs even faster.

For each background job in Sidekiq, we need a specialized class that includes the Sidekiq::Worker concern and responds to the perform method. To enqueue the job, we need to call perform_async(arg1, arg2) on the worker with the arguments.

The Sidekiq ‘Getting Started’ guide explains this and other usage patterns in good detail.

Using Active Job with Delayed Job or Sidekiq

Rails already provides a mature job framework for top-level declaration and handling of jobs. Both Delayed Job and Sidekiq support running jobs through ActiveJob’s unified API. Just inherit from ApplicationJob and call perform_later on your job class to enqueue the job to the configured queuing backend.

The advantage of running jobs with Active Job is that your application code becomes framework agnostic, and switching from Delayed Job to Sidekiq (or vice versa) becomes pretty easy. The ActiveJob::TestHelper also makes testing enqueued jobs a breeze.

But the abstraction provided by Active Job also comes with a performance overhead, as job data has to be wrapped before it’s pushed to the store. Sidekiq claims that ActiveJob is about 2-20x slower when pushing to Redis, with ~3x the processing overhead.

Delayed Jobs vs. Sidekiq

Now that we know the basics of Delayed Jobs and Sidekiq, let’s dive deeper into their differences and what each brings to the table.

The Features

For basic applications, both Sidekiq and Delayed Job provide a good set of features out of the box. These include assigning job priorities, named queues, and auto-retry on failures.

Delayed Job also provides a way to configure max run time out of the box (Sidekiq does not).

Sidekiq, on the other hand, provides support for Middleware to update job metadata, skip queuing a job, or execute a job. Delayed Job does not have an equivalent option (but if you use it with Active Job, you can use the before_enqueue and around_perform callbacks inbuilt into Rails).

Web UI is another feature that comes out of the box with Sidekiq. This provides historical statistics about jobs and information about workers, currently enqueued and dead jobs. You can perform operations like deleting or running jobs immediately without going through the console.

Delayed Job does not have an inbuilt Web UI, but delayed_job_web gives access to a basic Web UI with similar features to Sidekiq’s.

Sidekiq Wins at Performance

Performance-wise, Sidekiq beats Delayed Job quite convincingly. According to Sidekiq’s open-source benchmark, it is approximately 30x faster than Delayed Job. There are two major reasons for this:

  1. Redis is much faster at querying data than traditional databases like Postgres because it stores data in memory as opposed to the disk.
  2. Delayed Job runs a single thread to process jobs, compared to Sidekiq, which uses multiple threads.

While all of this looks great on paper, the differences do not matter much unless you work on a big scale (something like 10k jobs per minute). The exact number also depends on the average run time of a job. The longer the run time, the less the performance overhead of Delayed Job matters.

If you’re worried about the performance of Delayed Job, you can make some performance optimizations. The exact indexes to use will depend on the statistics of your job system. For example, if you use multiple queues and only one gets a major chunk of jobs, a simple index on the queue column (add_index :delayed_jobs, :queue) can significantly improve performance.

Deployment

Both Delayed Job and Sidekiq have a similar deployment strategy for workers. Using Heroku, you just need to add entries inside your Procfile to start the job processor and run the workers.

For Sidekiq:

worker: bundle exec sidekiq -t 25 -c ${SIDEKIQ_CONCURRENCY:-5} [-q name,priority [-q another_queue,another_priority]]

For Delayed Job:

worker: [QUEUE=x,y,z] bundle exec rails jobs:work

Memory

Here’s where things start to get a bit more interesting. Sidekiq has a concurrency option to control how many threads it runs. Most of the Sidekiq vs. Delayed Job benchmarks mention Sidekiq’s very high concurrency of up to 25 threads, which contributes to its super-fast performance.

But in a real setting, you have to limit the threads to something more conservative. The actual number depends on how heavy your application is and what kinds of jobs you perform. What I have seen in practice is that if you run a worker on 512MB memory (equivalent to standard-1x on Heroku), the number of threads is somewhere between 2 and 5 instead of 25.

‘Taming Rails memory bloat’ by Mike Perham, the creator of Sidekiq, discusses memory issues in more detail and is well worth a read. I won’t jump into the full discussion, but he recommends that you set MALLOC_ARENA_MAX=2 on all workers that run Sidekiq.

Using jemalloc instead of regular malloc helps too. The exact way to do this depends on the platform you use, but it is pretty simple on Heroku. Just set heroku-buildpack-jemalloc as the first buildpack (ahead of the heroku/ruby buildpack).

Delayed Job Uses Simpler Resources

As we discussed, Delayed Job runs on your existing database instance. You might need to increase:

  • available memory
  • disk space or
  • max connections

depending on the job load or the number of workers you run. But the only resource you need is the job processor.

On the other hand, Sidekiq requires a Redis instance to handle jobs. If you also use Redis as a cache store, it is recommended that you use a separate instance configured as a “persistent store” for Sidekiq jobs.

Since Redis works best when everything fits in memory, if you have too many jobs (for example, if Sidekiq stops processing them for some time due to an issue in the app), it might take some downtime to clear everything up. This is especially troublesome if you have Redis on the same server as your app. They will start competing for memory, leading to swapping and eventually destroying your app’s performance.

One important point to note about Redis is that it has to be configured with maxmemory-policy noeviction to avoid silent drops of Sidekiq’s data. Otherwise, you will find yourself missing jobs that need to be performed, without any trace.

A Side-Note: Paid Upgrades in Sidekiq

If you need extra features, Sidekiq comes with Pro and Enterprise versions.

The most notable addition in Pro is Batch Jobs that can run in parallel, be monitored, and interact as a group, invoking a callback when all jobs are done. Pro also has improved reliability features to ensure that no jobs are dropped silently, even during network problems.

The Enterprise version comes with yet more features. If you are looking for something that a regular Sidekiq installation can’t solve, explore the paid Sidekiq features.

In practice, the free version of Sidekiq still works great. But it is good to know that there are paid options to upgrade when needed instead of switching to a different solution.

Community and Development Status: Sidekiq Has the Edge

There is a huge community behind both Sidekiq and Delayed Job. However, it is not always easy to find quick answers to your questions in StackOverflow or the official documentation.

On the development side, things are not looking very bright for Delayed Job. It doesn’t seem like it is getting any major developments going forward. The last commit was more than a year ago, and there are 145 open issues on Github.

In contrast, Sidekiq is still under active development, and its creator is working full time on it. There are very few open issues, and they get addressed regularly.

Wrap-up: Sidekiq or Delayed Job? It Depends on Your Needs

In this post, we covered two major job processing systems for Rails applications - Sidekiq and Delayed Job - taking a look at some of their pros and cons.

There are different use cases for each. It all depends on the budget and scale of your operation.

If performance and long-term maintainability are of importance, Sidekiq is a no-brainer. On the other hand, if running costs are a concern, Delayed Job can help you there.

Whether you choose Delayed Job or Sidekiq, good luck with your project and happy coding!

This article was originally posted on AppSignal Blog