HOW TO: Stop using wp-cron.php and do this instead...

WordPress cron (wp-cron) works really well. Until it doesn't. For a small single site blog with relatively low traffic, you don't need this. But as you scale, or run a Woocommerce store or multisite - this is absolutely worth setting up. There's a bit of reading here, feel free to skip right to the installation bit.

What's the problem with wp-cron.php?

First of all, WordPress cronjobs are not... well... cronjobs at all. Crons are jobs scheduled within Unix-type machines to schedule automated tasks. A daemon (ie. Cron.d) runs in the background to check for jobs that need to be done set by a server admin. These jobs could be anything, from ensuring another service like mysql is running to more obvious tasks like running backups, checking for plugin/theme/core updates or sending emails to subscribers.

Wp-cron was created to allow WordPress to manage a complex set of tasks without requiring the site admins to SSH into their machine and set crons manually.

In contrast to a Unix cron which is automated, wp-cron requires some manor of interaction with your PHP layer (ie. a pageload by a human or a bot) because there's no service (daemon) supervising it. That would truly suck, as a standard WordPress install can set 10's of jobs out of the box, scaling to 100's or 1,000's as you install more and more plugins like Woocommerce.

On top of this, many folks who use WordPress on a day-to-day basis aren't hugely comfortable with server admin tasks like adding or removing Unix crons.

With that said, wp-cron is actually a super clever solution but there are some major tradeoffs.

First, because there's no background system (daemon) running in the background to check for new tasks - wp-cron.php works by checking if there are overdue tasks on every pageload, so it requires traffic to function. Not so great if you need jobs processed at off-peak traffic times - especially very frequent ones (every 1-5 minutes).

Fun Fact: You can use a service like http://cron-job.org/ as way to help guarantee a hit to wp-cron.php every minute with a free account, ensuring that even without real traffic - WordPress crons will execute. You just need to set up a 60 second GET request to https://YOURDOMAIN.com/wp-cron.php. A cool trick to have up your sleeve in a pinch.

Second, wp-cron.php runs tasks sequentially. If one fails or simply takes too long (triggering a timeout or memory exhaustion) - the tasks sequentially later in the list won't get to execute in that cycle. What's even worse, because there's no daemon determining tasks, wp-cron.php requires that the PHP code executing a task completes and then resets itself in the cron event list to set itself up for the next cycle. If the task fails, due to a timeout or memory outage, the task will never get to set itself again and will disappear from the list.

You can see how this might not be ideal for a Woocommerce site with hundreds or thousands of order tasks to process or even just a site with scheduled posts (posts will be marked as "missed" schedule and just kind of... sit there, until an author or admin notices and manually publishes).

Fun Fact: WordPress "alternate cron" still requires site activity to function correctly. It was created as a work around for servers that don't allow loopback connections (which breaks "regular" WordPress crons). You can identify this is enabled by checking wp-config.php for this line:
define('ALTERNATE_WP_CRON', true);

Also, if you ever noticed the ?doing-cron query popping up in a redirect and didn't really know why - that's alternate cron doing it's thing (redirecting a user secretly to trigger wp-cron without the user really noticing... sneaky).

Using a Unix cron with WP CLI?

A really neat method for ensuring WordPress crons process is to create an actual cron on the server to run a WP CLI command that will process any WordPress tasks that are "due now" (that's WordPress-speak for late).

This method assumes you have WP-CLI installed of course (see the installation guide here), as well as the ability to add crons on your server (type crontab -e in your server terminal or simply check your hosting panel for a cron page if you're not self-hosted).

The cron looks something like this:

* * * * * cd /var/www/html/YOUR_SITE_ROOT && /usr/local/wp site list --due-now --allow-root

Every minute, the daemon will run a command within WordPress to process any overdue WordPress crons. Note the --allow-root flag at the end. You'll need that if the server cron is setup under root (which is not recommended because it's a potential major security problem), otherwise WP-CLI will throw an error. It's best this up with a non-root user.

In some circumstances, for several reasons (each more obscure and unpredictable than the last) wp-cron can skip right over a portion of crons for individual subsites. Here's another version of that specific to multisites:

* * * * * cd /var/www/html/YOUR_SITE_ROOT && /usr/local/wp site list --field=url --allow-root | xargs -i -n1 wp cron event run --due-now --url="{}" --allow-root

It uses xargs to extract each subsite URL and then run any overdue tasks on a subsite-by-subsite basis. Again, I've left --allow-root in there but you really should not set this cron as the root user.

"That's cool and all, but aren't we still susceptible to failing crons, and slow cron processing times because they're still executed sequentially?", I hear you ask.

Yes.

So let's get to the meat of why you're here. You want to ensure tasks are completed reliably, promptly and in parallel on your WordPress site because you're serious about your site. What's the solution?

Cavalcade: A better wp-cron

The team at HumanMade built Cavalcade out of necessity for their own projects. They tinkered with Gearman but it wasn't specific to WordPress so required lots of configuration and tweaking. They toyed with Action Scheduler, which is specifically for WordPress and works really well, though they saw some opportunities for improvement.

They commented "Unfortunately, it didn’t quite fit our use case, as it builds atop wp-cron (and inherits some of the downsides), it includes its own API, and it stores data in custom post types rather than finely tuned custom tables."

You can read more about how Cavalcade works here on the official Github repo. I won't delay any longer, sufficed to say this tool meets our criteria - reliable, prompt, parallel task processing for scale.

Installation:

Create an mu-plugins directory (if you don't already have one) and a cavalade plugin file as well as a directory containing the code from the repo.

The easiest way is by SSHing to your WordPress instance, cd to /wp-content/mu-plugins and running:

git clone https://github.com/humanmade/Cavalcade cavalcade

Create the plugin file that actually loads the code (while still in /wp-content/mu-plugins) by running:

echo "<?php require_once __DIR__ . '/cavalcade/plugin.php';" > cavalcade.php

Sweet! That's the plugin setup. Now we just need to add the "runner", which is a daemon that checks for jobs scheduled in WordPress.

Still within the /wp-content/mu-plugins directory run:

cd cavalcade && git clone https://github.com/humanmade/Cavalcade-Runner runner

Almost there...

Run echo $PATH in your terminal and copy the output, you'll need to paste that as the value for export PATH= in the script you're about to make.

Create a file called task_manager.sh in this same /wp-content/mu-plugins/cavalcade directory (or wherever you keep your bash scripts if there's a better place) and pop this code in there:

#!/usr/bin/env bash

# Paste the export path you just copied here in the quotes instead of mine

export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

# Put the absolute root for your site here
WORDPRESSROOT="/var/www/html/YOURSITEROOT/"

# Define the Cavalcade path.
cd ${WORDPRESSROOT}/wp-content/mu-plugins/cavalcade

# Check if Cavalcade is listed in ps.
ISCAVALCADEALIVE=$( ps -aux | grep -v grep | grep "mu-plugins/cavalcade/runner/bin/cavalcade" )

# Restart Cavalcade if it isn't listed, otherwise chill.
# Also, define a path for logging
if [[ -z "${ISCAVALCADEALIVE}" ]]; then
    echo "Cavalcade is not running, starting now..."
    /usr/bin/php ${WORDPRESSROOT}wp-content/mu-plugins/cavalcade/runner/bin/cavalcade ${WORDPRESSROOT}  > /var/logs/cron/cavalcade.log
    echo "All is well. Cavalcade is running."
fi

Don't forget to change the export PATH= value, the WORDPRESSROOT= value and the path to the logs at the bottom (if you want something different than /var/logs/cron/cavalcade.log)

Save that bad boy up and let's go finish this off by going full circle by adding a single Unix cron with crontab -e or cron.d to ensure this script is run regularly.

How regularly, I'll leave to you - but I like 60 seconds:

* * * * * /bin/bash /var/www/html/YOURSITEROOT/wp-content/mu-plugins/cavalcade/task_manager.sh > /tmp/cavalcadestart 2>&1

Credit to Mike at WP-Bullet for his tremendous help on simplifying the installation.

BONUS ROUND: A very not-recommended way to ensure a disappearing wp-cron stops disappearing with an mu-plugin

There may come a time when you find an essential wp-cron just vanishes with little to no explanation. This is a method to ensure a cron is reset in the "cron event list" if it's not detected on pageload.

Create an mu-plugin /wp-content/mu-plugins/revive-cron.php and pop this code in there (replace any instance of YOUR_CRON_NAME_HERE with the actual name of the wp-cron in question):

// Define a 60 second schedule
function add_cron_schedule_60($schedules){
    if(!isset($schedules["1min"])){
        $schedules["1min"] = array(
            'interval' => 1*60,
            'display' => __('Once every minute'));
    }
    return $schedules;
}
// Add 60 seconds to schedule list
add_filter('cron_schedules','add_cron_schedule_60');

// If this cron doesn't exist, spawn it
if ( ! wp_next_scheduled( 'YOUR_CRON_NAME_HERE' ) ) {
wp_schedule_event( time(), '1min', 'YOUR_CRON_NAME_HERE' );
// Don't forget to tell the error log
error_log( 'The YOUR_CRON_NAME_HERE cron has been revived!');
        }

The first 2 snippets define an interval for the wp-cron in a way that WordPress understands. In this case it's 60 seconds. The next snippet forces the wp-cron to spawn if it's disappeared. The last line logs the event to your PHP error log so you can review when the resuscitations occur - which is super helpful for troubleshooting the real underlying problem.

This is intended purely to buy you time to investigate the real reason for the disappearance and not be left in place long term as it performs this check on every pageload, and as such is a generally bad idea.