resque [Docs]

User Tools

Site Tools


resque

Resque

Resque is a Redis-backed library for creating background jobs, placing those jobs on one or more queues, and processing them later.

Features:

  • Workers can be distributed between multiple machines
  • Includes support for priorities (queues)
  • Resilient to memory leaks (forking)
  • Expects failure
  • Has the ability to track the status of jobs
  • Will mark a job as failed, if a forked child running a job does not exit with a status code as 0
  • Has built in support for setUp and tearDown methods, called pre and post jobs

Jobs

Queueing

Resque::enqueue('default', 'My_Job', $args);

Defining Job

class My_Job
{
 public function perform()
 {
	 // Work work work
	 echo $this->args['name'];
 }
}

Dequeueing Jobs

This method can be used to conveniently remove a job from a queue.

// Removes job class 'My_Job' of queue 'default' Resque::dequeue('default', ['My_Job']);

Tracking Job Statuses

php-resque has the ability to perform basic status tracking of a queued job. The status information will allow you to check if a job is in the queue, is currently being run, has finished, or has failed.

$token = Resque::enqueue('default', 'My_Job', $args, true); $status = new Resque_Job_Status($token);

Workers

$ QUEUE=file_serve php bin/resque

Priorities $ QUEUE=file_serve,warm_cache bin/resque

Running all queues $ QUEUE='*' bin/resque

Running multiple workers $ COUNT=5 bin/resque

Forking

Similarly to the Ruby versions, supported platforms will immediately fork after picking up a job. The forked child will exit as soon as the job finishes.

The difference with php-resque is that if a forked child does not exit nicely (PHP error or such), php-resque will automatically fail the job.

Signals

Signals also work on supported platforms exactly as in the Ruby version of Resque:

  • QUIT - Wait for job to finish processing then exit
  • TERM / INT - Immediately kill job then exit
  • USR1 - Immediately kill job but don't exit
  • USR2 - Pause worker, no new jobs will be processed
  • CONT - Resume worker.

Worker: PHP process that will run indefinitely, always monitoring for new jobs to execute. Pseudocode

while  (true)  { 
	$jobs = pullData();  # Pull jobs from the queues  
	foreach  ($jobs as $class => $args)  {  # For each jobs found 
		$job =  new $class(); $job->perform($args);  # Execute them  
	} 
	sleep(300);  # Then Sleep for 5 minutes (300 seconds), and retry  
}

Workers have to be started in the CLI. You can’t ceate worker through a browser, because :

  • You can’t do background jobs through a browser
  • PCNTL extension works only in CLI mode
QUEUE=notification VVERBOSE=1 php resque.php

nohup QUEUE=notification php resque.php & &: Daemon-ify the worker nohup: enables the commnad to keep running even if the user logged out

nohup QUEUE=notification php resque.php >> /path/to/your/logfile.log 2>&1 &
tail -F /path/to/your/logfile.log

Forking

when a Resque worker reserves a job it immediately forks a child process. The child processes the job then exits. When the child has exited successfully, the worker reserves another job and repeats the process. Resque assumes chaos => a job uses too much memory, a job delays

Thanks to Resque’s parent / child architecture, jobs that use too much memory release that memory upon completion. No unwanted growth.

With Resque’s parent / child architecture you can tell the parent to forcefully kill the child then immediately start processing more jobs.

Jobs

A job is a written order to tell the workers to execute a particular task.

Resque::enqueue('default', 'Mail',  array('dest@mail.com', 'hi!', 'this is a test content'));
// queue name, class name, arguments
// -------
class Mail  {  
	public  function perform()  {}
}

Locating the job

All your jobs classes must be discoverable by the workers. There’s more than one way to do that

Using APP_INCLUDE

QUEUE=default APP_INCLUDE=/path/file.php php resque

Track your jobs

All workers activities are logged. You’ll find in the logs:

  • When a job is enqueued with Resque::enqueue
  • When a worker detects a new job in the queue
  • When a worker begins executing a job
  • The status of the job execution (success/fail)

Depending on your verbose mode, they come with more or less details.

Bundle

Features:

  • Creating a Job, with container access in order to leverage your Symfony services
  • Enqueue a Job with parameters on a given queue
  • Creating background worker on a given queue
  • A UX to monitor your queues, workers and job statuses
  • ability to schedule jobs to run at a specific time or after a number of seconds delay
  • ability to auto re-queue failed jobs, with back-off strategies

TODOs:

  • Log management
  • Job status tracking
  • Redis configuration
  • Localisation
  • Tests

Running worker on a queueu

  • the default queue : app/console bcc:resque:worker-start default
  • the q1 and q2 queue : app/console bcc:resque:worker-start q1,q2

Supervisor

[program:myapp_phpresque_default]
command = /usr/bin/php /home/sites/myapp/prod/current/vendor/bcc/resque-bundle/BCC/ResqueBundle/bin/resque
user = myusername
environment = APP_INCLUDE='/home/sites/myapp/prod/current/vendor/autoload.php',VERBOSE='1',QUEUE='default'
stopsignal=QUIT
resque.txt · Last modified: 2020/07/02 16:06 (external edit)