Safely Update PHP Code Running Forever Loop For Processing Queue

Question

Let's say I have a PHP code that consumes from a message queue and run some class according to the message.

#/usr/bin/env php
//pseudo php code

while(true){
    $message = $queue->getMessage();
    $content = json_decode($message->content);
    switch($content->type){
        case 'a':
            runFunctionA($content->data);
            break;                
        case 'b':
            runFunctionB($content->data);
            break;
        case 'c':
            runFunctionC($content->data);
            break;
    }
    $message->delete();
}

This is my first time using a message queue. Since I'm already on top of AWS, I'm going to try using AWS SQS. I think it will be much simpler since I don't have to deploy some queue server from scratch.

My first problem is how to ensure that the process will run again if for example in the runFunctionA there is an error causing the script to halt. But after searching through StackOverflow, I found this thread 'Have a PHP script run forever, access a queue' and 'How can I keep an Amazon SQS PHP reciever script running forever?'. Basically I got the idea to create a cronjob to check a lock file. From my experience writing shell scripts, I can achieve this using start-stop-daemon like this. (1) Or is there a better or more common way?

But now I still can not figure out how to safely update the code, for example, when I want to update the content of runFunctionA. Currently I'm planning the queued tasks to be in the same codebase with the website because most of the business logic is already implemented there.

My current deployment mechanism is by storing the zipped code into the web-servers and change the link to new code. Something like below.

wget http://path/to/code.zip -O /path/to/temp/dir
rm /var/www
ln -s /path/to/temp/dir /var/www
#Let's say the forever loop is in script /var/www/queue_consume.php

Since the HTTP requests are short, I think deploying using this mechanism is pretty safe (and I think it's the most common way). But I'm not sure about updating long running PHP code.

My main question is (2) how to make the forever loop safely use the newly deployed runFunctionA code? Since I already have the idea using the start-stop-daemon above, I can basically restart the daemon. But again, (3) is there another more common way to do this. And lastly, (4) how to ensure the loop will at least finish running a task before the code being updated? My main concern is more about data consistency in the task. If the PHP script is restarted forcefully, there might be inconsistency in the database.

UPDATE

So far the only idea I have is to separate the loop and the tasks. In the loop I can do exec, for example,

 #/usr/bin/env php
//pseudo php code

while(true){
     $message = $queue->getMessage();
     $return = 1;
     $output = '';
     exec("/var/www/tasks.php {$message}", $output, $return);
     if ($return == 0)
          $message->delete();
}

As for the consistency, I'm thinking to use PHP Process Control to catch signal sent by the deployer and delay the script restart until the current task is processing.

are you not confident of using transactions for data inconsistency in the database? — Netorica
– Netorica, Commented Jan 27, 2014 at 2:41
Ah, I just realized that I can use transactions for that since I'm using MySQL. Nice idea, actually. — Petra Barus
– Petra Barus, Commented Jan 27, 2014 at 3:22
You can also connect SQS with SNS and make SNS ping an HTTP request to an endpoint each time there is a new message. It also provides exponential dropoff and I've used it quite sucessfully. I've also tried PHP daemons and they're quite unstable and leaky (PHP is not really meant to do that). Let me know if you need more info and I'll post an answer if so. — andreimarinescu
– andreimarinescu, Commented Jan 27, 2014 at 10:02
@andreimarinescu: Actually that's a really awesome idea. Although I already had a hunch that PHP is not suitable for this, I still feel intrigued with the PHP daemon way. Already planned to rewrite some of the business logic in Java. I think this question could apply to other than PHP. Nevertheless, I would love to see your answer about that, particularly about how you can manage securing your HTTP endpoint. — Petra Barus
– Petra Barus, Commented Jan 27, 2014 at 10:19
If you consider the SNS method, the AWS SDK for PHP has some helpers for parsing and validating the signature of SNS messages: github.com/aws/aws-sdk-php/blob/master/src/Aws/Sns/… — Jeremy Lindblom
– Jeremy Lindblom, Commented Jan 27, 2014 at 22:18

xzag · Accepted Answer · 2014-01-28 06:49:41Z

1

Maybe the following will help you. I coded this a while ago, but still use it in some projects.

PHP Process Management classes

Usage example (pm.php and sleep.php) - article is in russian, but you'll get the idea via google translate

These classes allow you to setup communication between primary script and tasks scripts; so you can implement some sort of restart functionality - task script will perform graceful stop when received restart signal, then primary script will start the task again

Feel free to ask question about the code.

answered Jan 28, 2014 at 6:49

xzag

6245 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

andreimarinescu · Accepted Answer · 2014-01-28 08:09:13Z

I've been searching for an AWS doc that details adding an SQS subscription to a SNS topic, unfortunately I can't seem to find it right now. The general architecture is that you subscribe both your HTTP endpoint and an SQS queue to a SNS topic. You can then publish a SNS message whenever you need to process a job, with the job details somehow serialised in the message body. When the HTTP endpoint receives a new SNS message, it can then poll the queue for a new message, retrieve the message details and process the job. It can then delete the message from the queue. In this way, I think you might even bypass the authentication of the SNS message, as you are only going to use the SNS callback to call your PHP script, and the job details are securely stored in SQS.

Regarding daemonizing PHP, I've look at this about 3-4 years ago. At that time this was a very leaky and unstable solution, things might have changed since then. What I went for at the time was creating a very very simple Python daemon (just forking the process basically) that called my PHP script via PHP CLI every X seconds. You can then even multithread this and start more than one PHP script or reduce the interval, depending on your scaling needs and your server power. One nice effect is that you don't need to worry about multithreading in PHP.

Hope this helps a little, let me know how I can provide additional info in the comments, if needed.

Collectives™ on Stack Overflow

Safely Update PHP Code Running Forever Loop For Processing Queue

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related