php workers or queue processor

Question

Background/problem description:

I'm not even sure how to search for what I want to do. I have a queue of items that need to be processed, and I want PHP to check every 10 seconds (or whatever increment) to see if there is anything in the queue. If there is, it should begin processing it, but only if it isn't already processing another item (so, when processing takes 30 seconds, for example, it wouldn't have more than one item being processed). I think I can hack an 'already processing' flag with the existence of a lock file or something.

I'm probably going to be using AWS SQS for the queue, and have multiple servers running in parallel

The question:

Basically, the user needs to queue up a bunch of work to be done asynchronously. So my question is, how do you schedule PHP to do queue checking? I was looking at cronjobs, but those seem to only be capable of running every minute.

What is different between every 10 seconds and every 1 minute if you have multiple servers working on this queue? For example, if you have 6 servers running a cron once per minute, you could check the queue approximately every 10 seconds. Have you considered use of SNS as a notification mechanism for SQS? How often do you expect items to be enqueued? What is SLA for how quickly items in queue must be processed? — Mike Brant
– Mike Brant, Commented Dec 4, 2013 at 1:01
Hmm, SNS looks promising. It might help make it more event-based rather than procedural looping. Could it be set up so that the queuing server notifies the processing servers (which 'subscribe' to that topic somehow) whenever something new is ready? Then each processing server could grab an item, process it, and check if there is anything left to do. — Reese
– Reese, Commented Dec 5, 2013 at 6:28
@rwilson04 You can simply use SNS to publish to an http(s) endpoint which could be an address with DNS to a load balancer in front of your services. With each notification, exactly one server (chosen based on LB configuration) would receive the message and be able to go pull the next item off the queue to process it. This way if you only get 1 message a minute, you only have one server running your script each minutes rather than a bunch of server regularly polling to see if queue has changed. This is more of a "push" paradigm as opposed to the "pull" approach you are considering. — Mike Brant
– Mike Brant, Commented Dec 5, 2013 at 17:02
If you're always going to have messages in your queue, then @richard's answer is a good one. If you won't have messages very often, then something like SNS could be good. Or something like IronMQ's alerts: dev.iron.io/mq/reference/queue_alerts are made just for this type of thing. — Travis Reeder
– Travis Reeder, Commented Mar 23, 2014 at 16:44

TRiG · Accepted Answer · 2017-06-23 13:05:49Z

2

I would start with a simple loop. Every 10 seconds, the loop checks the queue. If there is something there it processes it, taking whatever time is necessary. When the processing finishes, the loop continues in the 10 second loop.

edited Jun 23, 2017 at 13:05

TRiG

10.7k9 gold badges62 silver badges113 bronze badges

answered Dec 4, 2013 at 0:58

Richard - Rogue Wave Limited

3723 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Mike Brant Over a year ago

This is probably not a reliable approach. You are going to have a never ending script? What happens when process dies? How do you deal with concurrency across multiple systems?

Richard - Rogue Wave Limited Over a year ago

There was nothing about multiple systems in the problem description. The process should start at boot and continue forever.

Mike Brant Over a year ago

The original problem states: "I'm probably going to be using AWS SQS for the queue, and have multiple servers running in parallel". This suggests multiple systems, and thus a concurrency problem.

Travis Reeder Over a year ago

Mike, Richard is right, this is how it should be done, this is how almost all worker systems work. Any number of processes can ask for messages off a queue to do some work and since you have a bunch of them, you have redundancy which takes care of the "what happens when a process dies?" question. What's the concurrency problem here?

Collectives™ on Stack Overflow

php workers or queue processor

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related