Queuing PHP processes

Question

I've got the following two files in my server

requests.php (controller) - receives incoming requests and executes process_query.php in the background.
process_query.php - contains a script which calls a C executable (using shell_exec) to run against a certain number of files and writes some information to the database.

Right now as soon as the server receives a request in the controller, it immediately triggers the process_query.php script in the background. If there are too many incoming requests, it may result in server CPU outage. So I planned to have a max. limit to number of process_query.php processes that can run at a particular instant. For an example, let's assume the max. limit is 4 which implies only 4 instances of process_query.php can run at a time and others cannot start until one of the 4 processes finish.

I tried implementing this by having a table with request_id's and a flag to indicate process completion status. If there are 4 requests running right now then the table will have 4 entries with status flag as 0 in each.

When a new request arrives the process_query.php script runs into an infinite loop as shown in the code below and checks in the database for the total number of processes running at the moment. Since it is 4 as per our scenario, the script goes to sleep state (eg.10s). The loop keeps executing for every 10s until one of the process is completed. When the condition is met, the script comes out of the loop and performs the C call. After implementing this, I expected my activity monitor to show only 4 processes running. Unlikely there were n (total number of requests) processes running as the infinite loop kept the process_query.php process ticking. It made sense to me.

process_query.php

while(1) {
    $req_count = execute("SELECT count(status) FROM `requests` WHERE status=0");
    if($req_count == $max_request_limit) {
        sleep(10);
        continue;
    } 
    break;
}

// ...code to execute the C script

// end of process_query.php

Which approach should I adapt in order to achieve this?

anotherdev · Accepted Answer · 2016-01-01 01:17:22Z

2

You should use a job manager like gearman.

Explanation:

there is a job server, with a list of the jobs to do
there are workers, which are programms registered to that server in order to treat a job
there are clients, which are the programms that requested a job to be done with some parameters.

Imagine that you have a job server, you will have to write:

a worker: myworker.php in which you call process_query.php, and you register to the job 'process_query'
a client: in requests.php, you do a call to launch the job 'process_query' with some parameters

Then you will launch myworker.php 5 times (in background), and so you have maximum 5 jobs in process, the others will wait until one worker is available.

You can either launch a job from requests.php in an asynchronous way, either with a synchronous way (to wait until the job ends).

answered Jan 1, 2016 at 1:17

anotherdev

2,5671 gold badge16 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

DhiwaTdG Over a year ago

Thanks for letting me know about gearman. I wish to implement it on a long term basis as I do not sufficient privileges on the server to install it right now. Is there a PHP centric solution without installing any external packages? bcoz I need to make it up and running in next two days or so.

anotherdev Over a year ago

The problem in your sql solution is that there is no guarantee that $max_request_limit will be == to req count. You have to use >= but you may have lots of requests. Another way would be to use "file mutex", there is an implementation here code.google.com/p/mutex-for-php , it allows you to create a file and when you finish, it is removed and then the second process now create the file and execute its code, etc..

Collectives™ on Stack Overflow

Queuing PHP processes

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related