PHP: how to make the parent script wait for all child scripts to finish

Question

During each iteration of a foreach loop, I would like to initialize a background process by using the exec function to pass a json file to the worker.php script.

I want each process initialized by the exec function to run in parallel. I want the remaining statements after the foreach loop in the parent script to wait for the background child process, and execute only when all the background child processes have finished. How do I do it?

Here's my php code:

foreach($inputDataGroups as $inputsForTask){
    $json_string_input = json_encode($inputsForTask);
    $filename = 'ImportApp_input_data_'.($group_no + 1).".json";
    file_put_contents($filename,$json_string_input);
    exec('php worker.php "' . $filename . '"'.' ImportApp');
    $group_no++;
}
$dbappList = $this->getImportAppNameByLocationId($LocationID);
return $dbappList;

Gihan · Accepted Answer · 2020-04-23 09:36:49Z

1

You have to use asynchronous PHP to achieve this. https://github.com/spatie/async is a library that supports asynchronous.

use Spatie\Async\Pool;
$pool = Pool::create();

foreach($inputDataGroups as $inputsForTask){
    $json_string_input = json_encode($inputsForTask);
    $filename = 'ImportApp_input_data_'.($group_no + 1).".json";
    file_put_contents($filename,$json_string_input);
    $command = 'php worker.php "' . $filename . '"'.' ImportApp';

    // add to async pool
    $pool->add(function () use ($command){
        exec($command);
    });
    $group_no++;

}
$pool->wait(); // wait pool complets

$dbappList = $this->getImportAppNameByLocationId($LocationID);
return $dbappList;

I haven't tested this. But I hope it will put you in the right direction.

answered Apr 23, 2020 at 9:36

Gihan

4,2836 gold badges37 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ilovephp123456 Over a year ago

Thank you for the advice! May I ask is there any other way for not using any external library to achieve that?

SmasherHell Over a year ago

there is, using php.net/manual/en/function.pcntl-fork.php, but using libraries is so much simple, such as github.com/graze/parallel-process, I use it intensively

ilovephp123456 Over a year ago

@SmasherHell Thank you for the advice, would you mind to show a demonstration on how to use the pcntl-fork on my code?

Parsa Safavi Over a year ago

spatie/async will spin up child processes, with a lot of overhead. If you are using Laravel or anything with Dependency Injection, using that library means you will have to run another instance of your app (with dependencies) on each one of the processes.

SmasherHell · Accepted Answer · 2020-04-24 07:51:08Z

0

Using pcntl_fork, you can try something like that :

if (! function_exists('pcntl_fork')) {
  throw new \RuntimeException('PCNTL functions not available on this PHP installation');
}

$commands = [];
foreach($inputDataGroups as $inputsForTask){
  $json_string_input = json_encode($inputsForTask);
  $filename = 'ImportApp_input_data_'.($group_no + 1).".json";
  file_put_contents($filename,$json_string_input);
  $commands[] = 'php worker.php "' . $filename . '"'.' ImportApp';
  $group_no++;
}

// forking processes
$children = [];
foreach ($commands as $command) {
  switch($pid = pcntl_fork()) {
    case -1:
      throw new \Exception("Unable to fork process");
      break;
    case 0:
      // Child process
      exec($command, $output, $exitCode);
      // Child only need to execute command
      exit($exitCode);
    default:
      $children[] = $pid;
      break;
  }
}

// Waiting for children to finish
foreach ($children as $pid) {
  // We are still the parent.
  pcntl_waitpid($pid, $status);
  // Additional code handling none zero $status can be added here
}

I haven't tested this, but it is a good starter

Note : Beware when using forking, you might end up running too many processes at the same time and blow your memory or server away.

Implementing a maximum concurrency in parallel processing is not an easy task. I've made my own library to handle concurrency, as a result I spent hundreds of hour making it works, but was lacking crucial features, so ended up in trash when I found graze/parallel-process which implemented all feature we needed

edited Apr 24, 2020 at 7:51

answered Apr 24, 2020 at 6:39

SmasherHell

8849 silver badges20 bronze badges

2 Comments

ilovephp123456 Over a year ago

really thank you so much for your demonstration,I learnt from it a lot .may I ask where do you define $output and $exitCode and what are their contents?

SmasherHell Over a year ago

$output and $exitCode are reference variables (input/output) filled by exec function. $output contains the exec output (stdout) and $exitCode is the exit code of your worker (0 if it is successful).

Collectives™ on Stack Overflow

PHP: how to make the parent script wait for all child scripts to finish

2 Answers 2

4 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related