Multithreading PHP Function

Question

Currently when I execute this function with say 60 URL's I get a HTTP 504 error. Is there anyway to multithread this so that I no longer get a 504 error and iterate throughout the entire list of URL's?

    <?php

namespace App\Http\Controllers;

use Request;
use App\Http\Controllers\Controller;

class MainController extends Controller
{
    public function parse()
    {
        $input = Request::all();
        $csv = $input['laraCsv'];
        $new_csv = trim(preg_replace('/\s\s+/', ',', $csv));

        $headerInfo = [];
        //$titles = [];
        $csvArray = str_getcsv($new_csv, ",");
        $csvLength = count($csvArray);
        $i = 0;
        while ($i < $csvLength) {

            if(strpos($csvArray[$i], '.pdf') !== false) {
                print_r($csvArray[$i]);
            }
            else{
                array_push($headerInfo, get_headers($csvArray[$i], 1));
            }
            //sleep(3);
            //echo file_get_contents($csvArray[$i]);
            $i++;
        }
        return view('csvViewer')->with('data', $headerInfo)->with('urls', $csvArray);
    }
}

504 is usually caused by hammering the server. So just cut back a bit on the time between requests. — rjdown
– rjdown, Commented Dec 8, 2015 at 22:46

Dave Chen · Accepted Answer · 2015-12-08 23:07:09Z

I've used digitalocean in the past before but I'm not sure what error codes they give if you run out of time, (also set_time_limit(0); should already be in your code).

See if this works:

<?php

function getHeaders($data) {
  $curly = array();
  $result = array();

  $mh = curl_multi_init();

  foreach ($data as $id => $url) {
      $curly[$id] = curl_init();
      curl_setopt($curly[$id], CURLOPT_URL, $url);
      curl_setopt($curly[$id], CURLOPT_HEADER, true);
      curl_setopt($curly[$id], CURLOPT_NOBODY, true);
      curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, true);
      curl_multi_add_handle($mh, $curly[$id]);
  }

  $running = null;
  do {
      curl_multi_exec($mh, $running);
  } while ($running > 0);

  foreach($curly as $id => $c) {
      $result[$id] = array_filter(explode("\n", curl_multi_getcontent($c)));
      curl_multi_remove_handle($mh, $c);
  }

  curl_multi_close($mh);
  return $result;
}

$urls = array(
  'http://google.com',
  'http://yahoo.com',
  'http://doesnotexistwillitplease.com'
);

$r = getHeaders($urls);

echo '<pre>';
print_r($r);

So once you've gotten all your URLs into an array, run it like getHeaders($urls);.

If it doesn't work try it only with 3 or 4 urls first. Also set_time_limit(0); at the top as mentioned before.

thepiyush13 · Accepted Answer · 2015-12-08 22:41:54Z

0

Are you sure it is because of your code ? it could also be the server configuration.

about HTTP 504

This problem is entirely due to slow IP communication between back-end computers, possibly including the Web server. Only the people who set up the network at the site which hosts the Web server can fix this problem.

answered Dec 8, 2015 at 22:41

thepiyush13

1,3211 gold badge8 silver badges9 bronze badges

1 Comment

Brandon Over a year ago

The server is a 1gb digital ocean server that was created through laravel forge

Collectives™ on Stack Overflow

Multithreading PHP Function

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related