large php script slowing down the rest of application.... why?

Question

I have a large php web scraping script that logs the results onto a mysql database as it goes.The script generally runs for 5 to 10 minutes at a time.

The problem is that when this script is running other pages on the application will not load.

The script is on a dedicated server with plenty of RAM so I have tried increasing the allowed memory usage for MYSQL and PHP. Also increased the max allowed connections. None of this has helped.

Does anyone have any ideas about what else I can try?

Is your network connection saturated? Is your CPU maxed out? — Seventoes
– Seventoes, Commented Nov 1, 2014 at 18:55
InnoDB or MyISAM engine on your DB? Row level locks v.s. table level locks. huge difference. — Ohgodwhy
– Ohgodwhy, Commented Nov 1, 2014 at 18:57
- I'm not sure if my CPU is maxed out, how would I go about finding this out? — Jamie Deakin
– Jamie Deakin, Commented Nov 1, 2014 at 18:58

Maxim Dvorkin · Accepted Answer · 2014-11-01 19:10:43Z

2

Probably, problem in your session. Try to use session_write_close() before you start "big script".

answered Nov 1, 2014 at 19:10

Maxim Dvorkin

911 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jamie G Over a year ago

Thanks @maxim, this is a really useful thing to understand - it was relevant for me when running multiple php scripts from ajax calls. They were queuing up, now they're running concurrently.

Stefan Woehrer · Accepted Answer · 2014-11-01 19:04:36Z

0

well, there's a big difference between "slowing down" and "not load"!

try the following:

build a static html site and check if it is loaded well during the execution of the big script
build a php site that doesn't connect to the DB (just echo something) during the execution of the big script
build a small php site that connects to the DB (just select something from a table)

if 1 or 2 doesn't work well, your problem has something todo with the web server or server resources. if 3 doesn't work well, there could be resource issues with the mysql server.

if everything works well, check the scraping script. does it lock any table that is needed by the main application?

answered Nov 1, 2014 at 19:04

Stefan Woehrer

6904 silver badges12 bronze badges

1 Comment

Ohgodwhy Over a year ago

Your last question was answered in the comments, as the Engine is InnoDB, this means the engine only performs row level locks, as opposed to table level locks that occur in MyISAM.

Collectives™ on Stack Overflow

large php script slowing down the rest of application.... why?

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related