Symfony2 website with +100 users = 100% CPU charge

Question

I have a problem I can't solved with my server. I may miss a config somewhere, but I searched and find nothing fine.

Server hardware : Intel Atom N2800 2 cores, 4 threads (http://ark.intel.com/products/58917/) 4Gb RAM

It's running under Debian7 with NGinx, PHP5 FPM and MySQL.

There is only one website running. Average 50 visitors at the same time, peak up to 140 visitors.

This is what happen when a peak time :

top - 23:01:41 up 80 days,  8:40,  1 user,  load average: 19,78, 19,52, 19,37
Tasks: 166 total,  22 running, 144 sleeping,   0 stopped,   0 zombie
%Cpu(s): 90,6 us,  9,1 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,3 si,  0,0 st
KiB Mem:   4020300 total,  3232164 used,   788136 free,   362764 buffers
KiB Swap:   525308 total,   121404 used,   403904 free,  1191508 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
32295 www-data  20   0  247m  37m 6456 R  19,5  1,0   4:02.15 php5-fpm
32469 www-data  20   0  251m  40m 6592 R  19,5  1,0   3:30.46 php5-fpm
32689 www-data  20   0  243m  32m 6816 R  19,2  0,8   2:56.03 php5-fpm
32692 www-data  20   0  251m  41m 6564 R  19,2  1,0   2:57.84 php5-fpm
 1115 www-data  20   0  251m  40m 6428 R  18,9  1,0   0:54.01 php5-fpm
 1249 www-data  20   0  251m  40m 6356 R  18,9  1,0   0:42.35 php5-fpm
 1251 www-data  20   0  250m  39m 6420 R  18,9  1,0   0:37.36 php5-fpm
31314 www-data  20   0  251m  40m 6440 R  18,9  1,0   6:52.49 php5-fpm
32296 www-data  20   0  251m  40m 6560 R  18,9  1,0   4:03.63 php5-fpm
32410 www-data  20   0  251m  40m 6708 R  18,9  1,0   3:53.65 php5-fpm
32468 www-data  20   0  247m  36m 6720 R  18,9  0,9   3:27.14 php5-fpm
32471 www-data  20   0  251m  40m 6472 R  18,9  1,0   3:33.24 php5-fpm
32691 www-data  20   0  251m  40m 6584 R  18,9  1,0   3:03.90 php5-fpm
32294 www-data  20   0  251m  40m 6452 R  18,5  1,0   4:18.55 php5-fpm
32297 www-data  20   0  251m  40m 6456 R  18,5  1,0   4:07.43 php5-fpm
32467 www-data  20   0  246m  35m 6456 R  18,5  0,9   3:41.64 php5-fpm
 1250 www-data  20   0  241m  30m 6312 R  18,2  0,8   0:37.06 php5-fpm
32114 www-data  20   0  247m  37m 6428 R  18,2  0,9   4:38.27 php5-fpm
32470 www-data  20   0  243m  32m 6424 R  18,2  0,8   3:30.56 php5-fpm
32527 www-data  20   0  239m  28m 6428 R  18,2  0,7   3:19.96 php5-fpm
 1629 root      20   0 17228 5444 1308 S   2,9  0,1   0:01.63 rkhunter
23561 mysql     20   0  488m  73m 7944 S   1,3  1,9 227:34.60 mysqld
 4016 root      20   0 23740 1660 1136 R   0,7  0,0   0:00.07 top
20368 www-data  20   0 66340 4732 2116 S   0,3  0,1  10:34.08 nginx

You surely understand that I can't let it like that.

The website is built with Symfony2. The config file of symfony is available here : http://pastebin.com/CJbea0Qf

The phpinfo is there : https://drive.google.com/file/d/0ByNmwlJzaeKpNkNTcF9GZGNQVmM/edit?usp=sharing

This is the config of Nginx for the website (I found an example and modify it a little bit) Rewrite rules are the because of old addresses.

server {
    listen *:80;

    server_name ***.fr www.***.fr;

    root /home/www/clients/client1/web7/web/beta/web/;


    rewrite ^/index.php / permanent;
    rewrite ^/disclaimer.php / permanent;
    rewrite ^/disclaimer-mobile.php / permanent;
    rewrite ^/news.php /news permanent;
    rewrite ^/contact.php /contact permanent;

    rewrite ^/app\.php/?(.*)$ /$1 permanent;

    try_files $uri @rewriteapp;

    location @rewriteapp {
            rewrite ^(.*)$ /app.php/$1 last;
    }

    # Deny all . files
    location ~ /\. {
            deny all;
    }

            # Do some static file caching
            location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
                    log_not_found     off;
                    expires           1d;
            }

    location ~ ^/(app|app_dev)\.php(/|$) {
            fastcgi_split_path_info ^(.+\.php)(/.*)$;
            include fastcgi_params;
            fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_index app.php;
            send_timeout 1800;
            fastcgi_read_timeout 1800;
            fastcgi_buffer_size 128k;
            fastcgi_buffers 256 16k;
            fastcgi_busy_buffers_size 256k;
            fastcgi_pass unix:/var/run/php5-fpm.sock;
            #fastcgi_pass 127.0.0.1:9000;
    }

     location ~ ^/(mystatus|myping)$ {
            access_log off;
            include fastcgi_params;
            fastcgi_pass unix:/var/run/php5-fpm.sock;
    }


    # Statics
    location /(bundles|media) {
            access_log off;
            expires 30d;

            # Font files
            #if ($filename ~* ^.*?\.(eot)|(ttf)|(woff)$){
            #       add_header Access-Control-Allow-Origin *;
            #}

            try_files $uri @rewriteapp;
    }
}

Finally, PHP-FPM config.

pm = dynamic
pm.max_children = 20
pm.start_servers = 10
pm.min_spare_servers = 8
pm.max_spare_servers = 15
pm.max_requests = 400

I don't know if you need other data to help me solve that.

The website is quite "light". Not a lot of user interactions, just simple pages (it's mainly a gallery), so just pages generation.

Thanks a lot

ONE THING THAT MAY BE (or not) IMPORTANT When I reload php5-fpm, the load charge is quite calm (~2), but it amplifies with time until not going under 8.

Evolution I set up a cache with nginx fastcgi_cache parameters. Each page is cached for 5minutes. That will break up all "view" counters but I don't care as the website is clearly faster. I noticed that now, when I do a service php5-fpm reload, the server is going down to 1 in charge ! And it continues like that even with +100 visitors at the same time. But strangely, one time yesterday, it began to go up again, 20 on charge. I did reload php5-fpm and it all went down to 1 ! And it's working on 1 since yesterday. It seems that there some sort of trigger point that makes php-fpm going crazy !

"You surely understand that I can't let it like that." actully no - i know many shared hosts that operate under that sort of load 24\7. Less than 100% is actully waistfull. server load is not a relal world metric, it bears no resemblance to the users experience on the site — user557846
– user557846, Commented Nov 25, 2013 at 22:54
140 users simultaneously making requests is actually a huge amount of traffic. At any given time with that load, you might have 3,000 people on your site. Depending on how that goes, you're probably looking at around 5 to 10 million unique visitors per month. Consider monetizing your site so you can use something other than an Atom CPU in a server. — Brad
– Brad, Commented Nov 25, 2013 at 23:00
Just i was looking for be sure. You are using APC? This will improve the performance. — mcuadros
– mcuadros, Commented Nov 25, 2013 at 23:13
@mcuadros APC is enabled as the profiler in Symfony is telling me : Xdebug : disabled, PHP acceleration : enabled, XCache : disabled, APC : enabled, Zend OPcache : disabled, EAccelerator : disabled. I've made this as the doc recommends symfony.com/fr/doc/master/book/performance.html . — Link14
– Link14, Commented Nov 26, 2013 at 0:20
@byf-ferdy GG Analytics is showing ~6 sec response time on average. There is no significant big pages. And 6 sec is quite enormous. — Link14
– Link14, Commented Nov 26, 2013 at 12:18

Link14 · Accepted Answer · 2013-12-09 00:39:32Z

4

After one week with new settings, here is what I do to improve the performance of my server.

First, I used Nginx fastcgi cache. This clearly improve performance !! The cached pages are very very fast. I followed something like http://www.digitalocean.com/community/articles/how-to-setup-fastcgi-caching-with-nginx-on-your-vps or http://techminded.net/blog/transparent-cache-in-nginx.html It's easy to understand how it works with these. My cache is setted up with 5 minutes. It's too much for some pages, not enough for others. I must work on that point. But as this was PHP-FPM taking all resources, not executing php code has clearly lower the CPU charge.

Second, I set the APC cache size up to 128M And I think that was the biggest problem of my server config. Symfony use APC a lot and the APC cache was 100% fragmented ! I saw this with the apc stat page (got it from here as it was not present on my server : http://jesin.tk/apc-php-download/ ). Now the APC cache is 0.76% fragmented, 95% full, 100% cache hit ! The settings I changed : apc.shm_size 128M and apc.max_file_size 3M.

BTW, on the APC stats page, I can see 1728.50 cache requests/second. Of course, if your cache is bad, the performance will be bad. It's all clear now ! This should be explain somewhere in Symfony's documentation as it's a very important point.

The server is now really stable, CPU load about 2. I think I solved it !

answered Dec 9, 2013 at 0:39

Link14

9451 gold badge16 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Link14 Over a year ago

After several months, I saw that 128M of cache is not enough in my case, I set it up to 512M. The thing is, you never want your cache to be fragmented. If your cache is nearly full, it will get fragmented badly and it will decrease your performance.

gray · Accepted Answer · 2013-11-26 17:24:01Z

1

Seems like it is normal situation. Server not so powerful, and there is a lot of requests per second. To make sure that all ok, please provide page generation time, when site is not under load, and how many request does site gets per second (from access logs for example).

One of solutions to this high load problem is to enable Nginx cache. If you have a few pages, even 1 seconds page cache will be huge improvement. Nginx can serve thousands pages per second from cache.

answered Nov 26, 2013 at 17:24

gray

3812 silver badges14 bronze badges

1 Comment

Link14 Over a year ago

With this tail -f access.log | pv -l -i5 -r >/dev/null it's between 3 and 9 request per second. That doesn't seems to be a lot... I'm also working on fastcgi_cache with nginx. I just go a little problem that I posted there : stackoverflow.com/questions/20223974/…

ishenkoyv · Accepted Answer · 2013-11-26 18:14:54Z

-1

nginx configuration - you can limit number of connections from one ip address

limit_zone one $binary_remote_addr 4m; limit_conn one 4;
you can try to enable symfony 2 file cache or use something like varnish. If you are going to use symfony 2 cache you get more hard drive loading
you can try to use Doctrine Array Hydration

answered Nov 26, 2013 at 18:14

ishenkoyv

6734 silver badges9 bronze badges

3 Comments

Link14 Over a year ago

Are you sure about limitation of connections ? Because if there are more than 4 external elements in the page, they will not be loaded. Like css + js + images. I will take a look at Doctrine Array Hydration, but as I said somewhere in the comments, I've got results cached that don't even require a DB request. So I don't think this is the problem. The main problem is the CPU overcharge I think. And with the top command, we clearly see that php-fpm is taking too much resources.

ishenkoyv Over a year ago

Nginx config is some sort of DDOS defence. You can configure it according to your requirements. php-fpm cpu overload related to - building twig templates and if you are using a very deep template inheritance this can be a problem - doctrine hidration and entities managment - some app specific operations. As I understand you build some sort of gallery and there are a lot of image related work (like resize, watermarks etc) which you can move to background with something like german

Link14 Over a year ago

There is no image related work. Twig templates are ok I guess, there one template as base, one to generate page content (with 2 calls to a controller to include menus). Adding some caches with nginx are greatly helping, but I have a little problem for configuration as I mentioned here : stackoverflow.com/questions/20223974/…

Collectives™ on Stack Overflow

Symfony2 website with +100 users = 100% CPU charge

3 Answers 3

1 Comment

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related