Doctrine ORM out of memory, PHP symfony

Question

i am running below script to select list of users from my DB and pass it over to my service.

    $qb = $this->getEntityManager()->createQueryBuilder();
    $qb
        -> select('s')
        -> from('AppBundle\Entity\SubscriberDetails', 's')
        -> where($qb->expr()->not($qb->expr()->exists($qb1->getDQL())))
        -> setMaxResults($numcampaigns)
    ;
    $subscribers = $qb->getQuery()->getResult();

WHERE

    $qb1 = $this->getEntityManager()->createQueryBuilder();
    $qb1
        ->select('send')
        ->from('AppBundle\Entity\Subscribers', 'send')
        ->where('DATE_FORMAT(now(), \'%e-%b-%Y\') - DATE_FORMAT(FROM_UNIXTIME(send.last_campaign), \'%e-%b-%Y\') <=360')
        ->andwhere('send.bounced <> 1')
        ->andwhere('send.complaint <> 1')
        ->andwhere('s.emailaddress = send.emailaddress');

Below scripts works for 30k users while i am trying to expand it for 300k users selection. When running 300k users i get below error:

6.0662  132446208  15. Doctrine\ORM\Internal\Hydration\ObjectHydrator->hydrateRowData()/Applications/MAMP/htdocs/mediaff/vendor/doctrine/orm/lib/Doctrine/ORM/Internal/Hydration/ObjectHydrator.php:165
6.0663  132448496  16. Doctrine\ORM\Internal\Hydration\ObjectHydrator->getEntity()/Applications/MAMP/htdocs/mediaff/vendor/doctrine/orm/lib/Doctrine/ORM/Internal/Hydration/ObjectHydrator.php:494
6.0663  132448496  17. Doctrine\ORM\UnitOfWork->createEntity() /Applications/MAMP/htdocs/mediaff/vendor/doctrine/orm/lib/Doctrine/ORM/Internal/Hydration/ObjectHydrator.php:273
6.0663  132449488  18. ReflectionProperty->setValue() /Applications/MAMP/htdocs/mediaff/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:2586
6.0781  132445688  19. Symfony\Component\Debug\ErrorHandler::handleFatalError()/Applications/MAMP/htdocs/mediaff/vendor/symfony/symfony/src/Symfony/Component/Debug/ErrorHandler.php:0
6.0782  132434104  20. spl_autoload_call() /Applications/MAMP/htdocs/mediaff/vendor/symfony/symfony/src/Symfony/Component/Debug/ErrorHandler.php:613
6.0782  132434264  21. Symfony\Component\Debug\DebugClassLoader->loadClass()/Applications/MAMP/htdocs/mediaff/vendor/symfony/symfony/src/Symfony/Component/Debug/ErrorHandler.php:613

where second column is memory used i believe.

Any idea how can i overcome this issue. Any help will be highly appreciated.

If you don't want a "just increase your memory_limit" kind of answer, let's start from "why do you need to select 300k users in the first place?" — Federkun
– Federkun, Commented Jun 23, 2017 at 12:19
Definitely not a "memory increase" guy! Those are selected for further processing, eg. mapping in additional details, sending processed information to partner via API, receiving and parsing responses etc. — Sky21.86
– Sky21.86, Commented Jun 23, 2017 at 13:14
ORMs (and even php) are not designed for large scale batch operations. There are several dozen other sof questions on the exact same subject. You can do a bit with $em->clear() and a few other tricks but actually processing 300k users in one gulp? Not going to happen. Surprised you even got 30k to work. — Cerad
– Cerad, Commented Jun 23, 2017 at 13:30
As @Cerad said, 300k users at once is probably not going to happen. Assuming 128mb memory limit that leaves approx. 447 bytes per user. You can take quite a chunk off that due to several overheads (frameworks like doctrine, symfony (eg. your controllers and services) etc). So there's really not much left for your actual data. Factor in some 20-50 bytes for username/firstname/lastname per user, related entities like adresses (which can be 100 bytes aswell) and you're just not going to make it with that memory. — Kep
– Kep, Commented Jun 23, 2017 at 15:09
Batch processing, yea - Doctrine actually has an article about it. See the last part in particular. — Kep
– Kep, Commented Jun 23, 2017 at 18:14

ste · Accepted Answer · 2017-06-26 15:02:35Z

2

If your kind of data processing allow it, you could use a paginator and iterate over the pages, calling em->clear() each iteration, so that you mantein constant the memory consumption.

answered Jun 26, 2017 at 15:02

ste

1,57914 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sky21.86 Over a year ago

that is what i ended up doing! Plus added em ->getConnection()->close() after each iteration as my DB started crashing due to "to many connections"

Collectives™ on Stack Overflow

Doctrine ORM out of memory, PHP symfony

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related