Architecture of large data in PHP

Question

I have a site where a user can serach hotel rooms in all over the world.
I make a request to some server and retrieve xml response.
I parse every xml and print hotel, image and combinations of room (because if someone search a single room and a dobule room and a triple room I have more combinations).
Well, the problem is that I can retrieve many and many hotel and every hotel can have more type of room like 5 for each hotel (with 5 kind of room I can have 7 / 8 combination calculated in php).

I put every hotel and rooms in an array that pass to my view, this is my array constructor (I have a cut a lot of informations):

$hotel = array();

            foreach ($xml->DATA as $entry){
                foreach ($entry->HOTEL_DATA as $entry2){
                    $id = (string)$entry2->attributes()->HOTEL_CODE;
                    $hotel_array2 = array();
                    $hotel_array2['id'] = $id;

                    $hotel_array2['currency'] = (string)$entry2->attributes()->CURRENCY_CODE;

                    $i=0;
                    foreach($entry2->ROOM_DATA as $room){
                        $room_array = array();
                        $room_array['id'] = (string)$room->attributes()->CCHARGES_CODE;
                        $room_array['code'] = (string)$room->attributes()->ROOM_CODE;
                        $room_array['name'] = utf8_decode($room->ROOM_NAME);
                        $hotel_array2['rooms'][$i] = array($room_array);

                        $i++;
                    }
                    if (!isset($hotel_array[$hotel_array2['id']])) {
                        $hotel_array[$hotel_array2['id']] = $hotel_array2;
                    } else {
                        $hotel_array[$hotel_array2['id']]['rooms'] = array_merge($hotel_array[$hotel_array2['id']]['rooms'], $hotel_array2['rooms']);
                    }
                }
            }
        }
        return ($hotel_array);

To print this array I have to make some foreach inside it (about 3, 4) and calculate each combination of rooms that correspond to the user search.

The oriblem is that the I take 30 second to retrieve data and print it (a lot of time) but the problem is that I have to insert many other provider (server to take hotel then more hotel) and if a user have to ordinate by price or by location, order this array with many level is very slowly I think and I have to pass by post this array every time.,

I have thinked a solution:
Store all data into a database in some table: hotel, rooms and viuew only some hotel (20 maximum) and if I have to order it I makle another query that search inside this table (every table is a search or have an external key that correspond to this search).

Is a good way to store data search inside it and after print or ios better to have a very big array woth many foreach and pass every time in Post this?

Is there other solutions?

Is a question about architectural and, how to store (if I have to store data), how to print, how to MANAGE a lot of data I have no experience with this amount of data.

you can retrieve and store in db on a per call basis so first call would be slow next ones after would be from local db (same principal as with memcached and mysql) you could also just take a cache of the xml file+request so you could load a local copy rather than doing the remote request every time again on an if it doesn't exist request it. you could move your processing to a schedule engine outside of the main app and alert users when their search is completed. Or you could do what other comparison sites do and show a "please wait" style screen while the search completes — Dave
– Dave, Commented Jun 28, 2013 at 15:25
Most sites use Pagination. That means when you request hotels from the server it only gives you the XML for the first 10 (or so) and then when the user wants more you load the next 10, etc — chrislondon
– chrislondon, Commented Jun 28, 2013 at 15:27
isn't possbile I have to retrieve all data from xml because isn't in order of price for example @chrislondon — user2287208
– user2287208, Commented Jun 28, 2013 at 15:29

Halcyon · Accepted Answer · 2013-06-28 15:28:21Z

1

So rephrasing your suggested solution:

You have several data providers that provide you with listings of hotel rooms. Querying them every time a user does a search takes too long so you want to cache their listing in your database. Then you query your local database instead.

This seems like a good idea. Make sure the import from the data providers works well: old listings are removed, new listing appear etc. Work out how often you need to update your local cache.

answered Jun 28, 2013 at 15:28

Halcyon

57.7k10 gold badges93 silver badges131 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2287208 Over a year ago

Local cache means my data stored into database right? is the only way that I have found because when I'll have 10 providers I can retrieve too many array and the page is too much slow

Collectives™ on Stack Overflow

Architecture of large data in PHP

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related