I am trying to scrape a website in order to extract XHTML data to save as variables in PHP.
The website uses AJAX which means the data I am seeking is returned following an XMLHttpRequest using a search string such as http://website.com/ajax.php?mid=2&pid=4
I have tried setting the request headers obtained from both Chrome and Firefox (which differ slightly) and included them trying both curl_setop() and stream_context_create() options with file_get_contents(), but the data is still not being returned. When I check the request headers of my script in Chrome it is displaying the method as GET even though this has been set to POST. It does this for both cURL and stream contexts.
The website does not require a login, but I have noticed that it sends session cookies as part of the request, which I have also included in the HTTP request to no avail.
One other point to note is that removing the Content-Length value from the request header returns a blank page immediately, whereas leaving it in stays loading for around one minute before timeout, but this may be irrelevant.
My questions are has anyone else had experience with PHP and AJAX requests using HTTP headers and why am I seeing a GET request when I have specified this as POST?
Thankyou in advance for any insights.