Get HTML from url

Question

I'm trying to use C#'s HttpWebRequest to get page source based on url.

In the browser, I'm doing this way:

I go to http://brokercheck.finra.org/Search/Search.aspx
Insert a zipcode, say 00501. and do a search.

I am then redirected to http://brokercheck.finra.org/Search/SearchResults.aspx?SearchGroup=Individual&IndlText=&FirmText=&ZipCode=00544&ZipRadius=5&PageNumber=1 This page's html is what I want.

The simple way of using a get method of HttpWebRequest doe not work. It will just get the html of search page.

I then tried to send a post and get cookies and send a get request with the cookies. But still, the response html was the search page's html.

neoistheone, I want to get the page source of brokercheck.finra.org/Search/… — Singed
– Singed, Commented Oct 29, 2013 at 18:06
It's not apparent to me that this is a GET. I would say you really need to analyze what's sent to the server (e.g. headers, body, request type). — Mike Perrenoud
– Mike Perrenoud, Commented Oct 29, 2013 at 18:30
Try to do a POST Request with these parameters to "Search.aspx" instead of "SearchResults.aspx" — Comfortably Numb
– Comfortably Numb, Commented Oct 29, 2013 at 18:32

Jay · Accepted Answer · 2013-10-31 15:41:47Z

3

Use http sniffer tool(WireShark, or HttpWatch) to capture the http request and responses through web browser.

The search.aspx page uses : onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions("ctl00$phContent$ucUnifiedSearch$lbtnFreeFormSearch", "", true, "", "", false, false))" to submit the request.

The SearchResults.aspx page may use some request header information(e.g previos page) to find out if it is a direct access.

You need find out the exact data in the request for SearchResults.aspx.

edited Oct 31, 2013 at 15:41

answered Oct 30, 2013 at 19:13

Jay

462 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Get HTML from url

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related