C# WebRequest with php redirecting page

Question

I am messing around trying to write a small web crawler. I parse out a url from some html and sometimes I get a php redirect page. I am looking for a way to get the uri of the redirected page.

I am trying to use System.Net.WebRequest to get a a stream using code like this

        WebRequest req = WebRequest.Create(link);
        Stream s = req.GetResponse().GetResponseStream();
        StreamReader st =  new StreamReader(WebRequest.Create(link).GetResponse().GetResponseStream());

The problem is that the link is a PHP redirect, so the stream is always null. How would I get the URI to the page the php is redirecting?

Isnt it returning a HTTP 302? if so, there should be a response header indicating the new location. Check out stackoverflow.com/questions/1391373/… for more information. — Tejs
– Tejs, Commented Mar 17, 2011 at 1:13

user220583 · Accepted Answer · 2011-03-17 03:24:14Z

1

    HttpWebRequest req = (HttpWebRequest)WebRequest.Create(link);
    req.AllowAutoRedirect = true;
    reg.AutomaticDecompression = DecompressionMethods.GZip;

    StreamReader _st = new StreamReader(_req.GetResponseStream(), System.Text.Encoding.GetEncoding(req.CharacterSet));

the AllowAutoRedirect will automatically take you to the new URI; if that is you're desired effect. The AutomaticDecompression will auto decompress compressed responses. Also you should be executing the get response stream part in a try catch block. I my exp it throws alot of WebExceptions.

Since you're experimenting with this technology make sure you read the data with the correct encoding. If you attempt to get data from a japanese site without using Unicode then the data will be invalid.

answered Mar 17, 2011 at 3:24

user220583

Sign up to request clarification or add additional context in comments.

Comments

Josh M. · Accepted Answer · 2011-03-17 01:16:14Z

0

Check the "Location" header from the response - it should contain the new URL.

answered Mar 17, 2011 at 1:16

Josh M.

28.1k27 gold badges135 silver badges223 bronze badges

Collectives™ on Stack Overflow

C# WebRequest with php redirecting page

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related