how can i download a web page from my web app, then reading "title" and "description" metatag ? Like a web crawler, but in Asp.net and called by ..an asp.net web page ?
Thanks!
Add a comment
|
2 Answers
You can do a screen scrape of an external URL in .NET using the WebClient class, which you'll find in the System.Net namespace. Use the DownloadData method to download the content from a specified URL. The downloaded data comes down as a byte array, but you can convert this to a string.
The following snippet shows how to use WebClient to grab the HTML from my blog's homepage, http://scottonwriting.net/sowblog/default.aspx:
// Create a new WebClient instance.
WebClient myWebClient = new WebClient();
// Download the markup from
byte[] myDataBuffer = myWebClient.DownloadData("http://scottonwriting.net/sowblog/default.aspx");
// Convert the downloaded data into a string
string markup = Encoding.ASCII.GetString(myDataBuffer);
Once you have the markup you can use regular expressions or string searching methods to pick out the markup of interest.
Comments
Use the HTML Agility Pack and its HTMLWeb class.
Comments
