Extracting specific data from a webpage-source programmatically

Question

The following question applies to any programming language

I am working on a program which on providing the webpage-source-code as input will extract some specific kind of data.

Suppose I provided the following page-source as input to my program:

<table>
    <tr>
        <td id="a" class="product-name">Product A</td>
        <td id="1" class="product-price">$100</td>
    </tr>

    <tr>
        <td id="b" class="product-name">Product B</td>
        <td id="2" class="product-price">$200</td>
    </tr>

    <tr>
        <td id="c" class="product-name">Product C</td>
        <td id="3" class="product-price">$300</td>
    </tr>
</table

On this webpage, there are products mentioned along with their selling price. The webpage look like this:

 Product A: $100
 Product B: $200
 Product C: $300

I want to use this page-source to copy this data to database. Since the product names and their prices are mentioned in fix tags and classes (like <td> or <div> etc), How can I extract the data these programmatically? Is there any good algorithm/code/library to extract such data from a page-source?

I think this can be done by using getElementByID in Javascript. But I am not sure. Or XML can be used? How? Any other good method/algorithms?

Note: I am doing this to my own website. I already have an old website and I want to use all the data in my new one. Entering all the data manually again is a huge task. So I want to copy data from my old one. Any programming language is okay with me.

yoda · Accepted Answer · 2011-08-06 13:22:28Z

2

Just use this : http://simplehtmldom.sourceforge.net/

By the way, it's called crawling.

answered Aug 6, 2011 at 13:22

yoda

11k19 gold badges68 silver badges94 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hakre · Accepted Answer · 2011-08-06 13:35:21Z

2

To query data from XML documents you can make use of the xpath language that is available in many programming languages. It explicitly deals with XML documents, not strings.

Another approach is to do a string analysis of the documents you want to extract data from and extract the data as needed on a much lower level. This can sometimes be faster, but XPATH is much more robust as it can differ between tags, attributes and values.

Normally you just need a transformation from the documents into another form, for this many languages provide an interface to XSLT and XQuery.

A term you can search for is scraping, if you're looking for some resources for PHP, checkout web-scraping-with-php which is available in print as well.

edited Aug 6, 2011 at 13:35

answered Aug 6, 2011 at 13:29

hakre

200k55 gold badges454 silver badges868 bronze badges

Collectives™ on Stack Overflow

Extracting specific data from a webpage-source programmatically

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related