Get HTML code from dynamic page

Question

I am trying to get the HTML code of the following website. http://fortune.com/fortune500/list/

But the problem is when we visit this website in browser, it only shows the first 20 companies and when we go to the bottom part of website it loads the next 50 companies.

How do i get the first 700 companies in HTML code from this website? I tried the code from this website https://www.mkyong.com/java/how-to-get-url-content-in-java/ to get the HTML content but as expected it gives only the top 20 companies

Any help is much appreciated Thanks

Programmatically you won't be able to do that because Ajax calls are involved in that HTML. The approach in that link gets the HTML as it, a text with an HTML structure. — Ele
– Ele, Commented Dec 8, 2017 at 1:32
Thanks . I can parse the HTML structure in downstream but the problem is i need to get more companies list from the fortune500list website (Not first 20 companies) — user3757805
– user3757805, Commented Dec 8, 2017 at 1:41

Angelo C · Accepted Answer · 2017-12-08 01:43:49Z

1

CURL: http://fortune.com/api/v2/list/2013055/expand/item/ranking/asc/{{start_from}}/{{num_limit}}

Example: http://fortune.com/api/v2/list/2013055/expand/item/ranking/asc/1/100

The site "fortune.com" return max 100 elements form CURL.

The CURL return a JSON.

answered Dec 8, 2017 at 1:43

Angelo C

664 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3757805 Over a year ago

Thanks. Calling api returns the data but I am trying in a different approach of parse/crawl the website and find the data in it

Angelo C Over a year ago

The site fortune.com doesn't load all the data at the beginning.. Therefore you don't recover them.. (Sorry My English)

Julien Nioche · Accepted Answer · 2017-12-08 06:02:28Z

0

You should use Selenium for this. Here is a tutorial on how to use it with StormCrawler. You could also use it directly if you wanted to.

answered Dec 8, 2017 at 6:02

Julien Nioche

4,8741 gold badge24 silver badges30 bronze badges

Collectives™ on Stack Overflow

Get HTML code from dynamic page

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related