Jsoup Scraping HTML dynamic content

Question

I'm new to Jsoup and I have been trying to create a small code that gets the name of the items in a steam inventory using Jsoup.

public Element getItem(String user) throws IOException{
    Document doc;

    doc = Jsoup.connect("http://steamcommunity.com/id/"+user+"/inventory").get();
    Element element = doc.getElementsByClass("hover_item_name").first();
    return element;
}

this methods returns:

<h1 class="hover_item_name" id="iteminfo0_item_name"></h1>

and I want the information beetwen the "h1" labels which is generated when you click on a specific window. Thank you in advance.

yeah, i want that "XYZ" but its generated when you click a specific window — Setekorrales
– Setekorrales, Commented May 9, 2016 at 19:50

Community · Accepted Answer · 2017-05-23 11:53:49Z

2

You can use the .select(String cssQuery) method:

doc.select("h1") gives you all h1 Elements. If you need the actual Text in these tags use the .text() for each Element. If you need a attribute like class or id use .attr(String attributeKey) on a Element eg:

doc.getElementsByClass("hover_item_name").first().attr("id")

gives you "iteminfo0_item_name"

But if you need to perform clicks on a website you can't do that with JSoup, hence JSoup is a HTML parser and not a browser alternative. Jsoup can't handle dynamic content.

But what you could do is, firstly scrape the relevant data in your h1 tags and then send a new .post() request, respectively an ajax call

If you rather want a real webdriver, have a look at Selenium.

edited May 23, 2017 at 11:53

CommunityBot

11 silver badge

answered May 9, 2016 at 20:01

Mad Matts

1,11610 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Setekorrales Over a year ago

Thank you, the problem is that I want to handle dynamic content so i think ill be using another method different from Jsoup

Pedro Lobito · Accepted Answer · 2016-05-09 20:00:19Z

0

Use .text() and return a String, i.e.:

public String getItem(String user) throws IOException{
    Document doc;
    doc = Jsoup.connect("http://steamcommunity.com/id/"+user+"/inventory").get();
    Element element = doc.getElementsByClass("hover_item_name").first();
    String text = element.text();
    return text;
}

answered May 9, 2016 at 20:00

Pedro Lobito

99.8k36 gold badges274 silver badges278 bronze badges

Collectives™ on Stack Overflow

Jsoup Scraping HTML dynamic content

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related