Scraping iterative data: empty list

Question

I'm learning how to iterate scraping values from a webpage, I've used bs4 before but I can't figure out why, even if this is a simple example, I'm getting an empty list (or element).

The test I would like to print/save is:

<div data-v-1931433b="" class="mar-m c-r"> 9.1516 </div>

The code I've used is:

 1 import json
 2 import requests
 3 from bs4 import BeautifulSoup
 4
 5
 6 url = 'https://goldencoreex.com/trade/index.html?spAsseType=0#/index'
 7
 8 html_page = BeautifulSoup(requests.get(url).content, 'html.parser')
 9 elements = html_page.find_all("div", class_="mar-m c-r")
10
11 print(elements)

Only guess I have for this issue is that I may have messed-up with the url.

I would be also really much appreciate some hints on how to make the code run continuously, so to capture data whenever the element is updated.

page is dynamic, rendered through javascript. You'll need to look to see if the data is requested through API or use Selenium to allow the page to render before grabbing the html source — chitown88
– chitown88, Commented Nov 27, 2020 at 10:28

baduker · Accepted Answer · 2020-11-27 10:35:48Z

1

As mentioned in the comment, the page is rendered dynamically by JS, but you can query the API to get the data.

For example:

import requests
from tabulate import tabulate

api_url = "https://www.goldencoreex.com/wap/api/realtime!execute.action?symbol=btc,bts,nkn,etc,gdq,xrp,eth,ltc,akro,qtum,bhd"
response = requests.get(api_url).json()

table = [[item['name'], item['open'], item['change_ratio']] for item in response["data"]]
print(tabulate(table, headers=["Name", "Open", "Change"], tablefmt="pretty"))

Output:

+-----------+----------+--------+
|   Name    |   Open   | Change |
+-----------+----------+--------+
| BTC/USDT  | 17809.35 | -5.88  |
| BTS/USDT  |  0.0232  | -6.03  |
| NKN/USDT  | 0.019426 | -3.58  |
| ETC/USDT  |  6.2325  | -4.51  |
| GDQ/USDT  |  9.2947  | -1.34  |
| XRP/USDT  |  0.5456  | -2.38  |
| ETH/USDT  |  527.37  | -3.79  |
| LTC/USDT  |  77.23   | -11.9  |
| AKRO/USDT | 0.00913  | -8.54  |
| QTUM/USDT |  2.6376  | -6.06  |
| BHD/USDT  |  2.5054  | -2.16  |
+-----------+----------+--------+

answered Nov 27, 2020 at 10:35

baduker

20.2k9 gold badges44 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Leafyshark Over a year ago

Hey how did you find this API? Does this look like a legitimate crypto exchange to you?🤔

baduker Over a year ago

@Leafyshark I know almost nothing about crypto so I can't answer your question. As for the API endpoint, I've found it by tracking my browser's traffic. It comes in one of the requests. Check Developer Tool -> Network ->XHR.

Fornax-A Over a year ago

@Leafyshark hi, I believe its 100% a scam crypto webpage rather than illegal. I found it randomly somewhere in a forum. Definitly DON'T put any money on crypto exchange web-pages, in general. Definitely not in that one, in particular. :)

Leafyshark Over a year ago

Thanks, haha! But aren't you working on the site?

Collectives™ on Stack Overflow

Scraping iterative data: empty list

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related