Getting no data when scraping a table

Question

I am trying to scrape historical data from a table in coinmarketcap. However, the code that I run gives back "no data." I thought it would be fairly easy, but not sure what I am missing.

url = "https://coinmarketcap.com/currencies/bitcoin/historical-data/"

data = requests.get(url)

bs=BeautifulSoup(data.text, "lxml")
table_body=bs.find('tbody')
rows = table_body.find_all('tr')
for row in rows:
    cols=row.find_all('td')
    cols=[x.text.strip() for x in cols]
    print(cols)

Output:

C:\Users\Ejer\anaconda3\envs\pythonProject\python.exe C:/Users/Ejer/PycharmProjects/pythonProject/CloudSQL_test.py
['No Data']

Process finished with exit code 0

I found this xhr request and it appears to be populating that table, so I don't think you need to scrape: web-api.coinmarketcap.com/v1/cryptocurrency/ohlcv/… — bherbruck
– bherbruck, Commented Dec 30, 2020 at 22:00
But it is not available for basic version looks like. It seems like if you want historical data, you have to pay for it? — Dfhaa_DK
– Dfhaa_DK, Commented Dec 30, 2020 at 22:06

bherbruck · Accepted Answer · 2020-12-30 22:20:03Z

4

You don't need to scrape the data, you can get request it:

import time
import requests


def get_timestamp(datetime: str):
    return int(time.mktime(time.strptime(datetime, '%Y-%m-%d %H:%M:%S')))


def get_btc_quotes(start_date: str, end_date: str):
    start = get_timestamp(start_date)
    end = get_timestamp(end_date)
    url = f'https://web-api.coinmarketcap.com/v1/cryptocurrency/ohlcv/historical?id=1&convert=USD&time_start={start}&time_end={end}'
    return requests.get(url).json()


data = get_btc_quotes(start_date='2020-12-01 00:00:00',
                      end_date='2020-12-10 00:00:00')

import pandas as pd
# making A LOT of assumptions here, hopefully the keys don't change in the future
data_flat = [quote['quote']['USD'] for quote in data['data']['quotes']]
df = pd.DataFrame(data_flat)

print(df)

Output:

           open          high           low         close        volume    market_cap                 timestamp
0  18801.743593  19308.330663  18347.717838  19201.091157  3.738770e+10  3.563810e+11  2020-12-02T23:59:59.999Z
1  19205.925404  19566.191884  18925.784434  19445.398480  3.193032e+10  3.609339e+11  2020-12-03T23:59:59.999Z
2  19446.966422  19511.404714  18697.192914  18699.765613  3.387239e+10  3.471114e+11  2020-12-04T23:59:59.999Z
3  18698.385279  19160.449265  18590.193675  19154.231131  2.724246e+10  3.555639e+11  2020-12-05T23:59:59.999Z
4  19154.180593  19390.499895  18897.894072  19345.120959  2.529378e+10  3.591235e+11  2020-12-06T23:59:59.999Z
5  19343.128798  19411.827676  18931.142919  19191.631287  2.689636e+10  3.562932e+11  2020-12-07T23:59:59.999Z
6  19191.529463  19283.478339  18269.945444  18321.144916  3.169229e+10  3.401488e+11  2020-12-08T23:59:59.999Z
7  18320.884784  18626.292652  17935.547820  18553.915377  3.442037e+10  3.444865e+11  2020-12-09T23:59:59.999Z
8  18553.299728  18553.299728  17957.065213  18264.992107  2.554713e+10  3.391369e+11  2020-12-10T23:59:59.999Z

edited Dec 30, 2020 at 22:20

answered Dec 30, 2020 at 22:09

bherbruck

2,2161 gold badge9 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Dfhaa_DK Over a year ago

Wow nice, but is the data stored in a dictionary? What if I want to create a dataframe to preform som analysis?

Dfhaa_DK Over a year ago

Wow!! This cool. Thanks alot. Happy new year to you

chitown88 Over a year ago

you can also use json_normalize to convert the json into a dataframe: from pandas.io.json import json_normalize df = json_normalize(data['data']['quotes'])

bherbruck Over a year ago

@chitown88 wow that's cool, I didn't know about that pandas feature, I like how it puts the nested data in a quote.USD.close notation

chitown88 Over a year ago

Ya if it’s deeply nested, Need to a little more work (I also need to read up on some of the parameters), but at a very basic level, does flatten out for you into s dataframe

Jhoubert Rincon · Accepted Answer · 2020-12-30 22:13:22Z

4

Your problem basically is you're trying to get a table but this table is dynamically created by JS in this case you need to call an interpreter for this JS. But however you just can check the network monitor on your browser and you can get the requests and probably contains a full JSON or XML raw data and you don't need to scrape. I did it and I got this request:

https://web-api.coinmarketcap.com/v1/cryptocurrency/ohlcv/historical?id=1&convert=USD&time_start=1604016000&time_end=1609286400

Check it out and I hope help you!

edited Dec 30, 2020 at 22:13

answered Dec 30, 2020 at 22:06

Jhoubert Rincon

1606 bronze badges

Collectives™ on Stack Overflow

Getting no data when scraping a table

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related