Python: Request large payload from URL

Question

I am trying to get data from a URL using Python. The code I am using is this:

response = requests.get(
                url="https://api.covalenthq.com/v1/"+Chain_id+"/address/"+Address+"/transactions_v2/?    key=API_KEY",
            headers={
                "Authorization": "Basic AUTHORIZATION",
            },
        )
result = response.json()
result = result['data']['items']

So I am entering a Chain_id and wallet address in the URL to get all the transactions for that pair. However, I only get the latest 100 transactions at most. Is there a way to get all transactions? Is there a parameter I can add to requests to get all the data points?

For example, putting this wallet address, 0x60b86AF869f23aEb552fB7F3CaBD11B829f6Ab2F, into etherscan.io (https://etherscan.io/address/0x60b86af869f23aeb552fb7f3cabd11b829f6ab2f), you can see that there are over 51000 transactions however I only get the latest 100.

You can try to request with additional param page-number, and paginate it, meaning to construct a loop to loop through "pages" of data, with each time getting 100 rows. — tyson.wu
– tyson.wu, Commented Apr 27, 2022 at 22:45

Rodrigo Cava · Accepted Answer · 2022-04-27 22:36:13Z

2

This is not a problem with requests, it's a limitation from the API itself.

Looking into the API docs they say they have a limit of 100 request per page.

answered Apr 27, 2022 at 22:36

Rodrigo Cava

1889 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sprizgola · Accepted Answer · 2022-04-27 23:47:50Z

1

As @rodrigo-cava said, you can retrive 100 items (default value) for request. This means that you have to fetch the page in order to collect all the result.

A solution may be the following:

CHAIN_ID = "your chain id"
ADDRESS = "your address"
API_KEY = "your api key"

response_list = list()   

url = f"https://api.covalenthq.com/v1/{CHAIN_ID}/address/{ADDRESS}/transactions_v2/?key={API_KEY}"
response = requests.get(url, headers={"Authorization": "Basic AUTHORIZATION"}).json()

while response["pagination"]["has_more"]:
    page_number = response["pagination"]["page_number"] + 1
    url = f"https://api.covalenthq.com/v1/{CHAIN_ID}/address/{ADDRESS}/transactions_v2/?page-number={page_number}&page-size=100&key={API_KEY}"

    response = requests.get(url, headers={"Authorization": "Basic AUTHORIZATION"})
    if not response.ok:
        raise Exception("Error during the request")
    
    response_list.append(response.json())

edited Apr 27, 2022 at 23:47

answered Apr 27, 2022 at 22:46

Sprizgola

4412 silver badges12 bronze badges

9 Comments

MathMan 99 Over a year ago

I am trying this but it gives me the following error: TypeError: 'Response' object is not subscriptable specifically at ( while response["pagination"]["has_more"]:)

Sprizgola Over a year ago

have you invoked the .json() method after the request? the error seems to point that response is a requests.Response object instead of a json

Sprizgola Over a year ago

Can you check in the console what are the keys of the response object? Looks like that this field does not exists. I've tried in the official website and I found that there is a 'pagination' key.

Sprizgola Over a year ago

I'll check. (PS: remove the address!)

Sprizgola Over a year ago

I have updated my answer; let me know if it is ok

|

Collectives™ on Stack Overflow

Python: Request large payload from URL

2 Answers 2

Comments

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related