Nested while loop for API json collection

Question

I'm requesting 590 pages from the Meetup API. I've iterated with a while loop to get the pages. Now that I have the pages I need to request this pages and format them correctly as python in order to place into a Pandas dataframe.

This is how it looks when you do it for one url :

url = ('https://api.meetup.com/2/groups?offset=1&format=json&category_id=34&photo-host=public&page=100&radius=200.0&fields=&order=id&desc=false&sig_id=243750775&sig=768bcf78d9c73937fcf2f5d41fe6070424f8d0e3')
r = requests.get(url).json()
data = pd.io.json.json_normalize(r['results'])

But because I have so many pages I want to do this automatically and iterate through them all.

That's how nested while loops came to mind and this is what I tried:

urls = 0
offset = 0

url = 'https://api.meetup.com/2/groups?offset=%d&format=json&category_id=34&photo-host=public&page=100&radius=200.0&fields=&order=id&desc=false&sig_id=243750775&sig=768bcf78d9c73937fcf2f5d41fe6070424f8d0e3'

r = requests.get(urls%d = 'https://api.meetup.com/2/groups?offset=%d&format=json&category_id=34&photo-host=public&page=100&radius=200.0&fields=&order=id&desc=false&sig_id=243750775&sig=768bcf78d9c73937fcf2f5d41fe6070424f8d0e3').json()

while urlx < 591:
   new_url = r % urls % offset
   print(new_url)

   offset += 1

However, it isn't working and I'm receiving many errors including this one:

SyntaxError: keyword can't be an expression

Looks like the issue is with urls%d on line, requests.get(urls%d = 'https://api.meetup.com/2/.... That is not a valid python syntax. — user459872
– user459872, Commented May 8, 2018 at 10:54
urlx is not defined and even if it was in some code earlier, it doesn't change its value in the loop while urlx < 591:. This will lead to an infinite loop. — Tushar Aggarwal
– Tushar Aggarwal, Commented May 8, 2018 at 10:55

snow_fall · Accepted Answer · 2018-05-10 10:15:51Z

1

Not sure what you're trying to do, and the code has lots of issues.

But if you just want to loop through 0 to 591 and fetch URLs, then here's the code:

import requests
import pandas as pd

dfs = []

base_url = 'https://api.meetup.com/2/groups?offset=%d&format=json&category_id=34&photo-host=public&page=100&radius=200.0&fields=&order=id&desc=false&sig_id=243750775&sig=768bcf78d9c73937fcf2f5d41fe6070424f8d0e3'

for i in range(0, 592):
    url = base_url % i
    r = requests.get(url).json()
    print("Fetching URL: %s\n" % url)

    # do something with r here
    # here I'll append it to a list of dfs

    dfs.append(pd.io.json.json_normalize(r['results']))

edited May 10, 2018 at 10:15

snow_fall

2251 gold badge5 silver badges13 bronze badges

answered May 8, 2018 at 10:58

Ashish Acharya

3,4091 gold badge19 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Nested while loop for API json collection

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related