Scraping the stackoverflow user data

Question

import requests
from bs4 import BeautifulSoup
import csv

response = requests.get('https://stackoverflow.com/users?page=3&tab=reputation&filter=week').text
soup = BeautifulSoup(response, 'lxml')
for items in soup.select('.user-details'):
    name = items.select("a")[0].text
    location = items.select(".user-location")[0].text
    reputation = items.select(".reputation-score")[0].text
    print(name,location,reputation)

    with open('stackdata.csv','a',newline='') as csv_file:
        writer = csv.writer(csv_file)
        writer.writerow([name,location,reputation])

When we change the url of this code the output remains same.

when I change the url stackoverflow.com/users to stackoverflow.com/… the output is remain same — Jay Pratap Pandey
– Jay Pratap Pandey, Commented Nov 7, 2017 at 14:15
Would suggest to use SO API to fetch records stackapi.readthedocs.io/en/latest/user/… — MaNKuR
– MaNKuR, Commented Apr 11, 2018 at 12:26

MaNKuR · Accepted Answer · 2018-05-28 12:44:55Z

1

I came across a similar problem. The solution that works for me is using selenium. Though I used headless browser i.e phantomjs I assume it should work for other browsers too.

driver = webdriver.PhantomJS('/home/practice/selenium/webdriver/phantomjs/bin/phantomjs')
users = []
page_num = 1
driver.get('https://stackoverflow.com/users?page={page_num}&tab=reputation&filter=week'.format(page_num=page_num))
content = driver.find_element_by_id('content')

for details in content.find_elements_by_class_name('user-details'):
    users.append(details.text)

print(users)

Change the page_num to get the desired result.

Hope this will help!

answered May 28, 2018 at 12:44

MaNKuR

2,7141 gold badge22 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scraping the stackoverflow user data

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related