Facing issues writing names in a csv file using panda

Question

I've written a scraper in python to parse different names from a webpage. As I'm very new to panda so can't understand how to write data in a csv file properly. The way I've written my script does scrape desired names but in case of writing to a csv file it only write the last name. Any help on this will be vastly appreciated. Thanks in advance.

import requests ; from bs4 import BeautifulSoup
import pandas as pd

res = requests.get('https://bytes.com/topic/python/answers/596974-socket-error-10061-connection-refused').text
soup = BeautifulSoup(res,"lxml")
for name in soup.select(".smallfont a"):
    item = pd.DataFrame([name.text])[0]
    item.to_csv('file.csv')
    print(item)

Rafael Araujo · Accepted Answer · 2017-09-19 21:23:55Z

1

I'm not sure if this is your question, but the problem is that you are overwriting the dataframe in the loop. In this way, just the last data in the list will be recorded on csv file.

Try this code:

import requests ; from bs4 import BeautifulSoup
import pandas as pd

res = requests.get('https://bytes.com/topic/python/answers/596974-socket-
error-10061-connection-refused').text
soup = BeautifulSoup(res,"lxml")
items_list = []
for name in soup.select(".smallfont a"):
    items_list.append(name.text)

item = pd.DataFrame(items_list)
item.to_csv('file.csv')
print(item)

Out put:

                                                   0
0  boost::asio (Permission denied, Connection ref...
1               Error Code 10061: Connection refused
2                    socket.error connection refused
3              Connection refused error from browser
4  connection refused when uploading a package to...
5                smtplib (111, 'Connection refused')
6  sending of mail (smtp) - connection refused - ...
7              socket object, connection refused ...
8                                 socket error 10061

Is this your intention?

answered Sep 19, 2017 at 21:23

Rafael Araujo

1044 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

SIM Over a year ago

Yep it is. Thanks Rafael Araújo.

Loïc · Accepted Answer · 2017-09-19 21:14:32Z

0

Why do you need panda ?

with open('file.csv', 'w+') as f:
    for name in soup.select(".smallfont a"):
        item = name.decode_contents()
        f.write(item)
        print(item)

answered Sep 19, 2017 at 21:14

Loïc

12k2 gold badges35 silver badges52 bronze badges

1 Comment

SIM Over a year ago

Thanks Loïc, for your answer. I actually know how to write in a csv file if it were not for panda. Thanks again.

Collectives™ on Stack Overflow

Facing issues writing names in a csv file using panda

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related