Python script only writes partial results to file

Question

I have a little python script that I wrote to populate my proxychains.conf file with valid public proxies. For each request, 10 proxies are retrieved and added to proxychains.conf, and this is done 3 times so that a total of 30 proxies are added.

When I was writing the script and having results return to stdout everything worked as expected = 30 proxies were retrieved and returned. However when I added the file operations part of the script only 10 proxies are written to the file. I'm still learning Python and I've tried to rearrange some things but it's not working out. I can't figure out if :

my counter is in the wrong place?
my file operations are in the wrong place?

Here is the code:

#!/usr/bin/env python3

import requests
import sys,os

proxy_file = '/etc/proxychains.conf'
base_url = 'http://proxy.tekbreak.com/10/json'
headers = {'user-agent':'Mozilla/5.0 (Windows NT x.y; Win64; x64; rv:10.0) Gecko/20'}

def fetchprox():
    pf = open(proxy_file, 'r')
    lines = pf.readlines()
    pf.close()
    with open (proxy_file, 'w') as f:
        del lines[69:]
        f.writelines([item for item in lines[:-1]])
        r = requests.get(base_url, headers=headers)
        n = 0
        while n < 10:
            ip = r.json()[n]['ip']
            port = r.json()[n]['port']
            p_type = r.json()[n]['type']

         #output to proxychains.conf
            f.writelines(str(p_type + " " + ip + " " + port + "\n"))
            n += 1    

for i in range(0,3):  
    fetchprox()

Thanks for the help!

EDIT I found a solution based off of Giordano's answer, however I believe it could be implemented better. It seems redundant to have to access this file 3 times to just to write some data. So here is the portion of the script that was changed:

<--snip->
pf = open(proxy_file, 'r')
lines = pf.readlines()
pf.close()

f = open (proxy_file, 'w')
del lines[69:]
f.writelines([item for item in lines[:-2]])
f.close()

def fetchprox():
    with open (proxy_file, 'a') as f:
        r = requests.get(base_url, headers=headers)
        n = 0
        while n < 10:
<--snip-->

so is there a more efficient way to accomplish this?

try to change with open (proxy_file, 'w') as f: in with open (proxy_file, 'a') as f:. Probably you are overwrite the file each time — Giordano
– Giordano, Commented Mar 30, 2017 at 13:17
As I see, you read your list of proxies from proxy_file and write the result to the same file. You overwrite it, so after the first iteration only 10 lines are left. No matter how many times you run the loop, it would always be 10 lines. — Roman Kiselev
– Roman Kiselev, Commented Mar 30, 2017 at 13:23

Giordano · Accepted Answer · 2017-03-30 15:27:29Z

2

Your script is correct, but as I mentioned in a comment, you overwrite your file each loop.
More in detail, the wrong line is the following:

with open (proxy_file, 'w') as f:

With w, you write a file, but if it is not empty, you overwrite the content.
To fix it you can use append mode in this way:

with open (proxy_file, 'a') as f:

append does not overwrite the content of the file but adds new lines.

Answer after edit

First of all you can use with to open a file, and so i've removed also .close() because with handle closure automatically. I also changed the while loop with a for loop in this way:

lines = None
with open(proxy_file, 'r') as pf:
    lines = pf.readlines()

with open (proxy_file, 'w') as f:
    del lines[69:]
    f.writelines([item for item in lines[:-2]])

def fetchprox():
    with open (proxy_file, 'a') as f:
        data = requests.get(base_url, headers=headers).json()
        for element in data:
            ip = element['ip']
            port = element['port']
            p_type = element['type']

            #output to proxychains.conf
            f.writelines(str(p_type + " " + ip + " " + port + "\n"))

for i in range(0,3): 
    fetchprox()

Second edit

If you want remove one access to the file, you can use r+ mode in this way:

lines = None
with open(proxy_file, 'r+') as pf:
    lines = pf.readlines()
    del lines[69:]
    pf.writelines([item for item in lines[:-2]])
...

With r+ you can read and write, opening one time the file.

edited Mar 30, 2017 at 15:27

answered Mar 30, 2017 at 13:32

Giordano

5,6183 gold badges38 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Aaron Nelson Over a year ago

Hi Giordano thanks for the suggestion. When I implemented 'a' the script actually rewrote the entire contents of proxychains.conf and then only 10 proxy servers. I was able to find a solution which i will add as an edit to th original question because I do believe there is a better solution.

Aaron Nelson Over a year ago

Giordano - your edit works perfectly. I'm still a little confused when to use f = open(file, 'r') and with open (file, 'r') as f, and thanks for the tip about using the '+' to open the file in read/write. Additionally using a for statement instead of a while loop. I wasn't sure how to grab every instance of ip from the json file. Thanks for your help!

Giordano Over a year ago

Glad to help you! to understand better when to use "open" and when to use "with", please read the Python documentation. Please if your problems are fixed, mark the answer as accepted in order to help also other users with the same problem. Best regards

Collectives™ on Stack Overflow

Python script only writes partial results to file

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related