python 3.4 multiprocessing

Question

This question is asking for advice as well as assistance with some code.

I currently am learning Python with 3.4 I have built a basic network checking tool, i import items from a text file and for each of them i want python to check dns (using pydns), ping the ip (using subprocess to call OS native ping).

Currently i am checking 5000 to 9000 thousand IP address and its taking a number of hours, approx 4 to return all the results.

I am wondering if i can use multiprocessing or threading to speed this up but still the return the output to a list so that the row can be written to a csv file at the very end of the script in bulk.

I am new to python so please tell me if i have overlooked something i should of also.

Main code http://pastebin.com/ZS23XrdE

Class http://pastebin.com/kh65hYhG

multiprocessing is probably a good bet, using a joinable queue to get the data back. But that might be no better than subprocess, depending on how you are running it. Are you waiting for each to complete? — cdarke
– cdarke, Commented Dec 12, 2014 at 8:22
I have updated the main post with the current code i am using that is taking a long time. — clonelab
– clonelab, Commented Dec 12, 2014 at 9:02

Community · Accepted Answer · 2017-05-23 11:44:40Z

1

You could use multiple threads to run child processes (ping in your case) and collect their output but it is not necessary. Here's a code example how to make multiple http requests using a thread pool. Here's code that uses concurrent.futures to make dns requests concurrently.

You don't need multiple threads/process to check 5000-9000 IPs (DNS, ICMP). You could use gevent, twisted, asyncio to make network connections in the same process.

edited May 23, 2017 at 11:44

CommunityBot

11 silver badge

answered Dec 30, 2014 at 15:39

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

noxdafox · Accepted Answer · 2014-12-30 11:57:51Z

0

As most of the work seems IO based, you can easily rely on Threads.

Take a look at the Executor.map() function in cocurrent.futures: https://docs.python.org/3/library/concurrent.futures.html

You can pass the list of IPs and the function you want to run against each element, the returned value, virtually, is the list of results of the given function.

In your specific case you can wrap the two worker's methods (check_dns_ip and os_ping) in a single one and pass it to the ThreadPoolExecutor.map function.

answered Dec 30, 2014 at 11:57

noxdafox

15.1k4 gold badges37 silver badges46 bronze badges

Collectives™ on Stack Overflow

python 3.4 multiprocessing

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related