Safe way to exit an infinite loop within a Thread Pool for Python3

Question

I am using Python3 modules:

requests for HTTP GET calls to a few Particle Photons which are set up as simple HTTP Servers
As a client I am using the Raspberry Pi (which is also an Access Point) as a HTTP Client which uses multiprocessing.dummy.Pool for making HTTP GET resquests to the above mentioned photons

The polling routine is as follows:

def pollURL(url_of_photon):
    """
    pollURL: Obtain the IP Address and create a URL for HTTP GET Request
    @param: url_of_photon: IP address of the Photon connected to A.P.
    """
    create_request = 'http://' + url_of_photon + ':80'
    while True:
        try:
            time.sleep(0.1) # poll every 100ms
            response = requests.get(create_request)
            if response.status_code == 200:
                # if success then dump the data into a temp dump file
                with open('temp_data_dump', 'a+') as jFile:
                    json.dump(response.json(), jFile)
            else:
               # Currently just break
               break
        except KeyboardInterrupt as e:
            print('KeyboardInterrupt detected ', e)
            break

The url_of_photon values are simple IPv4 Addresses obtained from the dnsmasq.leases file available on the Pi.

the main() function:

def main():
    # obtain the IP and MAC addresses from the Lease file
    IP_addresses = []
    MAC_addresses = []
    with open('/var/lib/misc/dnsmasq.leases', 'r') as leases_file:
        # split lines and words to obtain the useful stuff.
        for lines in leases_file:
            fields = lines.strip().split()
            # use logging in future
            print('Photon with MAC: %s has IP address: %s' %(fields[1],fields[2]))
            IP_addresses.append(fields[2])
            MAC_addresses.append(fields[1])

            # Create Thread Pool
            pool = ThreadPool(len(IP_addresses))
            results = pool.map(pollURL, IP_addresses)
            pool.close()
            pool.join()

if __name__ == '__main__':
    main()

Problem

The program runs well however when I press CTRL + C the program does not terminate. Upon digging I found that the way to do so is using CTRL + \

How do I use this in my pollURL function for a safe way to exit the program, i.e. perform poll.join() so no leftover processes are left?

notes:

the KeyboardInterrupt is never recognized with the function. Hence I am facing trouble trying to detect CTRL + \.

Probably you have the same issue as this one

Adelin
– Adelin

2017-09-11 14:23:48 +00:00
Commented Sep 11, 2017 at 14:23 — Adelin
– Adelin, Commented Sep 11, 2017 at 14:23
Actually this Thread is the nearest possible thing.

Shan-Desai
– Shan-Desai

2017-09-11 14:26:19 +00:00
Commented Sep 11, 2017 at 14:26 — Shan-Desai
– Shan-Desai, Commented Sep 11, 2017 at 14:26

Community · Accepted Answer · 2020-06-20 09:12:55Z

2

The pollURL is executed in another thread. In Python, signals are handled only in the main thread. Therefore, SIGINT will raise the KeyboardInterrupt only in the main thread.

From the signal documentation:

Signals and threads

Python signal handlers are always executed in the main Python thread, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication. You can use the synchronization primitives from the threading module instead.

Besides, only the main thread is allowed to set a new signal handler.

You can implement your solution in the following way (pseudocode).

event = threading.Event()

def looping_function( ... ):
    while event.is_set():
        do_your_stuff()

def main():
    try:
        event.set()
        pool = ThreadPool()
        pool.map( ... )
    except KeyboardInterrupt:
        event.clear()
    finally:
        pool.close()
        pool.join()

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Sep 11, 2017 at 15:55

noxdafox

15.1k4 gold badges37 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Shan-Desai Over a year ago

Do you think pool.terminate() should be within the except block?

noxdafox Over a year ago

In general, threads cannot be terminated. Therefore, calling pool.terminate() would be redundant. What you need to make sure is that nothing blocks your workers indefinitely or they will never get to evaluate event.is_set(). Make sure your requests have a timeout set and it should be enough.

Collectives™ on Stack Overflow

Safe way to exit an infinite loop within a Thread Pool for Python3

Problem

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Problem

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related