using multiprocessing when running a script

Question

i am running a script which takes in a url and downloads the file locally, and then passes the filename as an argument to a function.The problem is doing this takes a lot of time.So i tried using Threadpool,but this does not give any improvements.Am i doing this wrong, here is what it looks like.

pool = ThreadPool(processes=8)
ocr_result = pool.apply_async(download_file, (url,))
file_name = ocr_result.get()
async_result = pool.apply_async(return_label, (file_name,))
prediction, prediction_list  = async_result.get()

Any suggestions would be really helpful. Thanks in advance.

You are not using multiprocessing. You are using the pool, but you are running just one task synchronously. — Klaus D.
– Klaus D., Commented Apr 17, 2019 at 6:21

knh190 · Accepted Answer · 2019-04-17 08:44:51Z

1

As suggested in comment, there's a sample using aiohttp and asyncio:

def main():
    # limit concurrency
    loop = asyncio.get_event_loop()
    connector = aiohttp.TCPConnector(limit=100)

    # login if required
    async with aiohttp.ClientSession(loop=loop, connector=connector) as sess:
        async with sess.post(
                LOGIN_URL, data=payload) as resp:

            # ensure login success
            assert resp.status == 200
            for url in download_links:
                await download(url, sess)

And your download function looks like:

async def download(url, sess):
    async with sess.get(url) as resp:
        if resp.status == 200:
            # post process

Finally use a main loop:

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

answered Apr 17, 2019 at 8:44

knh190

2,8921 gold badge21 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ryan Over a year ago

Thanks for your time

Collectives™ on Stack Overflow

using multiprocessing when running a script

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related