Python concurrent logging to avoid disk bottleneck in code

Question

I'm not super familiar with asyncio, but I was hoping there would be some easy way to use asyncio.Queue to push log messages to a Queue instead of writing them to the disk, and then have a worker on a thread wait for these Queue events and write them to disk when resources are available. This seems pretty widely necessary, as logging is a huge bottleneck in a lot of code but isn't always needed in real time. Are there any pre-existing packages for this or can anyone with more experience write a short example script to get me started? NOTE: This needs to interface with existing code, so making it all packaged in a class would probably be preferred.

It sounds like you might want queue.Queue, not asyncio.Queue. asyncio is single threaded. — dirn
– dirn, Commented Jun 10, 2018 at 1:32
Thanks! I don't see that highlighted in documentation. It looked like they had moved queue.Queue at first glance! — Ryan
– Ryan, Commented Jun 12, 2018 at 12:50

Vinay Sajip · Accepted Answer · 2018-06-10 07:22:01Z

1

It's handled in the standard library in recent Python versions. See this post for information, and the official documentation. This functionality predates asyncio, and so doesn't use it (and doesn't especially need to).

For Python 2.7, you can use the logutils package which provides equivalent functionality.

answered Jun 10, 2018 at 7:22

Vinay Sajip

100k15 gold badges184 silver badges196 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python concurrent logging to avoid disk bottleneck in code

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related