Atomic log file rotation with Flask and RotatingFileHandler

Question

I use standard RotatingFileHandler within my Flask application with next parameters: maxBytes=10 * 1024 * 1024, backupCount=50. App is managed by uWSGI behind nginx. uWSGI config file part looks like this:

processes = 16
enable-threads = true
threads = 10

Right after start of an app everything (I mean logging) works well. But after first log file rotation some processes (and maybe threads too) continue writing to rotated file and some - to new one. It is obvious. But for me it is not so obvious how can I rotate log file in the way that all of my processes (and threads) start to write messages to new file.

You can rotate files, but it's best done by having a single dedicated process which listens on a socket and writes to the file using a RotatingFileHandler as you do now. IMO It's not a large change, and none of the code which actually calls logging should be affected - just the code that does logging configuration. — Vinay Sajip
– Vinay Sajip, Commented Jan 19, 2015 at 12:30
Posting this link for future readers to explain why this behaviour occurs for Flask / pre-forked applications. It's not a solution, but certainly helps with understanding. — s3dev
– s3dev, Commented Sep 8, 2020 at 19:59

Vinay Sajip · Accepted Answer · 2015-01-19 10:05:19Z

2

Note that writing to a single log file from multiple processes isn't supported, because there is no cross-platform synchronisation mechanism that can be used. See this cookbook entry for a suggested approach which might work for you.

answered Jan 19, 2015 at 10:05

Vinay Sajip

100k15 gold badges184 silver badges196 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ivan Velichko Over a year ago

So, if I understood correctly, my logging solution is wrong (because I have multiple processes) and I should not try to rotate files atomically, but change entire logging model instead?

s3dev Over a year ago

@VinaySajip - May I get your input on this, please? Given the explanation on this page and this SO answer, would you agree that using syslog is a reasonable solution for logging a small / locally hosted / handful of users Flask service? Thank you.

Vinay Sajip Over a year ago

@S3DEV Yes, it's reasonable, but it's not the only approach you can take. If "Small/locally hosted" means a single process in your deployment, then there's no reason not to use RotatingFileHandler. The article on Flask logging refers to a scenario with logging from multiple processes.

s3dev Over a year ago

@VinaySajip - Excellent, thank you for your response. The deployment has four workers running, so I was experiencing the issues here. Thanks again!

Collectives™ on Stack Overflow

Atomic log file rotation with Flask and RotatingFileHandler

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related