I am a Linux newbie and I'm running the following Linux command:
nohup python my_script.py > my.out 2> my.err
and everything works as expected; all the output from my python (2.7) print statements get logged in the file called my.out.
The only problem is that after several days of running the file, my.out, gets very large. Is there a way (at the Linux shell level) I can have a new "my.out" created, say once a day, with a time stamp in the file name?
You should use logging module in your script :
import logging.handlers
LOG_FILE_NAME = '/var/log/my_script.log'
LOGGING_LEVEL = logging.INFO
formatter = logging.Formatter('%(asctime)s %(name)s %(levelname)s %(message)s')
handler = logging.handlers.RotatingFileHandler(LOG_FILE_NAME, mode='a', maxBytes=5000000, backupCount=5)
handler.setFormatter(formatter)
log = logging.getLogger("my_script")
log.addHandler(handler)
log.setLevel(LOGGING_LEVEL)
In this example, we configure a log file rotation of 5MB with a retention of 5 files. So each time your log file reach 5MB, the script creates a new log file and archive the old one (5 files max).
Here is an example of code to log data in your log file :
def do_something():
# do stuff here
log.info("Something has been done")
Once you've done that, simply run your python script :
python my_script.py
Everything will be logged in the file you want with automatic log rotation.
If you prefer a rotation based on time, please refer to TimedRotatingFileHandler
https://docs.python.org/2/library/logging.handlers.html#timedrotatingfilehandler
log will save data in your log file only. You can still redirect the std output to a file, but i dont see the point. If you make a good use of the command log you won't need to redirect console. And to debug, use a debuger :)What you want is what Apache fellows did with rotatelogs. Extract:
rotatelogs is a simple program for use in conjunction with Apache's piped logfile feature. It supports rotation based on a time interval or maximum size of the log.
Synopsis
rotatelogs [ -l ] [ -L linkname ] [ -p program ] [ -f ] [ -t ] [ -v ] [ -e ] [ -c ] [ -n number-of-files ] logfile rotationtime|filesize(B|K|M|G) [ offset ]
...
rotationtime The time between log file rotations in seconds. The rotation occurs at the beginning of this interval. For example, if the rotation time is 3600, the log file will be rotated at the beginning of every hour; if the rotation time is 86400, the log file will be rotated every night at midnight.
For your use case, you could do:
nohup python my_script.py 2> my.err | rotatelogs -l my.out.%Y.%m.%d 86400
to have one file per day changing file at 0h00 local time.
You can find it in the Apache HTTPD project which exists for almost any common architecture
Yes it is, use command date to add a date inside the name of file, like :
nohup python my_script.py > my_`date +"%d_%m_%y"`.out 2> my.err
4 example I've done the next example with the following command :
echo "hello" > name_`date +"%m_%y"`
and the name of the file is name_06_17
You can see more information about date command typing :
man date
This will allow you to create different files, one per month.
hope will help!
date +"%d_%m_%y".out 2> my.err
logrotate