check multiple files if they are empty in python

Question

I have an n number of log files that a script regularly downloads and upload on slack for monitoring purposes. However with recent improvements in our postgresql database some of the log files are now empty (meaning no errors or long queues were recorded) this being said, I would need to segregate files that are empty vs not empty and if it's empty skip the file from being uploaded entirely and proceed with the ones that are not empty.

348 postgresql.log.2021-09-28-0000
679 postgresql.log.2021-09-28-0100
  0 postgresql.log.2021-09-28-0200
  0 postgresql.log.2021-09-28-0300
  0 postgresql.log.2021-09-28-0400
  0 postgresql.log.2021-09-28-0500
  0 postgresql.log.2021-09-28-0600
  0 postgresql.log.2021-09-28-0700
  0 postgresql.log.2021-09-28-0800
  0 postgresql.log.2021-09-28-0900
  0 postgresql.log.2021-09-28-1000
  0 postgresql.log.2021-09-28-1100
  0 postgresql.log.2021-09-28-1200
  0 postgresql.log.2021-09-28-1300
  0 postgresql.log.2021-09-28-1400
  0 postgresql.log.2021-09-28-1500
  0 postgresql.log.2021-09-28-1600
  0 postgresql.log.2021-09-28-1700
  0 postgresql.log.2021-09-28-1800
  0 postgresql.log.2021-09-28-1900
  0 postgresql.log.2021-09-28-2000
  0 postgresql.log.2021-09-28-2100
  0 postgresql.log.2021-09-28-2200
  0 postgresql.log.2021-09-28-2300

In this case we can see that only the files

348 postgresql.log.2021-09-28-0000
679 postgresql.log.2021-09-28-0100

contains 348 bytes and 679 bytes of data respectively.

How do I make it so that the python script that is being used right now would validate if the file is empty first before being uploaded?

The closest thing I have found right now is

import os
if os.stat("postgresql.log.2021-09-28-2300").st_size == 0: 
    print('empty')

but this only checks for one file at a time, and I would rather do it in the whole directory as the file names (Date in particular and time) would change.

I'm relatively new at this and this was just handed down to me at work and I'd appreciate guides on how to make it work thank you so much.

Does this answer your question? How can I iterate over files in a given directory? — mkrieger1
– mkrieger1, Commented Sep 29, 2021 at 9:19

Marius Wallraff · Accepted Answer · 2021-09-29 15:11:18Z

1

Your approach is good already, you just have to combine it with a way to check all files per directory, such as glob.glob.

from glob import glob
import os

path_of_directory = "..."
list_of_files = glob(path_of_directory+"/postgresql.log.*")
for f in list_of_files:
    if os.stat(f).st_size != 0:
        # do something with file f

edited Sep 29, 2021 at 15:11

answered Sep 29, 2021 at 9:15

Marius Wallraff

4701 gold badge8 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Gifter Villanueva Over a year ago

I'm getting an error here saying that grep is undefined, perhaps you may have an idea on how to solve that? Thank you so much I appreciate the input!

sj95126 Over a year ago

I'd imagine grep is meant to be glob.

Marius Wallraff Over a year ago

@sj95126 Right, sorry; I first wrote grep everywhere and then forgot that one in my edit.

Eshwar S R · Accepted Answer · 2021-09-29 20:16:52Z

0

You can use glob to find files following a particular pattern. Then use os.stat to find its size. See the following code for example:

import glob
import os

# TODO: Modify this accordingly
file_pattern = 'postgresql.log.2021*'

# Get a list of files (file paths) 
files = glob.glob(file_pattern) )

# Find out their sizes 
files_and_size = [ (file_path, os.stat(file_path).st_size) 
                    for file_path in files ]

# Printing the file paths and corresponding sizes
for file_path, file_size in files_and_size:
    print(file_path, '==', file_size)
    if file_size > 0:
        # process the file with path "file_path"
    else:
        # delete the file
        os.remove(file_path)

edited Sep 29, 2021 at 20:16

answered Sep 29, 2021 at 9:16

Eshwar S R

1362 silver badges7 bronze badges

2 Comments

Gifter Villanueva Over a year ago

Thanks! This gave out an output, however i'm trying to only keep the files with values that have more than 0 bytes in them so I used this instead files_and_size = [ (file_path, os.stat(file_path).st_size != 0) now i'm getting a boolean output like so: ('postgresql.log.2021-09-28-1300', '==', False) ('postgresql.log.2021-09-28-0000', '==', True) Deleted some extra outputs to make comment fit. How do I make so that I only get to work on files that are "True" and the ones with "False" are deleted?

Eshwar S R Over a year ago

@GifterVillanueva Now that you have file_paths and their corresponding sizes, add an if else condition in the last for loop. Editing the answer with the same.

Collectives™ on Stack Overflow

check multiple files if they are empty in python

2 Answers 2

3 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related