A best way to get list of files modified in specific time

Question

I am trying to get files created in directory between 7AM-7PM.

ERROR_FOLDER = os.path.join(ROOT_FOLDER,"Folder","archive")

for csv in os.listdir(ERROR_FOLDER):
    path = os.path.join(ERROR_FOLDER,csv)      
    filetime = datetime.fromtimestamp(
            os.path.getctime(path))   
    if (date_start < filetime < date_end):
        files.append(csv)

Its working ok-ish spending 13 sec to return the result from 6000 item. Is there any faster way you would suggest to me? Also how to turn it into list-comprehension?

 [os.path.basename(x) for x in glob.glob(os.path.join(ROOT_FOLDER,"Excavation","Archive","*")) if (date_start < filetime < date_end)]

doesnt seem to work

try with a list comprehension instead of append. It's in general faster, but I am not sure which is the bottleneck here. — Valentino
– Valentino, Commented Feb 21, 2021 at 19:23
Also comparing timestamps instead of datetime could help. Looks like you are comparing full dates anyway, not only the hours — Valentino
– Valentino, Commented Feb 21, 2021 at 19:30
@Valentino [os.path.basename(x) for x in glob.glob(os.path.join(ROOT_FOLDER,"Excavation","Archive","*")) if (date_start < filetime < date_end)] i tried this but not working, can you see any issue here ? — Royal
– Royal, Commented Feb 21, 2021 at 19:35
Better to add the code in your question, is not really readable in the comment. And add the error you get, please. Just "not working" is not really useful for debugging. — Valentino
– Valentino, Commented Feb 21, 2021 at 20:12

bobflux · Accepted Answer · 2021-02-21 19:30:34Z

1

Use os.scandir instead, and get the file times from it instead of calling getctime() on each file.

Using scandir() instead of listdir() can significantly increase the performance of code that also needs file type or file attribute information, because os.DirEntry objects expose this information if the operating system provides it when scanning a directory. All os.DirEntry methods may perform a system call, but is_dir() and is_file() usually only require a system call for symbolic links; os.DirEntry.stat() always requires a system call on Unix but only requires one for symbolic links on Windows.

answered Feb 21, 2021 at 19:30

bobflux

11.6k3 gold badges31 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ulrich Eckhardt · Accepted Answer · 2021-02-21 20:39:50Z

1

One inefficiency is that you take N date-time values and convert them, just to compare them to two other values. Convert those two other values so that you can avoid those N conversions.

Another thing is that you build this list at all. Just yield each path in turn instead of storing them in a list, which (I assume) is getting processed in sequence anyway.

This also gives way to another optimization and that is to do the scanning of the directory asynchronously. That allows you to block just one thread of execution doing the harddisk IO while a second thread is processing the results already.

answered Feb 21, 2021 at 20:39

Ulrich Eckhardt

17.7k5 gold badges31 silver badges63 bronze badges

Collectives™ on Stack Overflow

A best way to get list of files modified in specific time

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related