Re-running a python script that reads in files as a numpy array does not actually re-read?

Question

I am trying to read a large number of small binary format files (~200,000) as numpy arrays into a dictionary in python:

import os
import numpy as np

def readfiles(limit):
    filelist = {}
    i=1
    for filename in os.listdir('folder'):
        filelist[filename] = np.fromfile('folder/'+filename, 'float32')
        i += 1
        if i > limit:
           break

    return filelist

The limit argument is just for testing with a smaller number of files, normally I would read all the files in the folder.

The first time I run the script with a fairly large limit (90,000), it takes ~68 s. If I immediately re-run the script it runs in ~1.2 s. The cProfiles give:

>>> cProfile.run('readfiles(90000)')

90005 function calls in 68.768 seconds
Ordered by: standard name
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.284    0.284   68.690   68.690 <ipython-input-57-939c6a92cd68>:1(readfiles)
    1    0.079    0.079   68.768   68.768 <string>:1(<module>)
    1    0.000    0.000   68.768   68.768 {built-in method builtins.exec}
90000   68.313    0.001   68.313    0.001 {built-in method numpy.core.multiarray.fromfile}
    1    0.093    0.093    0.093    0.093 {built-in method posix.listdir}
    1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}


>>> cProfile.run('readfiles(90000)')

90005 function calls in 1.970 seconds
Ordered by: standard name
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    1    0.137    0.137    1.900    1.900 <ipython-input-57-939c6a92cd68>:1(readfiles)
    1    0.070    0.070    1.970    1.970 <string>:1(<module>)
    1    0.000    0.000    1.970    1.970 {built-in method builtins.exec}
90000    1.673    0.000    1.673    0.000 {built-in method numpy.core.multiarray.fromfile}
    1    0.090    0.090    0.090    0.090 {built-in method posix.listdir}
    1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Subsequently, when I rerun the script in a completely different session, I still get ~1.2s. This seems rather strange to me. It seems that np.fromfile is not truly re-reading files after it has done once, but reading of some cached files the second time. But I have not heard of cached data being reused in another session in a situation like this. Is that right? If yes, how do I change this so that the code actually re-reads the files? If not, why does the first run takes so long?

I am using Python 3.5.1 with NumPy 1.11.2

Edit: By restarting the system I get the longer runtime back, so this must be a OS-level caching, as pointed out in the comments. Any way to around that without rebooting my system?

How completely different is your new session? Because I think there is some caching going on on the OS / filesystem level which would survive, say, your just starting a new python interpreter. No expert, though. — Paul Panzer
– Paul Panzer, Commented Feb 7, 2017 at 12:35
I only closed all open terminals and terminated all interactive python sessions. I didn't reboot the system. The code is supposed to run on a cluster so rebooting is not an option, although I can try and see if that helps on my machine. — krm
– krm, Commented Feb 7, 2017 at 12:46
Yup! Restarting seems to clear whatever the cache was. I will edit the question with this information — krm
– krm, Commented Feb 7, 2017 at 12:55

krm · Accepted Answer · 2021-08-24 04:55:37Z

0

As mentioned in the comments, the following command clears all disk caches.

sync; echo 3 > /proc/sys/vm/drop_caches

answered Aug 24, 2021 at 4:55

krm

9578 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Re-running a python script that reads in files as a numpy array does not actually re-read?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related