Importing large number of images into Python to convert to Numpy array

Question

I am attempting to import a large number of images and convert them into an array to do similarity comparisons between images based on colors at each pixel and shapes contained within the pictures. I'm having trouble importing the data, the following code works for small numbers of images (10-20) but fails for larger ones (my total goal is to import 10,000 for this project).

from PIL import Image
import os,os.path
imgs=[]
path="Documents/data/img"
os.listdir(path)
valid_images =[".png"]

for f in os.listdir(path):
    ext= os.path.splitext(f)[1]
    if ext.lower() not in valid_images:
        continue
    imgs.append(Image.open(os.path.join(path,f)))

When I execute this I receive the following message

OSError: [Errno 24] Too many open files: 'Documents/data/img\81395.png'

Is there a way to edit how many files can be open simultaneously? Or possibly a more efficient way to convert these tables to arrays as I go and "close" the image? I'm very new to this sort of analysis so any tips or pointers are appreciated.

Karol Żak · Accepted Answer · 2020-05-19 23:46:41Z

1

Don't store PIL.Image objects and just convert them into numpy arrays instead. For that you need to change the line where you append image to a list to this:
''' imgs.append(np.asarray(Image.open(os.path.join(path,f)))) '''

answered May 19, 2020 at 23:46

Karol Żak

2,44626 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2355903 Over a year ago

Thanks, that is super helpful! I'm a little confused at what the output structure is. It looks like a list of tuples, am I to interpret these as the RGB values for a particular pixel? If so how do I relate it back to which pixel the value is associated with?

Karol Żak Over a year ago

These are RGB values for each layer of an image. Grayscale would have a single layer so e.g. an array of shape 100x100 would represent a grayscale image of 100x100 pixels and RGB image would be an array of shape 100x100x3. And so this is what you're seing there in that list. I suggest looking up some tutorials on image manipulation in python using arrays. Something like this for example: https://scipy-lectures.org/advanced/image_processing/

Collectives™ on Stack Overflow

Importing large number of images into Python to convert to Numpy array

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related