converting many text files into cell array in matlab

Question

I have 812 text files in one folder and 649 text files in another folder(these text files are image descriptors), and each of text files contains about 3000 numbers with this pattern: first 5 numbers are location of descriptors and 128 next numbers are values that I want to save them as a column in cell array, and this pattern repeats till the end of text file. and my goal is extracting all descriptors in a 128*n cell array, in which n is the number of descriptors for all images. here is my code for extracting all descriptors of all text files in one cell array

function cel = affinedesc(fname)


FID = fopen(fname, 'r');
content = textscan(FID, '%s');
content = content{1,1};
cel = cell(1,str2num(content{2,1}));
content = content(3:end);
  fclose(FID);

counter = 1;
for i=1:133:length(content)-1
t1 = i+5;
t2 = i+4+128;
cel{counter} = content(t1:t2);
counter = counter+1;


end

cel = cat(2,cel{:});
end

function descscel = affinedescs(dir) 

desccel = {};
for i=1:length(dir)
fname = dir(i).name;
cel = affinedesc(fname);
desccel{i} = cel;
end
descscel = cat(2,desccel{:});%here my pc freezes!

end

now here is my question: it works correct but the final cell doesn't appear in matlab workspace and I can't save the final cell for all text files that it is concatenation of all cell of all text files, and my PC screen freezes. I think it's because my final cell array is too LARGE, I wanted to know if there is a better way?

any help is appreciated!

Do you need all the data available in RAM at the same time? If not, save the cell arrays to different 'mat' files during the processing, if yes, consider working with data types that consume less memory and work faster in Matlab, like ND array of uint (or whatever fits your numbers). — EBH
– EBH, Commented Jan 3, 2019 at 21:43
I don't need them at the same time, but I need the final cell array containing all data in another process. if I save different mat files. finally I have to concatenate them!; I can't understand your mean by ND array of unit — AZiZA Saber
– AZiZA Saber, Commented Jan 6, 2019 at 14:47
and I can't save them in different mat files, because each mat file is about 400 MB! then 800 mat file will be about 320GB — AZiZA Saber
– AZiZA Saber, Commented Jan 6, 2019 at 15:06
Take a look at these data types and choose the one that fits your numbers (without loss of data). Try to save several variables as one of these types, and see how much you can reduce the file size. — EBH
– EBH, Commented Jan 6, 2019 at 19:02
It looks like you're reading in and leaving most of these values as strings instead of converting them to numbers at all? Numbers of any type would be more memory-efficient. And is there any chance you could post example data files so we could reproduce this ourselves to help you debug? — Andrew Janke
– Andrew Janke, Commented Jan 7, 2019 at 10:13

JockeR · Accepted Answer · 2019-01-04 08:27:01Z

1

Can you verify that your computer memory runs out? (For example via Task Manager in Windows.)

If it is a memory problem, try avoiding dynamically growing your arrays/cells inside the loops. Pre-allocate memory by defining a null-variable of the correct size. The cell array does not require contiguous memory but each cell does. Read more here

Also, there's a typo in the line that freezes your computer. Is the descel variable created elsewhere?

I'm aware that this might not qualify as an answer, but I haven't got enough reputation to post comments.

answered Jan 4, 2019 at 8:27

JockeR

111 bronze badge

Sign up to request clarification or add additional context in comments.

5 Comments

AZiZA Saber Over a year ago

I think my memory runs out!. i didn't understand preallocating because I don't know the size that I need for my cell array!. would you please guide more?

AZiZA Saber Over a year ago

No desccel is just defined in this function

EBH Over a year ago

@AZiZASaber you don't have to know the size of each cell array, only the largest among them.

JockeR Over a year ago

@AZiZASaber Maybe you should look closer at your variable naming. Currently you have 3 variables with similar names: desccel, descscel and descel

AZiZA Saber Over a year ago

its desccel not descel. I edited it. and descscel is the final cell containing all columns of all cells, and desccel is all columns of one text file.

Collectives™ on Stack Overflow

converting many text files into cell array in matlab

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related