Reading Irregular Text Files with MATLAB

Note:

There are a quite a few improvments you can make for speed but there are also corrections. You preallocate you final output variable with hard coded values:

pan_mat_2 = zeros(2359,4126);

But later you populate it in a loop which run for i = 1:length.

length is the full number of lines picked from the file. In your example file there are only 784 lines. So even if all your line were valid (ok to be parsed), you would only ever fill the first 784 lines of the total 2359 lines you allocated in your pan_mat_2. In practice, this file has only 400 valid data lines, so your pan_mat_2 could definitely be smaller.

I know you couldn't know you had only 400 line parsed before you parsed them, but you knew from the beginning that you had only 784 line to parse (you had the info in the variable length). So in case like these pre-allocate to 784 and only later discard the empty lines.

Fortunately, the solution I propose does not need to pre-allocate larger then discard. The matrices will end up the right size from the start.

The code:

%%
file_pan = 'Pandora107s1_BostonMA_20190814_L0.txt' ;
delimiterIn = '\t';
headerlinesIn = 41;
A = readtable(file_pan,'HeaderLines', 41, 'Delimiter', '\t'); %Reading the file as a table
A = table2cell(A); % Converting file to a cell
A = regexp(A, ' ', 'split'); % converting cell to a structure matrix.

%% Remove lines which won't be parsed
% Count the number of elements in each line
nelem = cell2mat( cellfun( @size , A  ,'UniformOutput',0) ) ;
nelem(:,1) = [] ;
% find which lines does not have enough elements to be parsed
idxLine2Remove = ~(nelem > 4120 & nelem < 4140) ;
% remove them from the data set
A(idxLine2Remove) = [] ;

%% Remove nesting in cell array
nLinesToParse = size(A,1) ;
A = reshape( [A{:}] , [], nLinesToParse ).' ;
% now you have a cell array of size [400x4126] cells

%% Now separate the columns with different data type
% Column 1          => [String] identifier
% Column 2          => Timestamp
% Column 3 to 4125  => Numeric values
% Column 4126       => empty cell created during the 'split' operation above
%                      because of a trailing space character.
LineIDs    = A(:,1) ;
TimeStamps = A(:,2) ;
Data       = A(:,3:end-1)  ; % fetch to "end-1" to discard last empty column

%% now extract the values

% You could do that directly:
%     pan_mat = str2double(Data) ;

% but this takes a long time. A much computationnaly faster way (even if it
% uses more complex code) would be:
dat = strjoin(Data) ;                                       % create a single long string made of all the strings in all the cells
nums = textscan( dat , '%f' , Inf ) ;                       % call textscan on it (way faster than str2double() )
pan_mat = reshape( cell2mat( nums ) , nLinesToParse ,[] ) ; % reshape to original dimensions

%% timestamps
% convert to character array
strTimeStamps = char(TimeStamps) ;
% convert to matlab own datetime numbering. This will be a lot faster if
% you have operations to do on the time stamps later
ts = datenum(strTimeStamps,'yyyymmddTHHMMSSZ') ;


%% If you really want them the way you had it in your example
strTimeStamps(:,9)   = ' ' ; % replace 'T' with ' '
strTimeStamps(:,end) = ' ' ; % replace 'Z' characters with ' '
%then same again, merge into a long string, parse then reshape accordingly
strdate = reshape(strTimeStamps.',1,[]) ;
tmp = textscan( strdate , '%d' , Inf ) ;
datetime_mat = reshape( double(cell2mat(tmp)),2,[]).' ;

The performance:

As you can see on my machine your original code takes ~102 seconds to execute, with 80% of that (81s) spent on calling the function str2double() 3,302,400 times!

My solution, run on the same input file, takes ~5.5 seconds, with half of the time spent on calling strjoin() 3 times.

When you read the code above, try to understand how I limited the repetition of function call in lengthy loops by trying to keep everything as vectorised as possible.