perl memory usage when processing a file inline

Question

I have a CGI script that's used by our employees to fetch logs from servers that they don't have direct access to. For reasons I won't go into, after a recent update to our app some of these logs now have characters like linefeeds, tabs, backslashes, etc. translated into their text equivalents. As such, I've modified the CGI script to invoke the following to convert these back to their original values:

perl -i -pe 's/\\r/\r/g && s/\\n/\n/g && s/\\t/\t/g && s/\\\//\//g' $filename

I was just informed that some people are now getting out of memory errors when they try to fetch logs that are fairly large (a few hundred MB).

My question: How does perl manage memory when an inline command like this is invoked? Is it reading the whole file in, processing it, then writing it out, or is it creating a temporary file, processing the lines from the input file one at a time then replacing the file once complete?

This is using perl 5.10.1 on a 64-bit Amazon linux instance.

Miller · Accepted Answer · 2014-08-21 22:26:00Z

The -p switch creates a while(<>){...; print} loop to iterate on each “line” in your input file.

If all of your newlines have been converted into "\\n", then your file would just be a single very long line. Therefore, your command would be loading the entire file into memory to perform your fix.

To avoid that, you'll have to intentionally buffer the file using either sysread or $/.

It would probably be easiest to create an actual script instead of a one-liner to do the work. However, if you know that all of your newlines are converted, then one simple fix would be to use $/ = "\\n"

As a secondary note, your regex is flawed. You're currently listing out your translations s/// using a shortcut operator. If any one of the earlier regexes doesn't match for a particular line, then no other translations would be attempted. You should instead use simple semicolons to separate your regexes:

's/\\r/\r/g; s/\\n/\n/g; s/\\t/\t/g; s|\\/|/|g'

Collectives™ on Stack Overflow

perl memory usage when processing a file inline

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related