Performance of sort command in unix

Question

I am writing a custom apache log parser for my company and I noticed a performance issue that I can't explain. I have a text file log.txt with size 1.2GB.

The command: sort log.txt is up to 3 sec slower than the command: cat log.txt | sort

Does anybody know why this is happening?

Is one call taking longer than the other because the log file has grown in the interim? — johnsyweb
– johnsyweb, Commented Jun 3, 2011 at 10:43
Is it because on the second try (the cat|sort) log.txt was read from cache, not disk, so you saved on disk access, not actual sorting? please run your sort through strace -C and see how much time goes toward disk access. Or just put the file on a ramdisk and run your experiments there. — Marcin
– Marcin, Commented Jun 3, 2011 at 13:09
If it's 1.2 GB it might be worth it to try this: LC_ALL=C sort log.txt LC_COLLATE=C might also help. — yingted
– yingted, Commented Aug 9, 2011 at 19:25

dogbane · Accepted Answer · 2011-06-03 10:32:03Z

4

cat file | sort is a Useless Use of Cat.

The purpose of cat is to concatenate (or "catenate") files. If it's only one file, concatenating it with nothing at all is a waste of time, and costs you a process.

It shouldn't take longer. Are you sure your timings are right?

Please post the output of:

time sort file

and

time cat file | sort

You need to run the commands a few times and get the average.

answered Jun 3, 2011 at 10:32

dogbane

276k77 gold badges407 silver badges415 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

gpol Over a year ago

You are absolutely right! Thanx! I was only checking a single output of the time command. When ran 5 times and taking the average the version without cat seems to be approximately 2 secs faster! Thanx a lot for the link also!

ninjalj Over a year ago

@user744734: before taking the average, you should discard outlier values, in this case, at least the run that warms the cache.

Michael · Accepted Answer · 2011-06-03 10:30:12Z

1

Instead of worrying about the performance of sort instead you should change your logging:

Eliminate unnecessarily verbose output to your log.
Periodically roll the log (based on either date or size).
...fix the errors outputting to the log. ;)

Also, are you sure cat is reading the entire file? It may have a read buffer etc.

answered Jun 3, 2011 at 10:30

Michael

7,48610 gold badges53 silver badges88 bronze badges

Collectives™ on Stack Overflow

Performance of sort command in unix

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related