18

I am trying to sort some simple pipe-delimited data. However, sort isn't actually sorting. It moves my header row to the bottom, but my two rows starting with 241 are being split by a row starting with 24.

cat sort_fail.csv
column_a|column_b|column_c
241|212|20810378
24|121|2810172
241|213|20810376

sort sort_fail.csv
241|212|20810378
24|121|2810172
241|213|20810376
column_a|column_b|column_c

The column headers are being moved to the bottom of the file, so sort is clearly processing it. But, the actual values aren't being sorted like I'd expect.

In this case I worked around it with 

sort sort_fail.csv --field-separator='|' -k1,1

But, I feel like that shouldn't be necessary. Why is sort not sorting?

Improve this question
2
  • 2
    use LC_COLLATE=C sort. Depending on what you're expecting, you may also need LC_COLLATE=C sort -t'|' -n Commented Dec 11, 2018 at 22:49
  • 3
    To sort "csv style" data you may want to use csvsort from csvkit, which properly handles quoted values. Commented Dec 12, 2018 at 8:17

3 Answers 3

Reset to default
33

sort is locale aware, so depending on your LC_COLLATE setting (which is inherited from LANG) you may get different results:

$ LANG=C sort sort_fail.csv 
241|212|20810378
241|213|20810376
24|121|2810172
column_a|column_b|column_c

$ LANG=en_US sort sort_fail.csv
241|212|20810378
24|121|2810172
241|213|20810376
column_a|column_b|column_c

This can cause problems in scripts, because you may not be aware of what the calling locale is set to, and so may get different results.

It's not uncommon for scripts to force the setting needed

e.g.

$ grep 'LC.*sort' /bin/precat
      LC_COLLATE=C sort -u | prezip-bin -z "$cmd: $2"

Now what's interesting, here, is the | character looks odd.

But that's because the default rule for en_US, which derives from ISO, says

$ grep 007C /usr/share/i18n/locales/iso14651_t1_common
<U007C> IGNORE;IGNORE;IGNORE;<j> # 142 |

Which means the | character is ignored and the sort order would be as if the character doesn't exist..

$ tr -d '|' < sort_fail.csv | LANG=C sort
24121220810378
241212810172
24121320810376
column_acolumn_bcolumn_c

And that matches the "unexpected" sorting you are seeing.

The work arounds are to use -n (to force numeric sorts), or to use the field separator (as you did) or to use the C locale.

Improve this answer
5
  • Fascinating. I did see some other hits about localization, but figured that would impact the relative ordering of 24 vs 241, not something like this. Commented Dec 12, 2018 at 0:24
  • 7
    something extra useful in GNU sort is the --debug option, which indicates the key (underlined) used to compare Commented Dec 12, 2018 at 3:36
  • running with --debug just underlines the entire line - sort is including the pipe character, it's just set to have no impact because of localization. It's a good feature but didn't help me in this case (I tried :) Commented Dec 12, 2018 at 17:37
  • That's exactly why I mentioned it, @user10777668 -- it indicates that sort is using the entire line instead of stopping at characters that we assume it is. Commented Dec 12, 2018 at 18:03
  • I wasn't expecting it to stop. I was expecting it to recognize the pipe character and include that in the sort, therefore treating 24|1 and 241 differently. I'm not sure how --debug would have changed that, and in fact given that it underlines the | seems like it would have actively distracted from the real problem where localization led to the pipe character being ingored Commented Dec 12, 2018 at 18:30
1

What irritates me is that the 24 doesn't move from its place between the two 241. The second field starts with a 1. Trying the sort with a leading 4 in the second field, the 24 is moved down, so I suspect sort just ignores the | unless told otherwise. Try sort -n...

Improve this answer
1

-n, --numeric-sort compare according to string numerical value

210
23

Without the -n, 210 by text is ahead of 23 as it goes character my character.

Improve this answer
1
  • You're right, but this doesn't explain the pipe char being different. The other answers show that because of the locale, the pipe is treated as not there so the next digit is what decides the ordering. Commented Dec 12, 2018 at 6:59

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.