I am using sort -k1,1 -k2,2 srcfile> tgtfile...
But this is throwing the error "No space in device" as the src file contains 180 million rows in it.
How to fix this issue??
1 Answer
GNU sort can sort files larger than what would fit in your RAM.
It does that by sorting a part of your input that does fit, writing the result to a temporary file, moving on to the next input part, sorting that, writing it to a temporary file and so on. Afterwards, it merges all the partial sorted files into one sorted output.
The file system that these temporary files are written to needs to be large enough to keep the whole content of the input file, potentially multiple times. GNU sort is not very smart about choosing the directory the temporary files are written to - it just uses the system default (typically, /tmp).
To mitigate your problem, tell sort to put the temporary files next to the target file (they will be automatically cleaned up, anyways), using
sort --temporary-directory=. -k1,1 -k2,2 -o tgtfile srcfile
-
Using
$TMPDIR, falling back to system default like GNUsortdoes at least seems the smartest thing to do to me. I wouldn't want it to take it upon itself to start using file systems that have not been intended to store temporary data.Stéphane Chazelas– Stéphane Chazelas2023-05-24 08:08:22 +00:00Commented May 24, 2023 at 8:08 -
1Note that it's not specific to GNU sort, the original Unix
sortimplementations from the 70s already stored temporary data to disk.Stéphane Chazelas– Stéphane Chazelas2023-05-24 08:12:04 +00:00Commented May 24, 2023 at 8:12 -
@StéphaneChazelas ah! Regarding specificity to GNU: Only had GNU sort on my phone, didn't want to write things only cobbled together from online documentationMarcus Müller– Marcus Müller2023-05-24 15:40:57 +00:00Commented May 24, 2023 at 15:40
-
Regarding "smartness in temporary storage choice": a detection of tmpfs mounts, combined with a warning that sorting into temporary files on a RAM-backed mount won't help when sorting files larger than RAM, would not be that unwelcome, I'd say - it's better to sort in swap directly then to run out of tmpfs space as you run out of RAM, of to swap tmpfs.Marcus Müller– Marcus Müller2023-05-24 15:43:54 +00:00Commented May 24, 2023 at 15:43
tgtfiledoes not have enough space to save the file. Showing any information about the the space available on your devices (dh -h) could help someone in figuring out what the specific issue is.