How can I split and write the output of my command in Linux bash, to multiple (text) 5M files (number of files does not matter)?
0
Add a comment
|
2 Answers
Pipe its output to split.
Example:
$ my-command-name-here | split -a 3 -b 5g - myFile.
will produce files of 5GB having the names myFile.aaa, myFile.aab, myFile.aac a.s.o.
Use -l instead of -b to produce files with the specified number of lines instead of bytes.
Read man split or the online documentation.
7 Comments
hashbrown
Good answer! +1. Question - how many files will be produced in this case?
axiac
The option
-a tells it to use the specified number of letters (small caps a to z) to generate the names. Using -a 3 provides 26*26*26 unique file names.
UserYmY
@axiac Vey good answer, Can I also get files with numbers like :myfile 1, myfile 2?
axiac
Use
-d for numeric suffixes instead of alphabetic.UserYmY
@axiac I keep getting this error that I am not sure why: my command| split -a 3 -d -b 500M /directory for output split: invalid option -- 'm' Try `split --help' for more information.
|
Another answer that adds the --filter option to support the request from this comment:
my-command | split -b 500m -d --filter='cat > $FILE; zip -m $FILE.zip $FILE' - myFile.
This tells split to use the command provided in parameter --filter to produce the files.
The command dumps the data it receives at stdin into the file $FILE (the variable is set by split with the name of the file it computed) then asks zip to move the file into the archive $FILE.zip.
