Reading multiple lines from a text file in a loop at bash

Question

I am trying to write an archiver script in bash but there are too many files. They are realy too many.. about 1 million files.

I planned that, I will create the list of files with;

cd /path/to/log/directory/
find . -type f > logfilelist.txt

And then, I will tar and zip them with;

tar -cvf logarchive.tar $(cat logfilelist.txt) 
gzip logarchive.tar

But, because of returning too many lines from cat, the tar gives "Arg list too long" error.

So I tought that, if I can read the file in a loop, I can archive them piece by pece by using append mode of tar. But making a million-line loop is not logical. So, can I read the list file with multiple lines like this;

tar -cf logarchive.tar $(first 50000 lines of logfilelist.txt) 

for loop
do
tar -rvf logarchive.tar $(2nd,3rd,...,99th,100th 50000 lines of logfilelist.txt)  
done

is it possible to cat multiple lines from a file?

And then, I will tar and zip them with; ? Why not just do just that from the start, like cd /path/to/log/directory/ ; tar -cvf logarchive.tar .? — KamilCuk
– KamilCuk, Commented Jan 11, 2021 at 18:55

that other guy · Accepted Answer · 2021-01-11 18:48:32Z

1

You can use mapfile which allows you to easily read lines and call a function after every N:

#!/bin/bash
archive() {
  tar -rvf logarchive.tar "${lines[@]}"
  lines=()
}
mapfile -t -c 50000 -C archive lines < logfilelist.txt
archive # Also call it for the last <50,000 lines

PS: This is an XYProblem in place of "How do I add a large number of files from a list to a tar archive?", and the answer to that is instead a much easier and better

tar -cf logarchive.tar --files-from logfilelist.txt

answered Jan 11, 2021 at 18:48

that other guy

125k12 gold badges188 silver badges214 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

KatsutoshiOtogawa · Accepted Answer · 2021-01-12 20:21:00Z

0

You use xargs. This script add tar archive by row.

# append and create tar.
cat logfilelist.txt | xargs -n 1 -I {} tar -rvf logarchive.tar {}
# gzip tar achive.
gzip logarchive.tar

answered Jan 12, 2021 at 20:21

KatsutoshiOtogawa

312 bronze badges

Collectives™ on Stack Overflow

Reading multiple lines from a text file in a loop at bash

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related