5

I would like to concatenate multiple files following a specific order from an other file.

I have multiple files called freq_<something> that I want to concatenate. The "something" are listed in another file called "list".

So here is my list:

$ cat list
003137F
002980F
002993F

I want to do:

cat freq_003137F freq_002980F freq_002993F > freq_all

But my list contains hundreds of values so I can't really do that!

What is a way to automate it? I thought I could append a file with a while read line but it fails...

Thanks!

M

Improve this question
1
  • Usually a ?while read-type loop should be fine here. Care to add your code so we might take a look? Commented May 19, 2021 at 9:27

3 Answers 3

Reset to default
11

Use xargs

xargs -i cat freq_'{}' < list > freq_all
Improve this answer
2
  • 3
    Use uppercase -I instead, as -i option is deprecated; so xargs -I{} cat freq_'{}' < list Commented May 19, 2021 at 9:38
  • 2
    Unfortunately this stops xargs from putting more than 1 arg on each cat command line, so there's a fork/exec of cat for each file, not just cat going through its arg list doing open/read. sed 's/^/freq_/' list | xargs -d'\n' cat would be more efficient. Commented May 19, 2021 at 21:53
7

You can do it with a while, why not? This should work:

while read suffix; do cat freq_"${suffix}"; done < list > freq_all; done

Alternatively, you can generate the command with printf and run it manually:

$ echo "cat $(printf 'freq_%s ' $(cat list)) > freq_out"
cat freq_003137F freq_002980F freq_002993F > freq_out

Or simply execute it directly:

cat $(printf 'freq_%s ' $(cat list)) > freq_out
Improve this answer
5
  • 2
    The latter two might fail with very big lists, though. You might want to mention that limitation. Commented May 19, 2021 at 9:50
  • @FelixJN when would they fail? There is no globbing here, so we shouldn't hit the MAX_ARGS and I tested with a file with 100000000 lines (cat $(printf 'freq_%s ' $(cat aa))>f 2>/dev/null). Commented May 19, 2021 at 10:07
  • 2
    Yes, I thought about the MAX_ARGS limit. I am baffled it would not take effect here. Commented May 19, 2021 at 11:36
  • 2
    It's not globbing per-se that's relevant, it's a kernel-defined limit on how big an argv[] array you can pass through an execve() system call (man7.org/linux/man-pages/man2/execve.2.html). You're running cat with those args, not a shell built-in, so it can't hide this. Agreed with @FelixJN; if you still have that test setup, you could run strace -o cat.tr cat $(printf ...) to see if it's actually getting invoked with that many args. Note that the actual limit on Linux is a few MiB, much higher than the guaranteed minimum from a .h file. Commented May 19, 2021 at 21:44
  • Fun fact: you can optimize away the cat list by using $( <list) to have bash just read the file itself, like an implicit cat in a command substitution with a redirect but not command. Commented May 19, 2021 at 21:45
2

If you're going to use xargs, generally you want to take advantage of its ability to put more than 1 arg on the command line of a single process. (Running one cat that loops over its args is more efficient than running a separate cat process for each file, given process startup overhead).

xargs '-I{}' cat freq_'{}' unfortunately implies -x -L 1, and refuses to allow overriding it back to doing multiple args per command.

Instead, we can use sed to do the substitution. This could be good if you have a very large list, which might be too large for the bash command-substitution with printf in @terdon's answer. xargs knows to batch args into chunks of up to 128kiB of text (by default).

sed 's/^/freq_/' list | xargs -d'\n' cat   > freq_out

s/^/foo/ prepends stuff to the start of a line. The pattern matches 0 width at the start of each line.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.