running awk within a bash script

Question

Is there a way to run a awk script within bash script? I have a large file (~40GB) that I want to split based on 3rd field. The third field can be either chr1, chr2 ... chr22, chrX and chrY (total 24 types). When I run

awk 'BEGIN{OFS=FS="\t"}$3=="chr1"{print $0}' inputfile.txt > inputfile_chr1.txt

It runs fine but when I try to loop it doesn't:

for i in {1..22} X Y; do 
awk 'BEGIN{OFS=FS="\t"}$3=="chr${i}"{print $0}' inputfile.txt > inputfile_chr${i}.txt
done

I tried using single quotes for $3 and back slash to escape $3 but everything failed. Is there a better way to do this?

Chris Seymour · Accepted Answer · 2013-04-29 16:42:25Z

3

You don't want to use your current bash approach. You are reading the 40GB inputfile.txt 24 times! Just simply parse the file once with awk:

awk '{file="inputfile_"$3".txt";print >> file;close(file)}' inputfile.txt

Demo:

$ ls
inputfile.txt

$ cat inputfile.txt 
1 foo chr1
2 bar chr1
3 abc chr2
4 zyz chr3
5 123 chr2

$ awk '{file="inputfile_"$3".txt";print >> file;close(file)}' inputfile.txt

$ ls
inputfile_chr1.txt  inputfile_chr2.txt  inputfile_chr3.txt  inputfile.txt

$ cat inputfile_chr1.txt 
1 foo chr1
2 bar chr1

$ cat inputfile_chr2.txt 
3 abc chr2
5 123 chr2

$ cat inputfile_chr3.txt 
4 zyz chr3

edited Apr 29, 2013 at 16:42

answered Apr 29, 2013 at 16:28

Chris Seymour

86.4k32 gold badges166 silver badges209 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

djechlin · Accepted Answer · 2013-04-29 16:26:12Z

1

Looks like you just need to dig out the i.

'BEGIN{OFS=FS="\t"}$3=="chr'${i}'"{print $0}'

answered Apr 29, 2013 at 16:26

djechlin

61.2k40 gold badges173 silver badges300 bronze badges

3 Comments

Chris Seymour Over a year ago

Whilst this may be a quick fix it really isn't the fix the OP wants. There current approach would read that 40GB input file 22 times. Not to mention this isn't the way shell variables values should be pass to awk.

arnstrm Over a year ago

@djechlin: Thanks very much! so if I understood correctly: If 2 single quotes are used, it will expand variable but if I use 1 it won't, right?

arnstrm Over a year ago

@sudo_O: can you please suggest a better way? I greatly appreciate any help. Thanks.

Eran Ben-Natan · Accepted Answer · 2013-04-30 15:04:59Z

0

Or, and in my opinion better, pass i as var:

for i in {1..22} X Y; do 
awk -v i=$i 'BEGIN{OFS=FS="\t"}$3=="chr" i {print $0}' inputfile.txt > inputfile_chr${i}.txt
done

answered Apr 30, 2013 at 15:04

Eran Ben-Natan

2,6152 gold badges19 silver badges19 bronze badges

Collectives™ on Stack Overflow

running awk within a bash script

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related