parse a text file in shell script

Question

I am running a hadoop command and saving the output to a txt file. I just want a specific part of the hadoop command to be in the text file.

#!/bin/sh

hadoop fs -ls /projects/abcd/ | egrep 'delta|snapshot' > /homes/abcd/tools/list_of_feeds.txt

The contents of 'list_of_feeds.txt' -

drwxr-x---   - abcd users                  0 2013-02-05 12:16 /projects/abcd/test1_delta
drwxr-x---   - abcd users                  0 2013-02-06 00:21 /projects/abcd/test2_snapshot

I just want the contents of the file to be -

test1_delta
test2_snapshot

how do i parse this file or do this in shell?

Thanks

well i wrote the for loop to try to parse the text file for the output i want...$10 is wrong there... @ooga — xyzzz
– xyzzz, Commented Jun 5, 2014 at 22:46

ooga · Accepted Answer · 2014-06-05 22:46:58Z

2

awk is useful (and easy) for selecting fields.

hadoop fs -ls /projects/abcd/ | awk -F '/' '/delta|snapshot/ {print $NF}' > /homes/abcd/tools/list_of_feeds.txt

I have no idea what your for loop is doing, however.

$NF selects the last field since it would be kind of stupid to hardcode the number of components in the pathname.

And obviously egrep is unnecessary in conjunction with awk.

answered Jun 5, 2014 at 22:46

ooga

15.6k2 gold badges23 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

xyzzz Over a year ago

Nice, I got it man. Yes, the for loop was a lame attempt - was trying out options. I am just starting to learn awk.

Stephen · Accepted Answer · 2014-06-05 22:46:26Z

-1

You can pipe your command into awk. Use the following:

#!/bin/sh

hadoop fs -ls /projects/abcd/ | egrep 'delta|snapshot' | awk -F '/' '{print $4} > /homes/abcd/tools/list_of_feeds.txt 
cat /homes/abcd/tools/list_of_feeds.txt

edited Jun 5, 2014 at 22:46

answered Jun 5, 2014 at 22:26

Stephen

631 silver badge7 bronze badges

6 Comments

Stephen Over a year ago

Sorry, had to edit that a few times since I put the ask in the wrong place, but that should work. Otherwise, let me know.

Stephen Over a year ago

Ya, I just copied his syntax and inserted the awk. Adjusted it to work correctly.

ooga Over a year ago

In what sense does it "work correctly"? Did you even test it?

Stephen Over a year ago

Indeed I did. Based on the output his command is already printing. I tested myself that the awk would parse out the specific info he wanted.

xyzzz Over a year ago

@Stephen - i guess this will not work...and plus egrep and awk would not work in conjunction..but thanks for posting.

|

Collectives™ on Stack Overflow

parse a text file in shell script

2 Answers 2

1 Comment

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related