Hadoop Streaming :Python Join different files

Question

I have a process which takes input data, processes it and outputs the data.During this it generates two logs IN.log and OUT.log

IN.log contains when the data came in and the of the data. OUT.log contains when the data was processed and the of the data. so... IN.log contains in-time id

OUT.log contains out-time id

Now, as part of processing using hadoop streams using python, I would like to join these two files and come with diff of intime and out time and the id of the data .

For eg:

2seconds id123

3seconds id112

Any pointers as to how this can be achieved using PYTHON?

What have you tried so far? What do you need hadoop, mr and other stuff for? — alko
– alko, Commented Nov 26, 2013 at 12:08
These files are going to be pretty big(few GBs),so chose the hadoop way...I was able to achieve this using Hive, but also wanted to check if hadoop streaming provides faster processing. — BJC
– BJC, Commented Nov 26, 2013 at 12:48

alko · Accepted Answer · 2013-11-26 13:00:54Z

1

Take a look at MRjob helper package for hadoop jobs running. It would be a pretty easy to write a map/reduce for this task, something along the lines of following code

from datetime import datetime
from MRJob import MRJob

class JoinJob(MRJob):
    fmt = '%Y-%M-%d'
    def steps(self):
        return [self.mr(mapper=self.mapper, 
                        reducer=self.reducer)]
    def mapper(self, rec_time, rec_id):
        yield rec_id, rec_time

    def reducer(self, rec_id, datetime_strs):
        datetimes = map(lambda x: datetime.strptime(x, self.fmt), 
                            datetime_strs)
        delta_secs = (max(datetimes) - min(datetimes)).total_seconds()
        yield rec_id, delta_secs

if __name__ == '__main__':
    JoinJob.run()

answered Nov 26, 2013 at 13:00

alko

48.7k12 gold badges99 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Hadoop Streaming :Python Join different files

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related