Hadoop Streaming Job failed in python

I got the same problem and was wondering because when I test my mapper and reducer on test data it run. But when I run the same test set via hadoop map-reduce, I used to get the same problem.

How to test your code locally:

cat <test file> | python mapper.py | sort | python reducer.py

On more investigation, I found that I didn't include the 'shebang line' in my mapper.py script.

#!/usr/bin/python

Please add above line as first line of your python script and leave a blank line after this.

If you need to know more about 'shebang line', please read Why do people write #!/usr/bin/env python on the first line of a Python script?

You can find the python error messages (for example traceback) and another things written by your script to stderr in hadoop web interface. It is a little bit hidden, but you will find it in link the streaming provides you with. You click to 'Map' or 'Reduce', then click on any task and then in column Task logs on 'All'

Finally I fixed the bug, and here is the lessons I learned. 1) The original code doesn't have error handling on bad data. I didn't spot the problem when I tested the code on a small dataset. 2) To handle the empty fields/variables, I found that it is a bit tricky in Python to test None and empty strings. Personally I like function len(strVar) which is easy to read and effective. 3) The hadoop command in this case is correct. Somehow the *.py with mode 644 can be run successfully in the environment I use.

Hadoop Streaming - Hadoop 1.0.x

I had the same "Broken pipe" problem. The problem was a "break" statement in my reducer. So, everything went alright until the "break". After that the running reducer stopped running printing the "Broken pipe" error. Also, another reducer started to run having the same fate with the previous. This circle was going on and on.

If I understood right, when a reducer starts to read from stdin( that was my case, in a for loop), then it has to read everything. You cannot "break" this operation even if you close the stdin( os.close( 0) as I tried).

I had the same problem today playing with Hadoop 1.0.1. Fortunately, I've solved it by:

hadoop ... -mapper $cwd/mapper.py -reducer $cwd/reducer.py ...

(My Python scripts were in the current directory). It looks like absolute paths are necessary now.

Best!

A dirty input could cause this problem.

Try to use try{} to avoid this case.

#!/usr/bin/env python
import sys
for line in sys.stdin:
    try:
         val=line.split("\t")
         (visidH,visidL,sessionID)=(val[4],val[5],val[108])
         print "%s%s\t%s" % (visidH,visidL,sessionID)
    except Exception  as e:
        pass

Python + Hadoop is tricky in some details that shouldn't be. Take a look here.

Try to enclose your input path in double quotes. (-input "/data/omni/20110115/exp6-10122")

One possible solution is including "python", i.e.:

-mapper  "python ./mapper.py"
-reducer "python ./reducer.py"