FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask during join operation in hive

Question

I am trying to run the join query in hive on the following two tables-

select b.location from user_activity_rule a inner join user_info_rule b where a.uid=b.uid and a.cancellation=true;

Query ID = username_20180530154141_0a187506-7aca-442a-8310-582d335ad78d
Total jobs = 1
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Execution log at: /tmp/username/username_20180530154141_0a187506-7aca-442a-8310-582d335ad78d.log
2018-05-30 03:41:51     Starting to launch local task to process map join;      maximum memory = 2058354688
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-4

Logs:

/tmp/username/hive.log
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

What does this error mean and how to resolve this?

Can we see /tmp/username/username_20180530154141_0a187506-7aca-442a-8310-582d335ad78d.log? Or if you're using YARN, did you look at the results there? — OneCricketeer
– OneCricketeer, Commented May 30, 2018 at 12:30

Batman · Accepted Answer · 2018-05-31 08:07:38Z

2

This happens when the job you are trying to run runs out of memory. one way of overcoming this is to use this command:

set hive.auto.convert.join = false;

This will help in join optimization.

Sometimes when the number of concurrent users using it is high(at some peak time), this happens. Alternatively, you can fire this query when not many users are using it. Apparently, there would be much free memory so that your job can consume required. This alternative can be adopted when nodes are less in Dev environment and you are sure that there will be no memory issues in production.

answered May 31, 2018 at 8:07

Batman

508 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rohit Nimmala · Accepted Answer · 2018-05-30 18:56:01Z

0

Instead of where can you use the below code and try

SELECT b.location FROM user_activity_rule a JOIN user_info_rule b ON(a.uid=b.uid) WHERE a.cancellation="true";

answered May 30, 2018 at 18:56

Rohit Nimmala

1,55912 silver badges28 bronze badges

Comments

MLavoie · Accepted Answer · 2018-07-20 10:28:48Z

0

First of all, make sure the HADOOP_USER that you used to run SQL can run MapReduce.

Then, use SQL like following:

set hive.auto.convert.join = false;
select b.location 
from user_activity_rule a 
inner join user_info_rule b 
where a.uid=b.uid and a.cancellation=true;

edited Jul 20, 2018 at 10:28

MLavoie

9,91641 gold badges42 silver badges58 bronze badges

answered May 31, 2018 at 8:14

Rearchy

765 bronze badges

Collectives™ on Stack Overflow

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask during join operation in hive

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related