0

I have got twitter data using flume on HDFS. Have 3 node cluster and MySQL Metastore for hive.

When i execute below query

select user_name.screen_name, user_name.followers_count from BBMP_election_tweets limit 10;

It is giving proper results.

But if i run below query

select user_name.screen_name, user_name.followers_count c from BBMP_election_tweets order by c desc;

It throws below error

Query ID = hduser_20150827102628_15a405ae-59a4-4c90-aa47-045d58c62f02 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapreduce.job.reduces= Starting Job = job_1439458788229_0016, Tracking URL = http://hdmaster:8088/proxy/application_1439458788229_0016/ Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1439458788229_0016 Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1 2015-08-27 10:27:33,567 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:27:43,908 Stage-1 map = 50%, reduce = 0% 2015-08-27 10:27:45,015 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:28:31,011 Stage-1 map = 100%, reduce = 100% Ended Job = job_1439458788229_0016 with errors Error during job, obtaining debugging information... Examining task ID: task_1439458788229_0016_m_000001 (and more) from job job_1439458788229_0016

Task with the most failures(4): ----- Task ID: task_1439458788229_0016_m_000001

URL:
http://hdmaster:8088/taskdetails.jsp?jobid=job_1439458788229_0016&tipid=task_1439458788229_0016_m_000001 ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec

When checked the logs created. It gives below message

2015-08-27 10:28:48,852 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1439458788229_0016, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@988145f) 2015-08-27 10:28:49,023 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-08-27 10:28:49,627 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1439458788229_0016 2015-08-27 10:28:50,363 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 2015-08-27 10:28:51,186 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-08-27 10:28:51,534 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685:0+52974,/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484188899:0+20119InputFormatClass: org.apache.hadoop.mapred.TextInputFormat

2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: ***************non-local mode*************** 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: local path = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Open file to read in plan: hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,719 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:51,720 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing MapWork via kryo 2015-08-27 10:28:52,043 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file hdfs://hdmaster:8020/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600

2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[3] 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 3 MAP 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Operator 3 MAP initialized 2015-08-27 10:28:52,262 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more

2015-08-27 10:28:52,269 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped. 2015-08-27 10:28:52,279 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.

Please suggest. How can this issue be resolved. and why is it occuring.

Thanks

Improve this question

1 Answer 1

Reset to default
0

It is a classpath problem Root cause is java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe

you can have a look here to add hive-serdes: https://github.com/cloudera/cdh-twitter-example

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.