hive udf execution via shell script

Question

I have a Hive Udf that works well in hive terminal, What I want i want to execute it via shell script. On hive terminal i am able to execute following commands :

use mashery_db;
add jar hdfs://nameservice1/tmp/nextdata_aggregations/custom_jar/readerCheck.jar;
add file hdfs://nameservice1/tmp/GeoLite2-City.mmdb;
CREATE TEMPORARY FUNCTION geoip AS 'com.mashery.nextdata.hive.udf.GeoIPGenericUDF';

But when I am adding the above code in shell script

hive -e "use mashery_db;"
hive -e "add jar hdfs://nameservice1/tmp/nextdata_aggregations/custom_jar/readerCheck.jar;"
hive -e "add file hdfs://nameservice1/tmp/GeoLite2-City.mmdb;"
hive -e "CREATE TEMPORARY FUNCTION geoip AS 'com.mashery.nextdata.hive.udf.GeoIPGenericUDF';"

The 1st 'hive -e' works well and adds the jar but the last one create temporary function doesn't work. I am getting below error:

FAILED: ParseException line 1:35 mismatched input 'com' expecting StringLiteral near 'AS' in create function statement

I have also tried with single quotes

hive -e "CREATE TEMPORARY FUNCTION geoip AS 'com.mashery.nextdata.hive.udf.GeoIPGenericUDF';"

then I am getting FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask

FAILED: Class com.mashery.nextdata.hive.udf.GeoIPGenericUDF not found

Does hive Udf supports shell script ,if it does whats wrong I am doing. Thanks in advance

Try importing the jar and creating the function all in one call to hive. i.e. hive -e "add jar path_to_jar/foo.jar; create temporary function foo as 'com.package.UDF';" — o-90
– o-90, Commented May 13, 2016 at 18:28

Roberto Congiu · Accepted Answer · 2016-05-17 22:03:03Z

1

Each invocation of hive -e spawns a new process with a new hive shell that has no memory of what the previous one did, so hive 'forgets' where the UDF is... One solution is to chain them in just one command, but it's better form to put all your hive commands in a file (for instance "commands.hql") and use hive -f commands.hql instead of -e.

File would look like this:

use mashery_db;
add jar hdfs://nameservice1/tmp/nextdata_aggregations/custom_jar/readerCheck.jar;
add file hdfs://nameservice1/tmp/GeoLite2-City.mmdb;
CREATE TEMPORARY FUNCTION geoip AS 'com.mashery.nextdata.hive.udf.GeoIPGenericUDF';"

answered May 17, 2016 at 22:03

Roberto Congiu

5,2531 gold badge29 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Aaron Faltesek · Accepted Answer · 2016-05-18 18:58:46Z

0

You can get this to work with both hive -e and hive -f:

hive -e "use mashery_db;
add jar hdfs://nameservice1/tmp/nextdata_aggregations/custom_jar/readerCheck.jar;
add file hdfs://nameservice1/tmp/GeoLite2-City.mmdb;
CREATE TEMPORARY FUNCTION geoip AS 'com.mashery.nextdata.hive.udf.GeoIPGenericUDF';"

Creating them as a file and using hive -f hive_file.hql would work as well.

answered May 18, 2016 at 18:58

Aaron Faltesek

3491 gold badge3 silver badges11 bronze badges

Collectives™ on Stack Overflow

hive udf execution via shell script

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related