2

Hive Version 2.1.1

Problem Description: collection items terminated values are inserted as Map Keys

Hive Table:

CREATE TABLE profiles(
id int,
name struct<first_name: string, middle_name: string, last_name: string>,
phone struct<home: string, office: string>,
address map<string,struct<streat:string, appartment:int, zip:string>>
) 
ROW FORMAT DELIMITED 
FIELDS TERMINATED BY ','
COLLECTION ITEMS TERMINATED BY '-'
MAP KEYS TERMINATED BY '='
LINES TERMINATED BY '\n'
STORED AS TEXTFILE;

Data:

1000,Suresh--S,1234567890-1234567890,home=Venkatapuram1-2020-500001
1001,Mahesh-X-M,1234567890-1234567890,home=Venkatapuram2-2021-500001

Data Load:

load data inpath '/handson/profiles_data.txt' overwrite into table profiles;

Actual data from select statement:

SELECT * FROM profiles; 

1000        
{"first_name":"Suresh","middle_name":"","last_name":"S"}        
{"home":"1234567890","office":"1234567890"}     
{"home": 
{"streat":"Venkatapuram1",**"appartment":null,"zip":null},"2020":null, 
"500001": null}

1001        
{"first_name":"Mahesh","middle_name":"X","last_name":"M"}        
{"home":"1234567890","office":"1234567890"}
{"home": 
{"streat":"Venkatapuram2",**"appartment":null,"zip":null},"2021":null, 
"500001": null}

Expected:

1000        
{"first_name":"Suresh","middle_name":"","last_name":"S"}        
{"home":"1234567890","office":"1234567890"}
{"home":{"streat":"Venkatapuram1",**"appartment":2020,"zip":"500001"}**}

1001        
{"first_name":"Mahesh","middle_name":"X","last_name":"M"}        
{"home":"1234567890","office":"1234567890"} 
{"home": {"streat":"Venkatapuram2",**"appartment":2021,"zip":"500001"**}}

1 Answer 1

0

As answered in: HIVE nested ARRAY in MAP data type, you can only override the first three delimiters in hive, while hive actually supports 8. In nested data structures, for each nesting level, a consequent delimiter is used.

In your hive table, the delimiter between fields in the struct that is inside the address map is \u004 (Unicode 4), and it can't be overridden.

You should change your input to:

1000,Suresh--S,1234567890-1234567890,home=Venkatapuram1\u00042020\u0004500001 
1001,Mahesh-X-M,1234567890-1234567890,home=Venkatapuram2\u00042021\u0004500001
Sign up to request clarification or add additional context in comments.

4 Comments

not working hive> select * from test_stg; OK 1 JOHN ["abu1","abu2"] {"key1":[null],"key2":[null]} Time taken: 0.126 seconds, Fetched: 1 row(s)
Still not working: hive> select * from profiles; OK 1000 {"first_name":"Suresh","middle_name":"","last_name":"S"} {"home":"1234567890","office":"1234567890"} {"home":{"streat":"Venkatapuram1\\u00042020\\u0004500001 ","appartment":null,"zip":null}} 1001 {"first_name":"Mahesh","middle_name":"X","last_name":"M"} {"home":"1234567890","office":"1234567890"} {"home":{"streat":"Venkatapuram2\\u00042021\\u0004500001","appartment":null,"zip":null}} Time taken: 0.083 seconds, Fetched: 2 row(s) hive>
It seems that you added \u0004 as text. You should insert the actual Unicode character
Appreciated, working awesome :) hive> select * from profiles; OK 1000 {"first_name":"Suresh","middle_name":"","last_name":"S"} {"home":"1234567890","office":"1234567890"} {"home":{"streat":"Venkatapuram1","appartment":2020,"zip":"500001 "}} 1001 {"first_name":"Mahesh","middle_name":"X","last_name":"M"} {"home":"1234567890","office":"1234567890"} {"home":{"streat":"Venkatapuram2","appartment":2021,"zip":"500001"}} Time taken: 0.051 seconds, Fetched: 2 row(s) hive>

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.