Constructing Java hashmap from Spark dataframe

Question

I am using spark-sql-2.4.1v with Java 8 in my project.

I need to construct a loop up hashmap from given dataframe as below:

List ll = Arrays.asList(
      ("aaaa", 11),
      ("aaa", 12),
      ("aa", 13),
      ("a", 14)
    )

Dataset<Row> codeValudeDf = ll.toDF( "code", "value")

Given the above dataframe I need to create a hashmap

i.e.

Map<String, String> lookUpHm = new Hashmap<>();

lookUpHm  => aaaa->11  , aaa->12 , aa->13, a->14

How can it it be done in Java?

Som · Accepted Answer · 2020-07-18 07:56:10Z

2

Try this-

 List<Row> rows = Arrays.asList(
                RowFactory.create("aaaa", 11),
                RowFactory.create("aaa", 12),
                RowFactory.create("aa", 13),
                RowFactory.create("a", 14)
        );

        Dataset<Row> codeValudeDf = spark.createDataFrame(rows, new StructType()
                .add("code", DataTypes.StringType, true, Metadata.empty())
                .add("value", DataTypes.IntegerType, true, Metadata.empty()));
        Map<String, Integer> map = new HashMap<>();
        codeValudeDf.collectAsList().forEach(row -> map.put(row.getString(0), row.getInt(1)));

        System.out.println(map.entrySet().stream().map(e -> e.getKey() +"->"+ e.getValue())
                .collect(Collectors.joining(", ", "[ ", " ]")));
        // [ aaa->12, aa->13, a->14, aaaa->11 ]

answered Jul 18, 2020 at 7:56

Som

6,3681 gold badge13 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Roobal Jindal Over a year ago

if dataset is huge, codeValudeDf.collectAsList() will collect data in driver, that would be an issue. We want it to be distributed and map should be prepared and collected from distributed datasets. How to do it/

code.gsoni · Accepted Answer · 2020-07-17 10:42:28Z

1

Simple add a new column of type map using withColumn and do a collect on your dataframe.

codeValudeDf.withColumn("some_map",
map(col("code"), col("value"))).select("some_map").distinct().collect()

answered Jul 17, 2020 at 10:42

code.gsoni

6953 silver badges12 bronze badges

1 Comment

BdEngineer Over a year ago

getting error org.apache.spark.sql.AnalysisException: Cannot have map type columns in DataFrame which calls set operations(intersect, except, etc.), but the type of column some_map is map<string,int>;;

Collectives™ on Stack Overflow

Constructing Java hashmap from Spark dataframe

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related