Data Analysis Scala on Spark

Question

I am new to Scala, and i have to use Scala and Spark's SQL, Mllib and GraphX in order to perform some analysis on huge data set. The analyses i want to do are:

Customer life cycle Value (CLV)
Centrality measures (degree, Eigenvector, edge-betweenness, closeness) The data is in a CSV file (60GB (3 years transnational data)) located in Hadoop cluster.

My question is about the optimal approach to access the data and perform the above calculations?

Should i load the data from the CSV file into dataframe and work on the dataframe? or
Should i load the data from the CSV file and convert it into RDD and then work on the RDD? or
Are there any other approach to access the data and perform the analyses?

Thank you so much in advance for your help..

Ged · Accepted Answer · 2019-07-24 09:26:21Z

1

Dataframe gives you sql like syntax to work with the data where as RDD gives Scala collection like methods for data manipulation.

One extra benefit with Dataframes is underlying spark system will optimise your queries just like sql query optimisation. This is not available in case of RDD's.

As you are new to Scala its highly recommended to use Dataframes API initially and then Pick up RDD API later based on requirement.

edited Jul 24, 2019 at 9:26

Ged

18.5k8 gold badges53 silver badges108 bronze badges

answered Sep 25, 2016 at 15:28

Nagarjuna Pamu

14.8k3 gold badges24 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Reactormonk Over a year ago

The somewhat typesafe version of dataframes are datasets, which are the recommended API because of better data compression. For the typesafe issue there's github.com/adelbertc/frameless

Shankar · Accepted Answer · 2016-09-25 16:47:07Z

0

You can use Databricks CSV reader api, which is easy to use and returns DataFrame. It automatically infer data types. If you pass the file with header it can automatically use that as Schema, otherwise you can construct schema using StructType.

https://github.com/databricks/spark-csv

Update: If you are using Spark 2.0 Version , by default it support CSV datasource, please see the below link.

https://spark.apache.org/releases/spark-release-2-0-0.html#new-features

See this link for how to use. https://github.com/databricks/spark-csv/issues/367

edited Sep 25, 2016 at 16:47

answered Sep 25, 2016 at 16:39

Shankar

9,02926 gold badges100 silver badges172 bronze badges

Collectives™ on Stack Overflow

Data Analysis Scala on Spark

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related