Read a csv file using scala and generate analytics

Question

I have started learning scala and I have tryed to solve a scenario as below. I have an input file with multiple transactions separated by ','. Below are my sample values:

transactionId, accountId, transactionDay, category, transactionAmount

A11,A45,1,SA,340
A12,A2,1,FD,567

and I have to calculate the total transaction value for all transactions for each day along with other statistics. Below is my initial snippet

import scala.io.Source 
val fileName = "<path of input file>" 
Transaction( 
  transactionId: String, accountId: String, 
  transactionDay: Int, category: String, 
  transactionAmount: Double)   
Source.fromFile(fileName).getLines().drop(1) 
val transactions: List[Transaction] = transactionslines.map { line => 
  val split = line.split(',') Transaction(split(0), split(1), split(2).toInt, split(3), split(4).toDouble) }.toList

@Scalway,for your reference I have highlighted my query in bold.Thanks Luis,but if you could elaborate a bit then it would help me understanding it better. — Durga
– Durga, Commented Jan 17, 2020 at 17:29
@Durga give it a look to the Scaladoc for List it has a lot of useful methods. Particularly for this problem you only need three groupBy, map & reduce. Or if you can use Scala 2.13 then you can just use gruopMapReduce. — Luis Miguel Mejía Suárez
– Luis Miguel Mejía Suárez, Commented Jan 17, 2020 at 17:53

Subash · Accepted Answer · 2020-01-18 08:10:19Z

2

You can do it as below:

val sd=transactions.groupBy(_.transactionDay).mapValues(_.map(_.transactionAmount).sum)

Further ,you can do complex analytics by converting it into a dataframe.

val scalatoDF = spark.sparkContext.parallelize(transactions).toDF("transactionId","accountId","transactionDay","category","transactionAmount")

scalatoDF.show()

Hope this helps!

answered Jan 18, 2020 at 8:10

Subash

8951 gold badge8 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Read a csv file using scala and generate analytics

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related