Scala : How to process objects in a list

Question

I am using spark and scala on jdk1.8.I am new to Scala. I am collecting a list of objects as :

val externalEntities  =  getExternalGenericExtractFiles()

That functions returns (from Java world):

public List<GenericExtractExternalEntity> getExternalGenericExtractFiles()
{
    return externalGenericExtractFiles;
}

Now I am trying :

externalEntities.forEach( externalEntity =>
  sqlContext.read.format(externalEntity.getExtractfileType)
    .option("compression", externalEntity.getCompressionCodec)
    .option("delimiter", externalEntity.getExtractDelimiter)
    .option("header", if (externalEntity.getHasHeader.toUpperCase == "Y") "true" else "false")
    .load(externalEntity.getFilePath)
    .createOrReplaceTempView(externalEntity.getExtractName)
)

and getting error :

error: missing parameter type
[ERROR]     externalEntities.forEach(externalEntity =>

I thought Scala is capable of inferring type. What am I missing here?

If I try just one object from the list, it seems fine like if I do

val externalEntity=  getExternalGenericExtractFiles().get(0)

sqlContext.read.format(externalEntity.getExtractfileType)
    .option("compression", externalEntity.getCompressionCodec)
    .option("delimiter", externalEntity.getExtractDelimiter)
    .option("header", if (externalEntity.getHasHeader.toUpperCase == "Y") "true" else "false")
    .load(externalEntity.getFilePath)
    .createOrReplaceTempView(externalEntity.getExtractName)

May I get any insight on what is happening and/or how to resolve? Thanks in advance

That forEach you are using is actually a java forEach, not the scala one. — Ryan Stull
– Ryan Stull, Commented Jan 28, 2019 at 23:16
I tried thus in a RELP (version 2.12.8), and worked. val l: java.util.List[Integer] = new java.util.LinkedList[java.lang.Integer](), l.add(0), l.forEach(x => println(x + 1)). Can you provide a MCVE we could test? — Luis Miguel Mejía Suárez
– Luis Miguel Mejía Suárez, Commented Jan 28, 2019 at 23:22
Never mind, I tried the same in a RELP (version 2.11.12) and it has the same error. Probably it is because scala 2.12 was the first version of Scala to require a minimum version of JDK 8, which allowed them to improve compatibility with Java. — Luis Miguel Mejía Suárez
– Luis Miguel Mejía Suárez, Commented Jan 28, 2019 at 23:27

Brian McCutchon · Accepted Answer · 2019-01-29 01:26:08Z

3

I assume you are not using Scala 2.12. Using Scala lambdas with Java functional interfaces requires Scala 2.12. That error message often occurs when you pass a lambda where one is not expected, which would happen in this case in 2.11. Upgrade your Scala version, or convert the list to a Scala list.

answered Jan 29, 2019 at 1:26

Brian McCutchon

8,6344 gold badges35 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

VictorGram · Accepted Answer · 2019-01-29 16:53:16Z

0

I had a typo ("forEach"instead of "foreach") and needed little extra:

import scala.collection.JavaConverters._

val externalEntities:List[GenericExtractExternalEntity]  =  
                  getExternalGenericExtractFiles()
                  .asScala
                  .iterator
                  .toList

externalEntities.foreach ( externalEntity =>
  sqlContext.read.format(externalEntity.getExtractfileType)
    .option("compression", externalEntity.getCompressionCodec)
    .option("delimiter", externalEntity.getExtractDelimiter)
    .option("header", if (externalEntity.getHasHeader.toUpperCase == "Y") "true" else "false")
    .load(externalEntity.getFilePath)
    .createOrReplaceTempView(externalEntity.getExtractName)
)

And it worked.

edited Jan 29, 2019 at 16:53

answered Jan 29, 2019 at 16:33

VictorGram

2,65911 gold badges57 silver badges90 bronze badges

Collectives™ on Stack Overflow

Scala : How to process objects in a list

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related