scala - getting java.util.NoSuchElementException

Question

getting java.util.NoSuchElementException

val data3 = data2.filter(x => type1_dm_med.contains(x._2._1.last.medicine))

when I try to print data3.count I get below error - not sure how to fix it, please help

[Stage 50:=====> (20 + 1) / 200]16/02/26 01:37:11 ERROR Executor: Exception in task 22.0 in stage 50.0 (TID 1636)
java.util.NoSuchElementException
at scala.collection.LinearSeqOptimized$class.last(LinearSeqOptimized.scala:135)
at scala.collection.immutable.List.last(List.scala:84)

more context:

 class Diagnostic(patientID:String, date: Date, code: String)
 class Medication(patientID: String, date: Date, medicine: String)

loaded data from .csv into medication:RDD[Medication] and diagnostic:
RDD[Diagnostic]

val mpairs = medication.map(x=>(x.patientID,x))
val dpairs = diagnostic.map(x=>(x.patientID,x))
val data = mpairs.fullOuterJoin(dpairs)

'data' tuple example: (000496120-01,(Some(Medication(000496120-01,Fri Jul 01 15:15:00 EDT 2005,protamine sulfate)),Some(Diagnostic(000496120-01,Mon Jan 07 15:00:00 EST 2013,v70.0))))

type1_dm_dx et'al are list of code or medicines that want to filter on

val cond1 = data.filter(x => type1_dm_dx.contains(x._2._2.last.code))
val data1 = data.subtractByKey(cond1)
val data2 = data1.filter(x => type2_dm_dx.contains(x._2._2.last.code))
val data3 = data2.filter(x => type1_dm_med.contains(x._2._1.last.medicine))

hope this helps

beside your question, x._2._1 this looks suspicious and like unreadable code/bad practice. You may read about case classes. — Nabil A.
– Nabil A., Commented Feb 26, 2016 at 8:12
You'll need to provide some more context for more meaningful help. All I can glean is that it looks like whatever x._2._1 is an empty List and as a result calling .last on it throws an error. — badcook
– badcook, Commented Feb 26, 2016 at 8:13

Ben James · Accepted Answer · 2016-02-26 08:28:15Z

2

Calling last on an empty List will cause this exception to be thrown.

You could write your filter predicate to safely handle an empty list, by using lastOption, for example:

x => x._2._1.lastOption.exists(y => type1_dm_med.contains(y.medicine))

answered Feb 26, 2016 at 8:28

Ben James

126k27 gold badges196 silver badges155 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

novicecoder123 Over a year ago

added more context - see the problem definition again - your suggestions ensures there is no error anymore but the filter is not happening correctly

novicecoder123 Over a year ago

you rock - I was making a coding error - in fact your suggestion works - thanks.

Community · Accepted Answer · 2020-06-20 09:12:55Z

1

Count, counts how many items are in a collection, for that equals returns true. You may want to use size or to pass a item to count.

EDIT

In addition to your editing:
Bens solution is beautiful, but you could write

x => x._2._1.nonEmpty && type1_dm_med.contains(x._2._1.last.medicine)

as well. It may makes your code more readable, depends on the context.
You should consider to use something else then tupels as well. A class or case class would make your code more readable then x._2._1.
If you want to use a tupel, case class is often the better solution. You can think about a case class like a named tupel. It fits the same needs, but is more readable.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Feb 26, 2016 at 8:09

Nabil A.

3,40020 silver badges30 bronze badges

Collectives™ on Stack Overflow

scala - getting java.util.NoSuchElementException

2 Answers 2

2 Comments

EDIT

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

EDIT

Comments

Your Answer

Sign up or log in

Post as a guest

Related