How can I select a non-sequential subset elements from an array using Scala and Spark?

Question

In Python, this is how I would do it.

>>> x
array([10,  9,  8,  7,  6,  5,  4,  3,  2])
>>> x[np.array([3, 3, 1, 8])]
array([7, 7, 9, 2])

This doesn't work in the Scala Spark shell:

scala> val indices = Array(3,2,0)
indices: Array[Int] = Array(3, 2, 0)

scala> val A = Array(10,11,12,13,14,15)
A: Array[Int] = Array(10, 11, 12, 13, 14, 15)

scala> A(indices)
<console>:28: error: type mismatch;
 found   : Array[Int]
 required: Int
              A(indices)

The foreach method doesn't work either:

scala> indices.foreach(println(_))
3
2
0

scala> indices.foreach(A(_))
<no output>

What I want is the result of B:

scala> val B = Array(A(3),A(2),A(0))
B: Array[Int] = Array(13, 12, 10)

However, I don't want to hard code it like that because I don't know how long indices is or what would be in it.

like this? val x = Array(10, 9, 8, 7, 6, 5, 4, 3, 2); Array(3, 3, 1, 8).map(x) — Tesseract
– Tesseract, Commented Apr 2, 2015 at 17:35

Justin Pihony · Accepted Answer · 2015-04-02 17:39:51Z

7

The most concise way I can think of is to flip your mental model and put indices first:

indices map A

And, I would potentially suggest using lift to return an Option

indices map A.lift

edited Apr 2, 2015 at 17:39

answered Apr 2, 2015 at 17:34

Justin Pihony

67.2k20 gold badges154 silver badges185 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ben Reich · Accepted Answer · 2015-04-02 18:24:34Z

7

You can use map on indices, which maps each element to a new element based on a mapping lambda. Note that on Array, you get an element at an index with the apply method:

indices.map(index => A.apply(index))

You can leave off apply:

indices.map(index => A(index))

You can also use the underscore syntax:

indices.map(A(_))

When you're in a situation like this, you can even leave off the underscore:

indices.map(A)

And you can use the alternate space syntax:

indices map A

You were trying to use foreach, which returns Unit, and is only used for side effects. For example:

indices.foreach(index => println(A(index)))
indices.map(A).foreach(println)
indices map A foreach println

answered Apr 2, 2015 at 18:24

Ben Reich

16.3k2 gold badges41 silver badges61 bronze badges

Collectives™ on Stack Overflow

How can I select a non-sequential subset elements from an array using Scala and Spark?

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related