Scala: Cumulative String Tokenisation

Question

I'm trying to split an incoming stream of strings into cumulative tokens per line item using a function below,

def cumulativeTokenise(string: String): Array[String] = {
  val array = string.split(" +")
  var result: Array[String] = Array()
  array.map { i => (
                    result = result :+ (
                                       if (result.lastOption == None) i 
                                       else result.lastOption.getOrElse("")+ " " + i
                                       ) 
                   )
             }
  result
}

Ex: output of cumulativeTokenise("TEST VALUE DESCRIPTION . AS") would be => Array(TEST, TEST VALUE, TEST VALUE DESCRIPTION, TEST VALUE DESCRIPTION ., TEST VALUE DESCRIPTION . AS)

Trying to figure out if there's another efficient in-built method in Scala or better ways of doing it with FP, without any mutable array. Any help is much appreciated.

Have you tried scanLeft where the initial parameter is an empty list? — shanif
– shanif, Commented Apr 22, 2020 at 12:53

jwvh · Accepted Answer · 2020-04-23 13:06:32Z

You can get the same results a little more directly.

def cumulativeTokenise(string: String): Array[String] =
  string.split("\\s+")
        .inits
        .map(_.mkString(" "))
        .toArray
        .reverse
        .tail

Or a, perhaps simpler, two step procedure.

def cumulativeTokenise(string: String): Array[String] = {
  val tokens = string.split("\\s+")
  Array.tabulate(tokens.length)(n => tokens.take(n+1).mkString(" "))
}

One problem I see here is that you rely on whitespace to separate tokens. That might not always be the case.

def cumulativeTokenise(string: String): Array[String] =
  string.split("((?=\\W)|(?<=\\W))")
        .filter(_.trim.nonEmpty)
        .inits
        .map(_.mkString(" "))
        .toArray
        .reverse
        .tail

cumulativeTokenise("here@there")
//res0: Array[String] = Array(here, here @, here @ there)

Probably not the best solution to the problem, but it's something to think about.

I like the Array tabulate approach, I'm using whitespace because that was the requirement given to me. Thanks jwvh — Wiki_91
– Wiki_91, Commented Apr 23, 2020 at 13:51

Stack Exchange Network

Scala: Cumulative String Tokenisation

1 Answer 1

You must log in to answer this question.

Hot Network Questions

Scala: Cumulative String Tokenisation

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions