Rowwise calculation using SparkR

Question

Here is my toy dataframe.

library(tibble); library(SparkR)

df <- tibble::tribble(
  ~var1, ~var2, ~maxofvar1var2,
  1L,    1L,    1L,
  2L,    1L,    2L,
  2L,    3L,    3L,
  NA,    2L,    2L,
  1L,    4L,    4L,
  8L,    5L,    8L)

df <- df %>% as.DataFrame()

How can I calculate the rowwise calculation using SparkR to get the max value of var1 and var2 as shown in the 3rd variable in the df above? If there is no rowwise function in SparkR, how can I get the desired output?

10465355 · Accepted Answer · 2018-11-06 11:43:53Z

1

To get a maximum value from a set of columns use SparkR::greatest:

df %>% withColumn("maxOfVars", greatest(df$var1, df$var2))

and in general case higher order functions, like aggregate (Spark 2.4 or later), on assembled data .

df %>% withColumn("theLastVar", expr("aggregate(array(var1, var2), (x, y) -> y)"))

or (version independent) composition of expressions:

scols <- c("var1", "var2") %>% purrr::map(column)

sumOfVars <- scols %>%
  purrr::map(function(x) coalesce(x, lit(0)))  %>%
  purrr::reduce(function(x, y) x + y, .init=lit(0))

countOfVars <- scols %>% 
  purrr::map(function(x) ifelse(isNotNull(x), lit(1), lit(0))) %>%
  purrr::reduce(
    function(x, y) x + y, .init=lit(0))

df %>% withColumn("meanOfVars", sumOfVars / countOfVars)

edited Nov 6, 2018 at 11:43

answered Nov 6, 2018 at 11:08

10465355

4,6412 gold badges24 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Geet Over a year ago

Greatest is an interesting name for a function! Thanks!

Collectives™ on Stack Overflow

Rowwise calculation using SparkR

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related