Reduce and Lambda on pyspark dataframe

Question

Below is an example from https://graphframes.github.io/graphframes/docs/_site/user-guide.html

the only thing I confused is the purpose of "lit(0)" from function of condition if this "lit(0)" mean to feed into "cnt"? if yes why is it after ["ab","bc","cd"]?

from pyspark.sql.functions import col, lit, when
from pyspark.sql.types import IntegerType
from graphframes.examples import Graphs
from functools import reduce

chain4 = g.find("(a)-[ab]->(b); (b)-[bc]->(c); (c)-[cd]->(d)")

chain4.show()

sumFriends = lambda cnt,relationship: when(relationship == "friend", cnt+1).otherwise(cnt)

condition = reduce(lambda cnt,e: sumFriends(cnt, col(e).relationship), ["ab", "bc", "cd"], lit(0))

chainWith2Friends2 = chain4.where(condition >= 2)
chainWith2Friends2.show()

mck · Accepted Answer · 2021-03-29 13:09:33Z

1

lit(0) is the initializer of the reduce statement. You need to initialize the sumFriends counter with cnt = 0 to start counting.

condition = reduce(lambda cnt,e: sumFriends(cnt, col(e).relationship), ["ab", "bc", "cd"], lit(0))

# should be equivalent to

condition = sumFriends(lit(0), col("ab").relationship)
condition = sumFriends(condition, col("bc").relationship)
condition = sumFriends(condition, col("cd").relationship)

answered Mar 29, 2021 at 13:09

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

gllow Over a year ago

Thanks for answering, one more question will be how is the function recognise cnt should be assigned by the initialiser ?

mck Over a year ago

@gllow that's how the reduce function was defined in Python. You can have a look at the code example in the linked docs, especially the lines value = initializer and then value = function(value, element).

mck Over a year ago

The initializer is used as the first argument of the provided lambda function.

Collectives™ on Stack Overflow

Reduce and Lambda on pyspark dataframe

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related