how to share the same broadcast-variable-contained filter functions in different pyspark script?

Question

there are some filter functions like this:

def filter1(x):
    if broadcast_variable1.value[x] > 1:
       return False
    return True

def filter2(x):
    if broadcast_variable2.value[x] < 1:
       error_accumulator_variable.add(1)
       return False
    return True

these function are included in my main pyspark script. Now I want to split them into a module file for easier maintenance (I have two pyspark scripts for different use but they have the same filters. ), keep the rest in different files.

how to share these-like filter functions in different pyspark script?

Thanks for your generous help！

Mazz · Accepted Answer · 2017-02-21 04:05:23Z

1

My basic purpose is to make the program more readable. So I find a simple way to achieve this.

def x1(x,broadcast_variable1):
   if broadcast_variable1.value[x] > 1:
   return False

def filter1(x):
   return x1(x,broadcast_variable1)

and this can also be rewriten in the currying way(code below is not in python):

def x1(broadcast_variable)(x) = 
   if broadcast_variable.value[x] > 1:
   return False
def filter1(x) = x1(broadcast_variable1)

now, I can write spark code like this:

RDD.filter(filter1).map(map1)....

answered Feb 21, 2017 at 4:05

Mazz

711 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user6022341 · Accepted Answer · 2016-11-10 06:04:53Z

0

Use it as argument for functions:

def filter1(x, bv):
   ...

def filter2(x, bv):
   ...

broadcast_variable1 = ...

someRDD.filter(lambda x: filter1(x, broadcast_variable1))

answered Nov 10, 2016 at 6:04

community wiki

user6022341

Collectives™ on Stack Overflow

how to share the same broadcast-variable-contained filter functions in different pyspark script?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related