create PySpark Dataframe column based on class method

Question

I have a python class and it has functions like below:

class Features():
    def __init__(self, json):
        self.json = json

    def get_email(self):
        email = self.json.get('fields', {}).get('email', None)
        return email

And I am trying to use the get_email function in a pyspark dataframe to create a new column based on another column, "raw_json",which consists of json value:

df = data.withColumn('email', (F.udf(lambda j: Features.get_email(json.loads(j)), t.StringType()))('raw_json'))

So the ideal pyspark dataframe looks like below:

 +---------------+-----------
 |raw_json         |email
 +----------------+----------
 |                 |  
 +----------------+--------
 |                 |  
 +----------------+-------

But I am getting an error saying:

TypeError: unbound method get_email() must be called with Features instance as first argument (got dict instance instead)

How should I do to achieve this?

I have seen a similar question asked before but it was not resolved.

mck · Accepted Answer · 2020-11-11 19:47:15Z

1

I guess you have misunderstood how classes are used in Python. You're probably looking for this instead:

udf = F.udf(lambda j: Features(json.loads(j)).get_email())
df = data.withColumn('email', udf('raw_json'))

where you instantiate a Features object and call the get_email method of the object.

edited Nov 11, 2020 at 19:47

answered Nov 11, 2020 at 18:38

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

kihhfeue Over a year ago

How can I apply the function on the "raw_json" column though..?

XXavier Over a year ago

Have you tried Features = Features() first, then use data.withColumn('email', Features.get_email(col("raw_json"))?

kihhfeue Over a year ago

@XXavier i got this error when doing Features = Features(): TypeError: __init__() takes exactly 2 arguments (1 given)

mck Over a year ago

@kihhfeue I edited my answer to let you apply on the raw_json column. Your error of not defined parse_date is irrelevant to my suggestion - it's something else that is wrong in your code.

Collectives™ on Stack Overflow

create PySpark Dataframe column based on class method

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related