passing extra variables in Spark UDF JAVA

Question

I have written a spark UDF in JAVA to encrypt particular columns in a dataframe. It is type 1 UDF and only accepts a string that needs to be encrypted or decrypted at a time. I want to pass corresponding password as well. I tried the currying approach but was not able to write the function properly. Can anyone suggest me any solution ?

public class EncryptString implements UDF1<String, String> {


@Override
public String call(String s) throws Exception {
    return Aes256.encrypt(s);  
    //Aes.encrypt needs to have another variable password.
    //So that while calling the UDF we can pass the required password.
}
}

werner · Accepted Answer · 2019-12-30 09:42:05Z

2

You can pass the password - as well as any other parameters - as constructor parameter to the EncryptString class:

public static class EncryptString implements UDF1<String, String> {

    private final String password; 

    public EncryptString(String password) {
        this.password = password;
    }

    public String call(String s) throws Exception { 
        return Aes256.encrypt(s, password);
    }
}

When instantiating the udf, you can pass the actual password:

spark.sqlContext().udf().register("EncryptUdf", new EncryptString("secret"), DataTypes.StringType);
[...]
spark.sql("select EncryptUdf(_c2) from df").show();

answered Dec 30, 2019 at 9:42

werner

15k6 gold badges36 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Fuad Efendi Over a year ago

And what to do if it is dynamic parameter, which is not column? Provided example will work for constant "secret" only. Should I register new function for each imaginable parameter value?

werner Over a year ago

@FuadEfendi at the point in your code when you register the UDF, you should know your password. You don't have to pass in a fixed string into the constructor, you can use a variable. However, if the password is not known at this point in time or if the password is not the same for all rows, then this solution will not work for you. Probably you will need to formulate another question.

Fuad Efendi Over a year ago

thank you for responding; after a lot of digging for Spark SQL UDF, including examples of Scala currying, I started to realize that there is no way to explicitly pass dynamic parameter to UDF except "org.apache.spark.sql.functions.col" (which will be super slow if values of column are dynamic). And, of course, UDF1 implementation may call external service to find unique "secret" for each user, but we cannot call it "passing extra variables in Spark UDF" neither; parameters must be columns (such as functions.col("MySecret"))

Collectives™ on Stack Overflow

passing extra variables in Spark UDF JAVA

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related