PySpark - using functions with variables instead of data frames as parameters raising exception

Question

I am new to Python, so it might be something obvious. I am trying to do some transformation on a string variables (or even literals), but any function I use fails with:

com.incorta.api.exceptions.DataException: INC_03070101: Transformation error [Error An error occurred while calling o74.getnewargs. Trace: py4j.Py4JException: Method getnewargs([]) does not exist ('Py4JError', ':', Py4JError(u'An error occurred while calling o74.getnewargs. Trace:\npy4j.Py4JException: Method getnewargs([]) does not exist\n\tat py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)\n\tat py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)\n\tat py4j.Gateway.invoke(Gateway.java:274)\n\tat py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n\tat py4j.commands.CallCommand.execute(CallCommand.java:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:238)\n\tat java.lang.Thread.run(Thread.java:750)\n\n',))

Spark version 2.4.7

Sample code:

from pyspark.sql.functions import substring
objectName = substring("fullFieldString", 1, 4)

Ekrem Üçüncü · Accepted Answer · 2023-10-20 20:19:50Z

0

I don’t know what you try to achieve but the usage of substring in pyspark is as following.

https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.functions.substring.html

answered Oct 20, 2023 at 20:19

Ekrem Üçüncü

2672 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

misiek8v Over a year ago

I am parsing XML (which is stored in a table). I pull one attribute into a string variable. Then I am trying to split that string into 3. How to achieve it? Sample I gave was simplified without any variable, but just string constant. Does it mean all these functions will only work on data frames? BTW - is my usage not following that documentation? I am providing string, start position and length.

misiek8v · Accepted Answer · 2023-10-23 07:43:03Z

0

I should probably do some Python basics course first :) I found my answer here:

Strings slicing in Python

schemaName = fullFieldString[:fullFieldString.find('.')]

It seems that variables have completely separate functions set and majority of references are describing data frames functions

edited Oct 23, 2023 at 7:43

answered Oct 23, 2023 at 7:42

misiek8v

294 bronze badges

Collectives™ on Stack Overflow

PySpark - using functions with variables instead of data frames as parameters raising exception

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related