1

My use case is as follows. I need to be able to call java methods from within python code

from py spark this seems to be very easy

I start the py spark like this ./pyspark --driver-class-path /path/to/app.jar

and from pyspark shell do this

        x=sc._jvm.com.abc.def.App
        x.getMessage()
        u'Hello'
        x.getMessage()
        u'Hello'

This works fine.

When working with spark job server though:

I use the WordCountSparkJob.py example shipped

from sparkjobserver.api import SparkJob, build_problems
from py4j.java_gateway import JavaGateway, java_import

class WordCountSparkJob(SparkJob):

def validate(self, context, runtime, config):
    if config.get('input.strings', None):
        return config.get('input.strings')
    else:
        return build_problems(['config input.strings not found'])

def run_job(self, context, runtime, data):
    x = context._jvm.com.abc.def.App        
    return x.getMessage() 

My python.conf looks like this

spark {
jobserver {
jobdao = spark.jobserver.io.JobSqlDAO
}

context-settings {
python {
paths = [
"/home/xxx/SPARK/spark-1.6.0-bin-hadoop2.6/python",
"/home/xxx/.local/lib/python2.7/site-packages/pyhocon",
"/home/xxx/SPARK/spark-1.6.0-bin-hadoop2.6/python/lib/pyspark.zip",
"/home/xxx/SPARK/spark-1.6.0-bin-hadoop2.6/python/lib/py4j-0.9-src.zip",
"/home/xxx/gitrepos/spark-jobserver/job-server-python/src/python    /dist/spark_jobserver_python-NO_ENV-py2.7.egg"
]
}
dependent-jar-uris = ["file:///path/to/app.jar"]
}
home = /home/path/to/spark
}

I get the following error

    [2016-10-08 23:03:46,214] ERROR jobserver.python.PythonJob []         [akka://JobServer/user/context-supervisor/py-context] - From Python: Error while calling 'run_job'TypeError("'JavaPackage' object is not callable",)
[2016-10-08 23:03:46,226] ERROR jobserver.python.PythonJob [] [akka://JobServer/user/context-supervisor/py-context] - Python job failed with error code 4
[2016-10-08 23:03:46,228] ERROR .jobserver.JobManagerActor []     [akka://JobServer/user/context-supervisor/py-context] - Got Throwable
    java.lang.Exception: Python job failed with error code 4
    at spark.jobserver.python.PythonJob$$anonfun$1.apply(PythonJob.scala:85)
    at scala.util.Try$.apply(Try.scala:161)
    at spark.jobserver.python.PythonJob.runJob(PythonJob.scala:62)
    at spark.jobserver.python.PythonJob.runJob(PythonJob.scala:13)
    at     spark.jobserver.JobManagerActor$$anonfun$getJobFuture$4.apply(JobManagerActor.scala:288)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-10-08 23:03:46,232] ERROR .jobserver.JobManagerActor [] [akka://JobServer/user/context-supervisor/py-context] - Exception from job 942727f0-dd81-445d-bc64-bd18880eb4bc:
java.lang.Exception: Python job failed with error code 4
at spark.jobserver.python.PythonJob$$anonfun$1.apply(PythonJob.scala:85)
at scala.util.Try$.apply(Try.scala:161)
at spark.jobserver.python.PythonJob.runJob(PythonJob.scala:62)
at spark.jobserver.python.PythonJob.runJob(PythonJob.scala:13)
at spark.jobserver.JobManagerActor$$anonfun$getJobFuture$4.apply(JobManagerActor.scala:288)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-10-08 23:03:46,232] INFO k.jobserver.JobStatusActor [] [akka://JobServer/user/context-supervisor/py-context/$a] - Job 942727f0-dd81-445d-bc64-bd18880eb4bc finished with an error
[2016-10-08 23:03:46,233] INFO r$RemoteDeadLetterActorRef [] [akka://JobServer/deadLetters] - Message [spark.jobserver.CommonMessages$JobErroredOut] from Actor[akka://JobServer/user/context-supervisor/py-context/$a#1919442151] to Actor[akka://JobServer/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

In the python.conf file I have the app.jar as an entry in dependent-jar-uris. Am I missing something here

2
  • - From Python: Error while calling 'run_job'TypeError("'JavaPackage' object is not callable",) Commented Oct 9, 2016 at 21:21
  • Have the same error. Appreciate for any hints. Commented Oct 26, 2016 at 11:05

1 Answer 1

2

Error "'JavaPackage' object is not callable" probably means that PySpark cannot see your jar or your class in it.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.