Spark-JobServer Java Client - ClassNotFoundException

Question

I am using the Spark JobServer Java Client from this GitHub project:

https://github.com/bluebreezecf/SparkJobServerClient

I am able to upload a Jar containing the Job I want to execute to Spark JobServer. The logs indicate it is stored in /tmp/spark-jobserver directory structure. However, when accessing it from the context I create, the Job class if not found, so the Jar is not being loaded for my Job request.

EDIT: I later discovered that the jar that is uploaded by the Java Client is corrupt. That is why Spark JobServer cannot use it. When I manually replaced it with a good Jar the JobServer ran it fine. Now, the real problem is related to the client's uploadSparkJobJar() API.

org.khaleesi.carfield.tools.sparkjobserver.api.SparkJobServerClientException: Spark Job Server http://sparkjobserverhost:8090/ response 404 { "status": "ERROR", "result": "classPath org.kritek.scalability.jobs.Olap1 not found" }

Here is my code:

        //POST /contexts/<name>--Create context with parameters
        Map<String, String> params = new HashMap<String, String>();
        params.put(ISparkJobServerClientConstants.PARAM_MEM_PER_NODE, "512m");
        params.put(ISparkJobServerClientConstants.PARAM_NUM_CPU_CORES, "10");
        params.put("dependent-jar-uris", "file:///tmp/spark-jobserver/filedao/data/olap1_job-2016-08-11T04_47_07.802Z.jar");
        boolean success = client.createContext(contextName, params);

        assertTrue(success);

        //dependent-jar-uris=file:///some/path/of/my-foo-lib.jar

        //Post /jobs---Create a new job 
        params.put(ISparkJobServerClientConstants.PARAM_APP_NAME, appName);
        params.put(ISparkJobServerClientConstants.PARAM_CLASS_PATH, "org.kritek.scalability.jobs.Olap1");

        SparkJobResult result = null;
        String jobId = null;

        params.put(ISparkJobServerClientConstants.PARAM_CONTEXT, contextName);
        params.put(ISparkJobServerClientConstants.PARAM_SYNC, "true");
        result = client.startJob("conf-1=1", params);

Can you try using curl command so that we can isolate the issue? — noorul
– noorul, Commented Aug 11, 2016 at 7:55
So the problem is with the client. Can you file an issue on GitHub project SparkJobServerClient ? — noorul
– noorul, Commented Aug 11, 2016 at 19:53

SharpLu · Accepted Answer · 2017-08-04 13:04:42Z

0

Check the answer from here spark submit java.lang.ClassNotFoundException

Note: if you are used maven project you could use mvn package assembly:single to include your dependecies.

spark-submit --class Test --master yarn --deploy-mode cluster --supervise --verbose jarName.jar hdfs:///somePath/Test.txt hdfs:///somePath/Test.out

Try to use, also you could check the absolute path in your project

--class com.myclass.Test

answered Aug 4, 2017 at 13:04

SharpLu

1,2242 gold badges14 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark-JobServer Java Client - ClassNotFoundException

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related