Exception in thread "main" java.io.IOException: Incomplete HDFS URI, no host: hdfs Spark RDD

Question

I'm trying to save the content of a text file in hdfs with Spark:

 import org.apache.spark.{SparkContext, SparkConf}

  object FormatTlfHdfs {   def main(args : Array[String]) {
     val conf = new SparkConf().setAppName("Clean data")
                  .setMaster("local").setSparkHome("/usr/lib/spark")

     val sc = new SparkContext(conf)

     var vertices = sc.textFile("hdfs:///user/cloudera/dstlf.txt").flatMap{ 
       line => line.split("\\s+") }.distinct()

I'm getting the error :

Exception in thread "main" java.io.IOException: Incomplete HDFS URI, no host: hdfs:///user/cloudera/metadata-lookup-tlf

Doing hdfs dfs -ls looks that is correct

cloudera@quickstart grafoTelefonos]$ hdfs dfs -ls /user/cloudera 
Found 6 items 
drwx------   - cloudera cloudera          0 2016-02-04 18:37 /user/cloudera/.Trash 
drwxr-xr-x   - cloudera cloudera          0 2016-05-02 13:38 /user/cloudera/.sparkStaging
-rw-r--r--   1 cloudera cloudera       1294 2016-05-02 13:34 /user/cloudera /dstlf.txt

OneCricketeer · Accepted Answer · 2016-05-05 18:01:07Z

3

Error seems obvious...

Incomplete HDFS URI, no host: hdfs:///user/cloudera/metadata-lookup-tlf

You didn't specify a host machine like

hdfs://quickstart:<hdfs_port>/user/cloudera...

You may not need the <hdfs_port> piece, but it doesn't hurt. I think that the correct port is 8020, so then you'd have

hdfs://quickstart:8020/user/cloudera...

answered May 5, 2016 at 18:01

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Exception in thread "main" java.io.IOException: Incomplete HDFS URI, no host: hdfs Spark RDD

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related