why remove header from csv file doesn't work

Question

 object test {

 case class Caserne(x: String, y: String, Name: String, Description: String)


def main(args: Array[String]): Unit = {

val conf = new SparkConf().setAppName("BankDataAnalysis").setMaster("local[1]")
val sc = new SparkContext(conf)
val sqlContext= new SQLContext(sc)

import sqlContext.implicits._

// load caserne data
val caserneTxt = sc.parallelize(
  IOUtils.toString(
    new URL("http://donnees.ville.montreal.qc.ca/dataset/c69e78c6-e454-4bd9-9778-e4b0eaf8105b/resource/f6542ad1-31f5-458e-b33d-1a028fab3e98/download/casernessim.csv"),
    Charset.forName("utf8")).split("\n"))

val header = caserneTxt.first()
val caserne = caserneTxt.map(s => s.split(",")).filter(s => s != header).map(
  s => Caserne(s(0),
               s(1),
               s(2).replaceAll("[^\\d]", "").trim(),
               s(3).replaceAll("""<(?!\/?a(?=>|\s.*>))\/?.*?>""", " ").trim()
               )).toDF()

 caserne.registerTempTable("caserne")
 sqlContext.sql("Select * from caserne").show()

 }
}

enter image description here

I have to remove csv file header. I used filter(s => s != header) but it did'nt work. Thank you for your help

Community · Accepted Answer · 2017-05-23 12:31:57Z

1

Try using :-

val rows = data.filter(s=> header(s,"X") != "X")

reference :- How do I convert csv file to rdd

I found this convenient method

val header = caserneTxt.first()
val no_header = caserneTxt.filter(_(0) != header(0))

edited May 23, 2017 at 12:31

CommunityBot

11 silver badge

answered Feb 24, 2017 at 6:12

Ashish Singh

5333 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Adil Blanco Over a year ago

Thank you for help, i found the easy way from reference :)

Prasad Khode · Accepted Answer · 2017-02-24 05:33:34Z

0

one way would be using the one of the header key and filter that from dataframe something like below

dataFrame.filter(row => row.getAs[String]("description") != "description").show

answered Feb 24, 2017 at 5:33

Prasad Khode

6,77712 gold badges48 silver badges62 bronze badges

Collectives™ on Stack Overflow

why remove header from csv file doesn't work

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related