i am new and i need help with this issue.
I have a csv file like this:
ANI,2974483123 29744423747 293744450542,Twitter,@ani
I need split the second column "2974483123 29744423747 293744450542" and create 3 rows like this:
ANI,2974483123,Twitter,@ani
ANI,29744423747,Twitter,@ani
ANI,293744450542,Twitter,@ani
Can someone help me? please!
flatMap is what you're looking for:
val input: RDD[String] = sc.parallelize(Seq("ANI,2974483123 29744423747 293744450542,Twitter,@ani"))
val csv: RDD[Array[String]] = input.map(_.split(','))
val result = csv.flatMap { case Array(s1, s2, s3, s4) => s2.split(" ").map(part => (s1, part, s3, s4)) }
Here is a slightly different solution that takes advantage of the built in SQL UDFs available to Spark. Ideally these should be used instead of custom functions to take advantage of performance improvements provided by the query optimizer (https://blog.cloudera.com/blog/2017/02/working-with-udfs-in-apache-spark/).
import org.apache.spark.sql.functions.{split, explode}
val filename = "/path/to/file.csv"
val columns = Seq("col1","col2","col3","col4")
val df = spark.read.csv(filename).toDF(columns: _*)
// import "split" instead of writing your own split UDF
df.withColumn("col2", split($"col2", " ")).
// import "explode instead of map then flatMap
select($"col1", explode($"col2"), $"col3", $"col4").take(10)
Pretty similar to Tzach's answer, but in python2 and being careful about multi-space separators.
import re
rdd = sc.textFile("datasets/test.csv").map(lambda x: x.split(","))
print(rdd.take(1))
print(rdd.map(lambda (a, b, c, d): [(a, number, c, d) for number in re.split(" +", b)])
.flatMap(lambda x: x)
.take(10))
#[[u'ANI', u'2974481249 2974444747 2974440542', u'Twitter', u'maximotussie']]
#[(u'ANI', u'2974481249', u'Twitter', u'maximotussie'),
# (u'ANI', u'2974444747', u'Twitter', u'maximotussie'),
# (u'ANI', u'2974440542', u'Twitter', u'maximotussie')]
