Pyspark/Python : Converting csv file which has Multiline rows file to single line row file

Did you have something like this in mind?

import re

items  = re.findall("[^ ,\n]+", """id1,id2,id3,id4,id5,id6,id7
1,2,3,4,5,6,7

1,2,3,4

,5,6,

7

1,2

3,4

,5,6,


7""")

rows = [items[i:i+7] for i in range(0,len(items),7)]
pd.DataFrame(rows[1:], columns=rows[0])

Output:

  id1 id2 id3 id4 id5 id6 id7
0   1   2   3   4   5   6   7
1   1   2   3   4   5   6   7
2   1   2   3   4   5   6   7

Since it has been requested here is a no loop version of the 2nd part:

rows = np.array(items).reshape(len(items)//7,7)
pd.DataFrame(rows[1:], columns=rows[0])

I have tested if it actually saves time by using jupter's %%timeit: it turns out:

• the regular expression part takes 6.66 µs ± 43.8 ns,
• the old loop part of then turning it into a dataframe takes 759 µs ± 2.81 µs
• and the new numpy version of the same takes 149 µs ± 4.82 µs