Delete empty lines in csv file

Question

I have a file with 4 million of lines, every line ends with the char $, but I mistakenly add a new line after the the line delimiter while scraping a website, so right now it is looking like this:

fist name, last name, phone, address, postal code, city, region,$

$

fist name, last name, phone, address, postal code, city, region,$

$

the new line '$' only shows up of course if I use :set list, but I'm trying to use this file for a bulk insert in mysql and I'm having problems with it now.

I would like to change the file to:

fist name, last name, phone, address, postal code, city, region,$

fist name, last name, phone, address, postal code, city, region,$

How can I do this? with sed or awk or even vi ? looked up around and what I found is not really applying to this case.

please don't take in consideration the extra empty line shown above.

Thanks in advance

Joseph Quinsey · Accepted Answer · 2014-03-13 02:20:37Z

12

To remove blank lines with sed:

sed -i '/^$/d' yourfile.csv

To remove lines consisting of a single $:

sed -i '/^$$/d' yourfile.csv

Most versions of sed support the -i switch; if yours does not you will need e.g. sed '/^$$/d' yourfile.csv > newfile.csv.

Removing blank lines with white space is more complicated. This usually works:

sed '/^ *$/d' yourfile.csv

If this is not sufficient, try checking also for tabs. For older sed's, this will work:

sed '/^[ X]*$/d' yourfile.csv

where X here a tab, entered via Control-V Tab.

Newer sed's will take a [ \t\r]* or \s* or [[:space:]]*, sometimes requiring a -E switch.

answered Mar 13, 2014 at 2:20

Joseph Quinsey

9,99210 gold badges58 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

that other guy · Accepted Answer · 2014-03-13 00:20:07Z

2

grep can filter lines by match (or negative match) against a regex. To exclude empty lines:

grep -v '^$' yourfile.csv > yourfile_fixed.csv

answered Mar 13, 2014 at 0:20

that other guy

125k12 gold badges187 silver badges214 bronze badges

1 Comment

user3344414 Over a year ago

Thanks for the reply, I believe this should work! But I actually just found that only the first 100 lines are like this because I had to do some copy and paste with these first 100 entries.

jaypal singh · Accepted Answer · 2014-03-13 02:06:21Z

1

Here are your options:

With `awk`:

awk 'NF' file > tmp && mv tmp file

With `sed` (in-place changes so make sure to backup your file using `-i.bak`):

sed -i '/^$/d' file

With `vi`:

:g/^$/d

answered Mar 13, 2014 at 2:06

jaypal singh

77.6k24 gold badges108 silver badges147 bronze badges

2 Comments

anishsane Over a year ago

Just loved awk 'NF' approach!

Raven Over a year ago

How does the NF number-of-fields work in your example?

Fedor · Accepted Answer · 2024-08-31 11:23:07Z

0

A blank line of a CSV file format is a comma delimiter in Linux; therefore you can delete it with the below command

sed '/^,/d' file

edited Aug 31, 2024 at 11:23

Fedor

24.8k45 gold badges59 silver badges189 bronze badges

answered Aug 29, 2024 at 21:00

Vinh Pho

1

Collectives™ on Stack Overflow

Delete empty lines in csv file

4 Answers 4

Comments

1 Comment

With `awk`:

With `sed` (in-place changes so make sure to backup your file using `-i.bak`):

With `vi`:

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

1 Comment

With awk:

With sed (in-place changes so make sure to backup your file using -i.bak):

With vi:

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related

With `awk`:

With `sed` (in-place changes so make sure to backup your file using `-i.bak`):

With `vi`: