Finding duplicate lines in a CSV file

Question

I'm trying to use Powershell to search a csv file and output a list of duplicate lines in a csv file. I can accomplish this pretty easily in bash with the following:

uniq -d myfile.csv > list.csv

In Powershell I can output a list of unique lines but how do I modify Get-Unique to display only the duplicate lines like I did in bash?

Get-Content c:\file\myfile.csv | Get-Unique | Set-Content c:\file\list1.csv

TessellatingHeckler · Accepted Answer · 2016-06-23 16:29:00Z

2

It's a bit weird to use the unique tool to get the duplicates. How about:

gc .\test.csv | group -NoElement |? Count -gt 1 | select -expand name

This groups the lines by how many there are, identifies the ones with duplicates, and outputs them. e.g. if:

test.csv contains:

a,b,c
d,e,f
a,b,c
z,z,z

gc test.csv | group

Count Name                      Group                                                                    
----- ----                      -----                                                                    
    2 a,b,c                     {a,b,c, a,b,c}                                                           
    1 d,e,f                     {d,e,f}                                                                  
    1 z,z,z                     {z,z,z}                                                                  
    1                           {}

and -NoElement stops it building the group contents, redundant in this case.

answered Jun 23, 2016 at 16:29

TessellatingHeckler

29.3k4 gold badges55 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user6297176 Over a year ago

Thank you! That's exactly what I needed.

Collectives™ on Stack Overflow

Finding duplicate lines in a CSV file

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related