I created a txt file using two requests, one LDAP and one SQL. Results of the two requests are stored in the same txt file.
The txt file looks like this :
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
Because a user can be in the two databases, I need to delete duplicate entries, using bash.
How can I do it?
If you don't mind your file ending up sorted, sort it and filter it; either
sort -u file
if your sort supports it, or
sort file | uniq
if not, and you'll get on standard output the sorted list of unique email addresses.
If you want to keep the addresses in the original order, use awk:
awk '!(count[$0]++)' file
sort -u doesn't report the unique line but the first in lines sort the same in current locale.
①@example.com and ②@example.com in en_US.utf8 locale.
LC_ALL=en_US.UTF-8; (echo ①@example.com; echo ②@example.com) | sort | uniq also merges both lines, so only the awk solution is viable in that case.
uniq -i.