how do u delete duplicate lines

bharatbsharma · 10-26-2007, 01:52 AM

I have i/p file

AAAA
AAAA
BBBB
BBBB
CCC
CCC

i want o/p file like this

AAAA
BBBB
CCC

jschiwal · 10-26-2007, 01:59 AM

Use the uniq command.

ghostdog74 · 10-26-2007, 02:22 AM

and also sort, if duplicates are all over the place

jschiwal · 10-26-2007, 03:57 AM

Quote:

Originally Posted by ghostdog74

and also sort, if duplicates are all over the place

Yes, if the sorted result is what you want. Otherwise you may need read each line, and check subsequent lines for duplicates. If you use sort, use the "-u" option to eliminate duplicates.

sundialsvcs · 10-29-2007, 06:04 PM

Sorting, in a situation like this, is crucial.

When you sort an arbitrarily-large file by any key, all of the occurrences of any key-value will become adjacent, and if there are any gaps in the key-sequence you automatically know that none of the keys in that gap exist anywhere.