Comparing 2 Files for Duplicates

Mr_H · 11-09-2005, 11:11 AM

I've searched the forums for this and I haven't found it, if someone's seen it out there, please point me in the right direction

Now, onto the problem:
I have two files:
List 1 is a list of users who are going to be deleted. (1,600 people)
List 2 is a list of users who have logged on in the past 30 days. (11,000 people)

The files are not necessarily in order (so line by line comparision doesn't work)

I need a way to compare these two files to look for Duplicates and print those out, IE: Is there someone on List 1 who is also on List 2?

I've looked at diff, it just shows the differences (of which there are a lot obviously).
Comm does line by line, cmp ditto.

Any suggestions from folks on what I can do here? It's probably something obvious that's shooting right over my head and I do apologize if it's a stupid question, but it's driving me nuts.

Thanks!
H

Ynot Irucrem · 11-09-2005, 11:31 AM

look into sort. then you can do a line by line comparison.

theYinYeti · 11-09-2005, 11:40 AM

This should do the trick:

Code:

sort list1.txt list2.txt | uniq -d

Yves.

Ynot Irucrem · 11-09-2005, 11:55 AM

isnt sorting redundant there? wouldn't

Code:

cat file1.txt `echo` file2.txt | uniq -d

be faster? yes i know i suggested sorting, but i didn't know about the uniq command until I just looked it up. (damn linux... there's too many useful utilities

Mr_H · 11-09-2005, 12:17 PM

Thanks folks, much appreciated! The sort one seems to have worked (the cat one gave me only four responses and 3 were the same)

Anyhow, helped out a lot

Ynot Irucrem · 11-09-2005, 12:43 PM

hmm.. i didn't read that man page properly. and I was thinking backwards. the way i was thinking, it should have been

Code:

echo -e "`file1.txt`\n`cat file2.txt`" | uniq -d

but i didn't see this part:

Quote:

Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first.