LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 08-15-2009, 02:22 AM   #1
boxb29
LQ Newbie
 
Registered: Aug 2009
Posts: 21

Rep: Reputation: 15
uniq -u : does not seem to remove duplicate lines


I am trying to comb through my local.cf file and remove all the duplicate blacklist_from entries. I ran

uniq -u local.cf output.cf

It did trim about 45 lines out of the file. But, there are still many many duplicate lines. I thought maybe they were different some how, but not visible to the eye...BUT, I ran:

sort output.cf | uniq -dc

this gave me a line count output for all the dups, and there are still many many...as you can see (below).

HELP


root@LINUX03:/home/backups# sort output.cf | uniq -dc
3
11 #
12 blacklist_from 1800FLOWERS@e.1800flowers.com
2 blacklist_from acrane@amgacademy.com
2 blacklist_from alejmagna@hotmail.com
10 blacklist_from alerts@personals.yahoo.com
3 blacklist_from Allen_Brothers@mail.vresp.com
8 blacklist_from Borders@e.borders.com
2 blacklist_from buy.com_offers@enews.buy.com
5 blacklist_from capitalone@email.capitalone.com
2 blacklist_from customerservice@duebrightlive.info
2 blacklist_from customerservice@ehealthinsurance.com
2 blacklist_from customerservice@mymorepayhomeonline.info
2 blacklist_from customerservice@youreraseduelive.info
2 blacklist_from directv@customerinfo.directv.com
3 blacklist_from email@email.creditreport.com
8 blacklist_from email@email.hotels.com
4 blacklist_from etrade@email.etradefinancial.com
30 blacklist_from group-digests@linkedin.com
4 blacklist_from HHonors@h3.hilton.com
4 blacklist_from info@aiueducationonline.com
4 blacklist_from info@birdiebug.com
2 blacklist_from info@promo-em.jetblue.com
2 blacklist_from info@samstailor.com
2 blacklist_from invite@naymz.com
6 blacklist_from iprint@specials.iprint.com
12 blacklist_from JobAlerts@CyberCoders.com
2 blacklist_from lilly@sportsub.com
16 blacklist_from listmaster@thegolfchannel.com
7 blacklist_from mail@netapp.com
2 blacklist_from mail@news.beachcamera.com
2 blacklist_from microsoft@reply.digitalriver.com
2 blacklist_from mike.moreno_at_mbofpleasanton.com@mmserver.com
2 blacklist_from Mimosa_Systems@mail.vresp.com
6 blacklist_from movies@news.fandango.com
2 blacklist_from mwilkinson@serrahs.com
2 blacklist_from nancyp@saintmatthew.org
2 blacklist_from newsletter@reply.ticketmaster.com
2 blacklist_from notifications@email.etradefinancial.com
6 blacklist_from NutriSystem@news.nutrisystem.com
4 blacklist_from paypal@email.paypal.com
2 blacklist_from PGATOUR@pgatouremail.com
2 blacklist_from PGATOUR@weic11.com
4 blacklist_from radioshack@em.radioshack.com
2 blacklist_from Rebecca_Salie@mail.vresp.com
4 blacklist_from replies@oracle-mail.com
10 blacklist_from reply@igmemail.com
2 blacklist_from rexspelling@resumespider.com
28 blacklist_from rushinahurry@rushlimbaugh.com
2 blacklist_from sanjoseexecutives@gmail.com
2 blacklist_from store-news@amazon.com
4 blacklist_from Store-News@ShopAETV.p0.com
2 blacklist_from support@myremoveliability.info
2 blacklist_from TheHartford@weic11.com
4 blacklist_from updates@linkedin.com
4 blacklist_from update@stubhub-mail.com
2 blacklist_from ups@upsemail.com
2 blacklist_from vmwareteam@connect.vmware.com
2 blacklist_from voyages@viator.messages1.com
2 blacklist_from WebEx@weic11.com
 
Old 08-15-2009, 02:38 AM   #2
JulianTosh
Member
 
Registered: Sep 2007
Location: Las Vegas, NV
Distribution: Fedora / CentOS
Posts: 674
Blog Entries: 3

Rep: Reputation: 90
you must supply uniq with sorted data. try 'sort local.cf | uniq -c'
 
Old 08-15-2009, 02:42 AM   #3
jschiwal
LQ Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682
The same goes for the "comm" command.
comm -3 <(sort list1) <(sort list2)
 
Old 08-15-2009, 03:47 AM   #4
Nevahre
LQ Newbie
 
Registered: Aug 2009
Posts: 21

Rep: Reputation: 16
why use 2 progrs: sort -u local.cf
sort has a unique (-u/--unique) option.....
 
Old 08-15-2009, 02:05 PM   #5
w1k0
Senior Member
 
Registered: May 2008
Location: Poland
Distribution: Slackware (personalized Window Maker), Mint (customized MATE)
Posts: 1,309

Rep: Reputation: 234Reputation: 234Reputation: 234
Quote:
Originally Posted by Nevahre View Post
why use 2 progrs: sort -u local.cf
sort has a unique (-u/--unique) option.....
What command to use depends on what result you expect.

Look at that file:

$ cat file
Code:
one
two
three
two
three
three
four
four
four
four
That command removes consecutive duplicated lines:

$ uniq -u file
Code:
one
two
three
two
That command counts consecutive duplicated lines:

$ uniq -dc file
Code:
      2 three
      4 four
That commands counts occurrences of all lines:

$ sort file | uniq -c
Code:
      4 four
      1 one
      3 three
      2 two
That command does the same but sorts result taking into considerance the numbers of occurrences:

$ sort file | uniq -c | sort -nr
Code:
      4 four
      3 three
      2 two
      1 one
That command displays each unique line only once:

$ sort -u file
Code:
four
one
three
two

Last edited by w1k0; 08-15-2009 at 02:07 PM.
 
Old 08-15-2009, 02:56 PM   #6
Nevahre
LQ Newbie
 
Registered: Aug 2009
Posts: 21

Rep: Reputation: 16
I know that.

I just try to say that: sort file | uniq is too long. use sort -u file
 
Old 08-15-2009, 03:15 PM   #7
w1k0
Senior Member
 
Registered: May 2008
Location: Poland
Distribution: Slackware (personalized Window Maker), Mint (customized MATE)
Posts: 1,309

Rep: Reputation: 234Reputation: 234Reputation: 234
I quoted in my post your comment but the entire post was directed rather to boxb29 than to you. I'm not sure what he'd like to achieve but I suppose he'd like to receive ``clean'' file including each unique line only once. If so your advice is the best solution of that problem.
 
Old 08-15-2009, 06:34 PM   #8
boxb29
LQ Newbie
 
Registered: Aug 2009
Posts: 21

Original Poster
Rep: Reputation: 15
perfect...

Code:
sort -u local.cf > new.file
That did the trick....thanks all !
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to compare two lines and delete the duplicate line from a file? Shobhna Linux - Newbie 10 12-05-2008 01:08 PM
Finding duplicate lines in a file MikeyCarter Linux - Software 3 10-05-2008 05:28 PM
how do u delete duplicate lines bharatbsharma Programming 4 10-29-2007 06:04 PM
checking for duplicate lines in text files (vb.net) mrobertson Programming 11 08-01-2005 12:40 PM
Removing duplicate lines with sed tireseas Programming 10 01-12-2005 03:27 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 08:13 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration