[SOLVED] deleting selected lines

sharky · 09-18-2013, 12:07 PM

looking for a one liner that will delete a line if the first term of the line is the same as the first term in a previous line.

Quote:

example;
original list;
xxx / 123
zzz / 123
zzz / 234
www / 123

processed list;
xxx / 123
zzz / 123
www / 123

Because "zzz" is seen in the second line the next occurrence seen in the third line is extracted. The "/" and numbers following the first term are not relevant.

This looks like a problem made for awk or sed but I'm not sure how to make it work.

I prefer a one liner because this problem is part of a bash script. However, beggars can't be choosers so I'm open to any suggestions.

Thanks in advance.

sharky · 09-18-2013, 12:36 PM

Forgot to mention, it is safe to assume that the original list is sorted alphabetically.

danielbmartin · 09-18-2013, 12:53 PM

Try this ...

Code:

awk '{if (1==++a[$1]) print}' $InFile >$OutFile

Daniel B. Martin

sharky · 09-18-2013, 01:05 PM

Quote:

Originally Posted by danielbmartin

Try this ...

Code:

awk -F/ '{if (1==++a[$1]) print}' $InFile >$OutFile

Daniel B. Martin

Bammo! That worked like a charm.

danielbmartin · 09-18-2013, 01:20 PM

Another solution, possibly faster for large input files ...

Code:

awk '{if (prev!=$1) print; prev=$1}' $InFile >$OutFile

Daniel B. Martin

danielbmartin · 09-18-2013, 01:34 PM

Another solution. I don't like this one because it is unreadable.

Code:

sed -r '$!N; /^(.*\/)(.*)\n\1/!P; D' $InFile >$OutFile

Daniel B. Martin

millgates · 09-22-2013, 05:16 AM

Quote:

Originally Posted by danielbmartin

Another solution, possibly faster for large input files ...

Code:

awk '{if (prev!=$1) print; prev=$1}' $InFile >$OutFile

Daniel B. Martin

Can be made slightly shorter:

Code:

awk 'p!=p=$1' $InFile >$OutFile

danielbmartin · 09-22-2013, 11:02 AM

Quote:

Originally Posted by millgates

Code:

awk 'p!=p=$1' $InFile >$OutFile

Excellent! This wins a prize for being concise.

Daniel B. Martin