Hi, we have a huge amount of duplicate files in a folder and I would like some pointers on to writing a bash script to create a list of the duplicate files. I've seen examples that check for the md5 sum of files... but I dont need that, the file name is enough. Can someone please help me?

Umm, it should not be possible to have two files with the same name in the same folder.

LOL! True Mr. Telengard. I meant in subfolders. So I have the directory /storage which holds about 10 subfolders which hold around 3 more subfolders each with around 300 + files in. Messy, I know. So the duplicates are between subfolders.

Okay, this has had only very minimal testing so use at your own discretion. As always, it is your responsibility to evaluate this code's suitability for your purposes.

I bet 3 Internets that someone else will have a much more elegant solution for you by tomorrow.

1 members found this post helpful.

Wow, thanks sir. Your help is appreciated.

Hi Telengard, your script is good, but there's a little problem: all the duplicated filenames are displayed twice (Obviously .. if basename of files are duplicated in the list, they fall twice doing "grep" on the same list).
Starting from your script (thanks! ), I applied some little change in order to display the duplicated couples only once. In my variant, also file size are shown.
Hoping to help somebody else, I paste the code hereafter:

Guys,

Telengard's script is definitely the business, found just what I was looking for, but I am afraid I cannot get gabolander's revision to run.

My Centos 5.8 seems to have an issue with grep in thisline:-

if !(grep -q "^${name}${TAB}" names.out); then

At least I am getting a grep error repeated and since the syntax from the orginal script is unchanged, I surmise that its the line I've pasted.

I'd appreciate some advice on this please.

Thanks

This is very weird really.. I just tested my script on a updated CentOS 5.8

And the line with "if !(grep -q ...)" is a standard way of grepping something into Bash scripts ....

Are you sure to have prefixed "#!/bin/bash" in the first line of the script? (I don't wish it runs with standard sh, that could not have the same sintax at all of Bash extensions ... )

This is the result of running my script on two dup files in two subdirectories:


Cheers,
G.

This is pasted from my script, I do expect you instantly to point at my foolishness!

#!/bin/bash

find -type f > names.lst

> names.out

TAB='echo -ne "\t"'
while read name
do
bn="$( basename "$name" )"
name2="$( grep "$bn" names.lst | grep -v "$name" )"
if [ "$name2" != "" ]
then
if !(grep -q "^${name}${TAB}" names.out); then
size='stat --format=%s "$name"'
size2='stat --format=%s "$name2"'
echo -e "$name${TAB}$size" >> names.out
echo -e "$name2${TAB}$size" >> names.out
fi
fi
done < names.lst

cat names.out
# rm -f names.lst

Right, I've a better result now.
I cut and pasted the original code into windows notepad....got in a state with line breaks and learned DOS2UNIX, but anyway, now got its precsiely into Linux, and it does run.

Sorry to mess you about.

However, while I scan the terminal window I see messages that it was unable to stat such and such a file and no such folder exists and the file name is too long, but maybe that's correct if there is no duplicate?

So, if there are no duplicates and I've a series of sub folder with thousands of files, it would be even better if the lines where there are no duplicates were not written to the output file.

the Duplicates do however end up with their stats on following lines.

Thanks

All,

Just to round this off, I failed to get this to work on my set up so looked around some more and in the end http://www.perlmonks.org/?node_id=855401 provided me with exactly what I needed.

Had to edit the 1st post following the posters, but here is what worked for me:-

#!/usr/bin/perl
use strict;
use warnings;

use File::Compare;
use File::Find;

#If you want to set a base_directory, you can do so here.

my $base_directory;

print "What directory? ";
my $directory = <>;
chomp $directory;

my %files;
sub files_wanted {
my $raw_file = $File::Find::name;
if ( -f ) {
my ($volume,$directories,$file) = File::Spec->splitpath($raw_file);
#update from a prior suggestion.
my $file_size = -s $raw_file;
push @{$files{"$file ($file_size bytes)"}}, $raw_file;
}
}
#If you set a base directory above, you will need to change

find(\&files_wanted,$directory);


open (MYFILE, '>>dupes.txt');



#This section searches the hash for any file with 2 or more files which share the same filename.ext and size.
#After that, it compares all of the files with those attributes to determine if they share the same contents.
#It will print the list of files with the same filename and size and will tell you which ones share the same
#contents.

for my $file (sort keys %files) {
if (@{$files{$file}} > 1) {
my $amount = @{$files{$file}};
print (MYFILE) "$file\t\t$amount\n";
for my $location1 (@{$files{$file}}) {
print (MYFILE) "\t$location1\n";
for my $location2 (@{$files{$file}}) {
unless ($location1 eq $location2) {
if (compare($location1,$location2) == 0) {
print (MYFILE) "\t\tExact copy: $location2\n";
}
}
}
}
print (MYFILE) "\n";
}
}
close (MYFILE);