Hi all

in the past I copied files and folders to a backup-folder and added the date of the backup manually to that folder.

Now/today it's time for deduplication


The simplest way I could imagine is a script that would add date and time to a file, cor the case there are serveral version available.

That script should check/look in subfolders as well (because there are many backup-filders).

If there are identical files, only one version should be kept.

Questions:
1.
Is this a good idea?

2.
How would such s script look like?



Thank's a lot in advance for every feedback!

John


PS
I am working with Debian/Ubuntu/Mint

rsync can do that alone, do not need to write any script, just use incremental backups.

1 members found this post helpful.

In the case of nested directories that may contain duplicates you would need to start at the top level and for each file check if there is a matching file in the next level down.
If there then verify if they match or not -- file size and checksums are both good attributes to verify match.
If they don't match then rename the one in the level below, if they match then delete the one below.
Repeat for each file and each subdirectory.

As noted, rsync can be used similarly with the delete option, but may not be precise for different versions of files with the same name -- especially if you wish to retain more than the latest version.

Once you have cleaned up the existing structure then rsync would do well for incremental backups, with the caveat that only the latest version backed up would be retained.

If you want to retain more than the latest version you can use generations of incremental backup and keep rsync cycling through the generations.

Existing files you can examine and eventually hard-link with
rdfind