Closest thing to rsync emulation with FTP client?

kikinovak · 06-24-2013, 06:19 AM

Hi,

I have 100 GB space on a mutualized web server, e. g. a machine on which I only have an FTP account, but no shell access. This machine hosts my new package repository. Unfortunately I have no rsync access, so synchronizing the local file tree with the remote file tree is a bit of a headache.

To make things worse, I have a slow DSL connection with a ridiculous upload bandwidth of about 40 kB/s. That's the price you pay for living in the south French countryside.

So far, GFTP and Filezilla don't really behave like they should. I guess the solution will be a short script using lftp, ncftp or whatever command-line client will be apt for the job.

Here's how the script should behave, preferably.

If a remote file is identical to the local copy, proceed to the next file.
If a remote file is present but different than the local file (in content), re-upload and squash it.
If a remote file exists where there is no local equivalent, delete the file.

In short, the whole thing should be equivalent to:

Code:

$ rsync -av --delete [localstuff] [remote_server]

Any suggestions?

cendryon · 06-24-2013, 06:33 AM

Hi

The "mirror" command from lftp is what you're looking for:

Code:

# lftp -c "open ftp://remote.host.example ; mirror /path/to/local/source path/to/remote/target ;"

Eric uses it in his mirror-alien-repository.sh for instance.
I use it to keep my local Slackware mirrors up to date.

Cheers

GazL · 06-24-2013, 08:04 AM

I believe you'll need "mirror -R" in lftp when uploading.

kikinovak · 06-24-2013, 08:19 AM

I've been playing around with this for a little while, and here's the result, which works as expected:

Code:

#!/bin/sh
LFTP=$(which lftp)
$LFTP -c "set ftp:list-options -a;
open ftp://user:password@remoteserver;
set net:limit-rate 25000
lcd localdir;
cd remotedir;
mirror --continue --reverse --dereference --parallel=5 --delete --verbose "

--continue allows to resume partial uploads after an interruption
--reverse means mirror from here to there and not to there from here
--dereference allows handling of symlinks in the local file tree
I've decided to limit the upload speed so I can continue to surf the web while uploading

I'll test this some more, but first results are quite encouraging. I guess I'll write a blog post about this.

Cheers,

Niki