Hi All,

I have a set of shell programs which will read zip,txt,csv and jpg files from different ftp locations and process it everyday.

The ftp accounts are configured in a Windows server and shell programs are running a in Linux server. Both are sharing the same filesystem through NAS storage. The storage is basically an NTFS filesystem and is mounted in linux serevr using NFS sharing service provided by Microsoft.

The linux shell programs are directly accessing the ftp home directories in the storage and picking user uploaded files at specified intervals.

The problem what I am facing is, sometimes the files can be corrupted as it is being uploaded by clients over internet. We faced similar problems several times with corrupted files and it will create problem in shell script processing.

The shell scripts are processing in different stages and the final stage will call database procedures with processed data.

Now my question is, how I can find out and skip the processing for corrupted and incomplete text and csv files (also jpgs).

We have already taken care of

1. all empty and missing files
2. corrupted zip files (unzip -t <filename>)

Thanks,

MMN

One way would be to have the client checksum the files (md5sum, sha1sum, ...) and upload the checksum as well. Then when you pull the file you can compare the checksum, if they don't match its a corrupted file.