Write a script to move specific files in various folders to one folder

linuxisthedevil · 11-16-2010, 10:06 PM

Hello all,

Terribly new to Linux and find it mindboggling. I work on brain imaging and unfortunately all of the analysis runs on Linux, and I do not understand computers well coming as I do from a medical background. So my question -

There are various folders of patient MRI scans (folders called P1, P2, P3 etc) and within them are enclosed certain files that I am interested in (always called the same name in all folders, say image001). I would like a script that enables me to copy and move this image001 in all these individual folders to another folder altogether.

Any help would be greatly appreciated. I am at my wits' end!

stress_junkie · 11-16-2010, 10:11 PM

Hmmm. Linux is the devil? Let's look past that for now.

You don't have any Linux experience but you want to manipulate important information on a production system. Bad idea. You shouldn't learn Linux on a production system. That is doubly true since you say you don't have a computer technical background.

If this script is supposed to move files with the same name from different directories to a common directory then each file will overwrite its predecessor. You will end up with only one file which will be the last file copied.

I recommend that you hire someone who knows Linux.

Now back to your LQ user name. A while back Larry the Cable Guy commented on the incident where the Dixie Chicks criticized George Bush during a concert in Texas. He said that was like going to a trailer park and saying "Wal Mart stinks". I'd say that giving yourself a user name that defames Linux at a Linux lovers forum is similarly misguided.

linuxisthedevil · 11-16-2010, 10:19 PM

All names I wanted were taken, and since I am currently tearing my hair, that name seemed apt and hey, it was not taken!

What I meant was all these separate folders have a similarly named file, and I would like a script that singles these files out from each of these folders and pools them in another folder. Is it possible also that while the files are being assembled in the new folder, a code may be written for them to automatically number themselves serially?

stress_junkie · 11-16-2010, 10:26 PM

Quote:

Originally Posted by linuxisthedevil

What I meant was all these separate folders have a similarly named file, and I would like a script that singles these files out from each of these folders and pools them in another folder. Is it possible also that while the files are being assembled in the new folder, a code may be written for them to automatically number themselves serially?

Yes it is possible. Appending some unique identifier such as serial numbering to the file name would preserve each file as it is moved into the common directory. That would work the first time that you run the script. What happens the second time that you run the script? A simple script would start numbering at 1. Would that overwrite the first file in the common directory? I don't know how similar the file names are so I don't know. A script that would first look into the common directory, find the highest number in the file names, then start numbering new files +1 higher than the files already in the common directory may be required. If so then this is going to take a little bit of effort to construct.

I still think that you should not experiment on a production machine. You should set up a test machine which could be any computer. You should be sure that all of the files are copied to another device before you try to move them in case Murphy's Laws influence the outcome of your efforts. Please set up a test machine and make backups of your production system data before proceeding.

Please please please please please

linuxisthedevil · 11-16-2010, 10:45 PM

Oh yes, I am just tinkering around with copied data. This is as much a test run as it gets. I do understand your concerns!

The filenames are incidentally exactly the same. So, would something like this work?

i in N* -find image001* mv 'new directory'

(all folders begin with the letter N)

That's probably far too simple. But any ideas would be great. Thanks!

stress_junkie · 11-16-2010, 11:26 PM

I've asked GrapefruiTgirl to help. She really knows a lot of stuff. I've got to sign off for now. I'll check back in the morning.

adityavpratap · 11-16-2010, 11:54 PM

Quote:

Originally Posted by linuxisthedevil

Hello all,

There are various folders of patient MRI scans (folders called P1, P2, P3 etc) and within them are enclosed certain files that I am interested in (always called the same name in all folders, say image001). I would like a script that enables me to copy and move this image001 in all these individual folders to another folder altogether.

What I follow from the above is that you are having files with same names in various directories. Is it right? If that's so, then when you mv all these same-named files to any particular folder, won't they be overwritten?

chrism01 · 11-17-2010, 12:06 AM

You don't need to read all through these, but they do cover all the scripting you'll need
http://rute.2038bug.com/index.html.gz
http://tldp.org/LDP/Bash-Beginners-G...tml/index.html
http://www.tldp.org/LDP/abs/html/

See 'for' loops in the above links. I suggest you make a start, then show us what you've got if you get stuck.
Programming isn't brain surgery, it just requires attention to detail. Should be no problem for you.

adityavpratap · 11-17-2010, 12:29 AM

I hope you try this out on a test machine.

First task is to attach a unique identifier to each file in a particular directory so that none of the files share the same name. I would suggest you add the folders name to each file in every directory. Suppose you want to add the name of the directory to all the files in a particular directory, say P1
$ cd P1
$ for i in *.img ; do mv $i `echo P1`_$i; done

this process can be repeated with all the desired folders. (I'm sure there is a better way than doing it manually!)

The second task would be to move all the desired files to one particular destination folder. For example, let us say we want to move all files of the type .img from all folders to any particular destination folder (say all_images).
$ cd ~ (assuming all the folders P1, P2, etc., are under the home folder)
$ find . -type f -name "*\.img" -exec cp {} all_images\ \;
this is assuming that no other folders other than the desired folders (i.e., P1, P2, etc.) contain files of the type .img, otherwise they would also be copied to "all_images" folder.

This the best I could think of at the moment, with my limited knowledge of scripting. I know the gurus here will come up with a better way.

GrapefruiTgirl · 11-17-2010, 05:41 AM

Code:

#!/bin/bash
num=$((0))
find /path/to/parent/directory/tree/ -type f -name "image001" | while read line; do
     if [ $num = 0 ]; then
         for num in $(seq 1 1000); do
             [ -f "/path/to/destination/directory/image001-${num}" ] && continue || break
         done
     else
         num=$((num+1))
     fi
     mv -v "$line" "/path/to/destination/directory/image001-${num}"
done

Above is my suggestion, although there will be other ways of accomplishing this, some better, some maybe not..

Anyhow, each time this is code run, before moving/copying anything, it will see if the target filename already exists with an extension between 1 and 1000, and if so, the program chooses the first free number (highest file number, plus one). It only loops through the numbers on the first run, after which it saves the number (and increases by one) for next file. It will never over-write a file (if there's possibility of more than 1000 such files in the target directory, increase the number 1000 to something larger).

OP will need to put your own paths for the source and destination directory, and/or adjust the code to suit your requirements. Any questions, I'll be happy to answer them as best as I can. And if you find something wrong with the code (doesn't work as expected) let us know, though I have tested it here and it works fine.

Cheers!

devnull10 · 11-17-2010, 06:22 AM

Without knowing the exact directory structure it is difficult to give a fool-proof solution. If we assume that there is at most one level of subdirectories under the root, ie:

Code:

- Scans
  |_ Folder1
     |_ Image001.imr
  |_ Folder2
     |_ Image001.imr
  |_ Folder3
     |_ Image001.imr
  |
  |
...

then one could simply use the find command issued from the root directory and prepend the parent folder name to the image filename (easily parsed from the output of find).
So you would end up with:

Folder1_Image001.img
Folder2_Image002.img
etc.

Obviously this wouldn't work if you had something like this though:

Code:

- Scans
  |_ Folder1
     |_ Image001.imr
  |_ Folder2
     |_ Folder1
        |_ Image001.imr
...

In such a case you would have to prepend the full directory tree structure to the filename which may mean long filenames depending on the depth of the directory.

The advantage of this approach of course is that if you run it in the future it will not get duplicate filenames. Just to be certain though, your script should first check whether the file already exists, if it does then do a diff on it to check whether it is the same file, if it is then do nothing but if the files are different then do some other action (flag up, save to a "to investigate" folder etc.

devnull10 · 11-17-2010, 06:56 AM

Here is a very quick example I threw together assuming one level of heirarchy. You could probably make it strip out the slash and use multiple levels if you wanted-I don't have the time to do it no though lol. It's not the cleanest code by any means but you can tidy it up and suit it to your needs. It does make a couple of assumptions by excluding things in the intial find.
You can also probably remove the grep -v's on your system and exclude directories using the find command, I am using aix at the moment where find doesn't have all the options in other versions.

Code:

#!/bin/ksh

DEST=./new/

find . -type f | grep -v $0 | grep -v $DEST | while read i
do
  PRE=$(echo $i | rev | cut -d/ -f2- | rev | cut -b3-)
  FLN=$(basename $i)
  F_NAME="${PRE}_${FLN}"
  echo "Copying $F_NAME to ${DEST}${F_NAME}"
  cp "$i" "${DEST}${F_NAME}"
done

Code:

 ls -R
f1       f2       f3       f4       new      process
./f1:
image001

./f2:
image001

./f3:
image001

./f4:
image001

./new:

Code:

 ./process; ls -R
Moving f1_image001 to ./new/f1_image001
Moving f2_image001 to ./new/f2_image001
Moving f3_image001 to ./new/f3_image001
Moving f4_image001 to ./new/f4_image001
f1       f2       f3       f4       new      process
./f1:
image001

./f2:
image001

./f3:
image001

./f4:
image001

./new:
f1_image001  f2_image001  f3_image001  f4_image001

catkin · 11-17-2010, 08:24 AM

Quote:

Originally Posted by linuxisthedevil

I would like a script that enables me to copy and move this image001 in all these individual folders to another folder ...

Do you want to copy them or move them?

Are the P1, P2 ... directory names unique and significant? If they are unique then the original directory name could be added to image001 in the destination folder, as in P1.immage001, readily generating unique names. If they are significant then this would have the advantage of the new name both showing the significance and indicating their source.

devnull10 · 11-18-2010, 06:29 AM

Quote:

Originally Posted by catkin

Do you want to copy them or move them?

Are the P1, P2 ... directory names unique and significant? If they are unique then the original directory name could be added to image001 in the destination folder, as in P1.immage001, readily generating unique names. If they are significant then this would have the advantage of the new name both showing the significance and indicating their source.

The code I posted above does pretty much that.