script to move file from local filesystem to hadoop filesystem

callis07 · 07-19-2014, 07:15 AM

Hi All

Can you provide me a shell script which will move file from local filseystem to hadoop filesytem , by the checking the count of files in hadoop filesystem if it is less than 20 fils then move 10 files from local filesystem to hadoop filesystem.

T3RM1NVT0R · 07-19-2014, 07:24 AM

From the requirement it appears that you only want to move files from local filesystem to hadoop filesystem if there are 20 files on hadoop file system. Taking that into consideration, if you have got the following mount points /local and /hadoop on the system you can do it as follows:

Code:

#!/bin/bash
NUM=`ls /hadoop | wc -l`
if [ $NUM -lt 20 ]
then
    {
      for i in `ls /local | head -n 10`
      do
      {
         mv /local/$i /hadoop
         echo "10 files moved from local to hadoop fs"
      }
      done
     }
else
     {
       echo "Hadoop fs has got more than 20 files"
     }
fi

EDIT - I forgot to put wc -l for file NUM count. Updated that in code section.

callis07 · 07-19-2014, 08:14 AM

yes. some script like this.
But hadoop filesystem will be read something like
hadoop fs -ls /hadoop filesystem path/ -- to read a files in a specific directory
And word count cannot be done like using wc -l

and for moving the files we need to use something like below

hadoop fs -moveFromLocal /local path/ /hadoop filesytem path/

before performing this operation we need to login as hbase user

su - hbase

I will try to modify it from your scripts and see if it works. And Thanks for your script.

Thanks//

callis07 · 07-19-2014, 08:31 AM

I tried to modify something like this but amgetting error, Please can you correct me.

[scripts]# ./process.sh
./process.sh: line 2: hadoop: command not found
./process.sh: line 3: [: -lt: unary operator expected
Hadoop fs has got more than 20 files
[root@TMSKCRDM01 scripts]#

#!/bin/bash
NUM=`hadoop fs -count /var/CRS/processing/SDP`
if [ $NUM -lt 20 ]
then
{
for i in `ls /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend | head -n 10`
do
{
hadoop fs -moveFromLocal /var/CRS/cdr_bkp/june01/SDP1_0601/var/DWS/incoming/CS40/Mediation/CDRmedSend/$i /var/CRS/incoming/SDP
echo "10 files moved from local to hadoop fs"
}
done
}
else
{
echo "Hadoop fs has got more than 20 files"
}
fi

T3RM1NVT0R · 07-19-2014, 08:37 AM

It says hadoop command not found, it appears that the search path is not set to find hadoop related commands. So instead of:

Code:

NUM=`hadoop fs -count /var/CRS/processing/SDP`

use:

Code:

NUM=`/full_path_to_hadoop_binaries/hadoop fs -count /var/CRS/processing/SDP`

Wherein, full path to hadoop binaries is the actual path where hadoop binaries are located.

Instead of:

Code:

if [ $NUM -lt 20 ]

you can try:

Code:

if [ $NUM < 20 ]

Also run process.sh with sh -x process.sh which will give step by step information on what script is doing.