progress - to monitor progress of commands
 

hi all,

i have found a really good command that monitors the progress of a bunch of commands, its here -

https://github.com/Xfennec/progress

once installed you can run this command -

cp -r /mnt/local/data/call_the_midwife_7_1708/ /mnt/local/data/new/

then open a new terminal and run this command -

watch -n 0.5 progress -w

this will give you this -

Every 0.5s: progress -w Wed Sep 13 15:05:16 2017

[12254] cp /mnt/local/data/call_the_midwife_7_1708/Promo/grading_output/for_approval/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or/192
0x1080/170818_ctm_7_mipcom_graded_1-1_10bit_422_ycc_f2l_bl_or_V1.mxf
19.6% (2.3 GiB / 11.9 GiB) 27.4 MiB/s remaining 0:05:55

has anyone heard of this

but this gives you details of individual files being copied over, i need something that can give me the ETA and percent of the whole directory copied over and not just individual files?

rob

you can check the command pv

mmm...

got me thinking can i use the command pv and progress together to get the total ETA/percent of the whole directory instead of an ETA/percent of each individual file in the directory?

smashed it -

[root@robw-linux data]# tar -c call_the_midwife_7_1708/ | pv -lep -s 32455212 | tar -x -C /mnt/local/data/new/
[=> ] 2% ETA 2:34:31

and to find the dir size i did -

du -s call_the_midwife_7_1708/

but doing it via this method takes ages as its creating the tar and extracting the tar, normally doing a normal copy only takes roughly 18 minutes

probably cp -v -r is enough for you.

just thought of another idea -

il get the size of the source path -

du -s /source_path/

then i will start the copy -

cp -r /source_path/ /destination_path/

while im copying i will monitor the progress -

watch -n 0.5 du -s /destination_path/

but i want to do this all in a bash script but my issue is it wont watch the destination path while the copy is going on, how do i do both at the same time

rob

what do you mean by that? (what information do you need? what do you want to really achieve?)

im trying to achive so the end user gets a real time monitor/progress of how far the copy command is going and when its going to finish

it would be nice to get an ETA, percent and a progress bar, or just count up how big the directory size gets

probably this helps: https://www.cyberciti.biz/faq/show-p...file-transfer/

i have seen this -

https://stackoverflow.com/questions/...=votes#tab-top

if you read the answer by mitch he has made a script that does this with echo commands

but when i try it and run it on my linux box i just get 33, 66, 100% with the hashes, how can i get this to count up from 1-100

also how can i implement this with my copy command "cp -r /source /dest"

many thanks,

rob

sorted it,


yum remove rsync


https://download.samba.org/pub/rsync/rsync-3.1.2.tar.gz


installed it by untarring it cd'd into the dir and running "./configure.sh" "make" "make install"


and now i get the result i wanted -


[root@robw-linux data]# rsync -a --info=progress2 call_the_midwife_7_1708/ new/
14,874,971,690 44% 27.58MB/s 0:10:48 xfr#16, to-chk=2/143)

now i have the rsync command progress sorted now i want to do the same with the find command

i have done the commands below -

cd /source_dir/

find . -type f -exec md5sum {} \; >> /md5sum/source.txt

this works fine but i want to see the progress of the find command, is there a way i can do this via pv command or something else

many thanks,

rob

something like this:
(if you want to say thanks just click on yes)

thanks pan64 but how is this going to show me the real life progress of the find command

find has no any idea about the amount of work/time required by the -exec commands.
find also has no idea about the amount of files to look for/to be processed, so it cannot tell you any kind or percentage/progress (or similar).
If you want to do that you need to collect all the files reported by find into a (status) file and you can use pv or similar on that file. But that will slow down the whole process.