Monitor file (size) progress


Please note that this blog has been moved.

Now it has its own domain: mynixworld.info🙂

If you want to read the latest version of this article (recommended) please click here and I open the page for you.

From time to time I do several maintenance tasks that involve file copying, compressing, disk image cloning, etc.

Because sometimes (in fact almost always) these administrative task involves several commands that use the output/result of each other, I cannot use the GUI applications (which usually provides end-user feedback, like progress) but only CUI applications:

etc.

These programs, despite their strengths,  don’t provide a way to watch the progress while running. Of course, there are workarounds, sometimes there even exists some additional software that one can use to satisfy this requirement (see my notes at the bottom). But I don’t want too many dependencies (I have to depend on myself, so I think that’s enough) so, when I have such a situation I use the following strategy:

  • knowing the source file size and watching continuously  the target file size, I can approximate the percentage of work done

So what I do is basically:

  1. get the size of the source file (expressed in MB)
  2. list the target file and get its current size
  3. print on screen (echo) the result of 100*current_size/source_file_size

E.g.:

ls -las --block-size=1M target.file | awk '{print 100*$1/xxxx;}'

where target.file is the file which progress I do monitor and xxxx is the size of the file being copied.

But if you want to watch this process continuously, every 1 second, then you can automate this little bit. You can use the watch command where you can specify the command to watch every N seconds. The problem with this combination (our command and watch) is that the progress’s value (100*current/total) is evaluated at the beginning, before the watch is starting using it. So watch is no use for us. Instead we can use a while loop, and the sleep command, like below:

while [ true ];do <our command>;sleep 1;done

where <our command> is just the command I shown you earlier.

E.g.:

while [ true ];do ls -las --block-size=1M <strong>target.file</strong> | awk '{print 100*$1/<strong>100</strong>;}';sleep 1;done

Can that be made easier than this? Yes, it can. I wrote a bash script named fwatch (file watch) that allows me to specify the source and the target file, optionally the frequency for displaying the progress, and it provides me the current progress, nothing more, nothing less. I don’t need to remember the whole line of the code, I don’t need to check the source file size, the script does this by itself.

#!/bin/bash

if [ $# -lt 2 ];then
	echo "Invalid number of arguments!"
	echo -e "Usage:\n\t$0 <source-file> <file-to-monitor> [frequency]"
	echo -e "where\n\tsource-file : the file that is copied (we're using to guess the final size)"
	echo -e "\tfile-to-monitor : is the file being created (we monitor its current size)"
	echo -e "\tfrequency (seconds): optional, how often to check/print the file status"
	exit 1
fi

SRC=$1
DST=$2

FREQ=1
if [ -n "$3" ];then
	FREQ=$3
fi

SIZE=$(ls -las --block-size=1M ${SRC}|awk '{print $1}')

if [ -z "$SIZE" ];then
	SIZE=0
fi

while [ true ];do ls -las --block-size=1M $DST | awk -v size=$SIZE '{if (size == 0) done=0; else done=100*$1/size;printf "\rStatus: %.2f%% done",done;}';sleep $FREQ ;done

Using this script the only thing I have to do, instead the command I shown you earlier, is:

fwatch source.file target.file 

Notes:

  • to monitor the progress of something that goes via a pipe, one may use the Pipe Viewer (pv command), though I never tried
  • to monitor the progress of dd command one may use the following trick:
    • find the process id for the dd command (like ‘pgrep dd’)
    • send the kill command to that process using a special USR1 signal; the dd command knows how to react to this particular signal, so in fact the dd command will print-out the current progress (i.e. kill -USR1 pid)

About Eugen Mihailescu

Always looking to learn more about *nix world, about the fundamental concepts of arithmetic, algebra and geometry. I am also passionate about programming, database and systems administration.
This entry was posted in linux, shell and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s