Redirecting exact output to a file

sanaz · 02-18-2013, 05:12 PM

Hi all, I'm trying to redirect output of a script to a file. But output in file differs from what has been shown on screen.
Is there any way to redirect exact screen output to a file?

As the hardest part of my problem, I'm using wget to download 999,999,999 files and I want to have the output in a file(then I'm processing these data), but each percentage of download which is completed is shown in file for several time, I want the exact output of screen to a file.

Kustom42 · 02-18-2013, 05:41 PM

Make sure you are redirecting stderr and stdout.

http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html

sanaz · 02-18-2013, 05:51 PM

using

Code:

 wget http://speedtest.wdc01.softlayer.com/downloads/test500.zip > a.txt

and

Code:

wget http://speedtest.wdc01.softlayer.com/downloads/test500.zip 1> a.txt

does not write anything in a.txt
and

Code:

wget http://speedtest.wdc01.softlayer.com/downloads/test500.zip 2> a.txt

is not the same as screen output.

Any help?

Kustom42 · 02-18-2013, 05:55 PM

Code:

wget http://speedtest.wdc01.softlayer.com/downloads/test500.zip &> ./output.txt

You must use the &> to redirect both stdout and stderr. You should always check the man page too as most have redirect options built in.

-o ./output.txt would work just fine for wget.

sanaz · 02-18-2013, 05:58 PM

Thanks for your answer, but still the same problem exists with both of them.

Kustom42 · 02-18-2013, 06:03 PM

Then you are not seeing either stdout or stderr you either need to be more specific or what you are seeing cannot be redirected(which should be impossible but some ncurses and other things could be a factor).

sanaz · 02-18-2013, 06:29 PM

For time it differs.

Also for wget, here is the result:

Code:

$ wget http://speedtest.wdc01.softlayer.com/downloads/test500.zip
--2013-02-18 19:26:15--  http://speedtest.wdc01.softlayer.com/downloads/test500.zip
Resolving speedtest.wdc01.softlayer.com... 208.43.102.250
Connecting to speedtest.wdc01.softlayer.com|208.43.102.250|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 524288000 (500M) [application/zip]
Saving to: “test500.zip”

100%[========================================================================================================================================>] 524,288,000 43.4M/s   in 9.5s    

2013-02-18 19:26:30 (52.5 MB/s) - “test500.zip” saved [524288000/524288000]

and

Code:

$ wget http://speedtest.wdc01.softlayer.com/downloads/test500.zip &> ./output.txt
$ less output.txt 

--2013-02-18 19:28:12--  http://speedtest.wdc01.softlayer.com/downloads/test500.zip
Resolving speedtest.wdc01.softlayer.com... 208.43.102.250
Connecting to speedtest.wdc01.softlayer.com|208.43.102.250|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 524288000 (500M) [application/zip]
Saving to: “test500.zip”

     0K .......... .......... .......... .......... ..........  0% 2.01M 4m8s
    50K .......... .......... .......... .......... ..........  0% 3.85M 3m9s
   100K .......... .......... .......... .......... ..........  0% 3.87M 2m49s
   150K .......... .......... .......... .......... ..........  0% 3.95M 2m38s
   200K .......... .......... .......... .......... ..........  0% 81.5M 2m8s
   250K .......... .......... .......... .......... ..........  0% 3.97M 2m8s
   300K .......... .......... .......... .......... ..........  0% 87.4M 1m50s
   350K .......... .......... .......... .......... ..........  0% 4.09M 1m52s
   400K .......... .......... .......... .......... ..........  0% 65.0M 1m40s
   450K .......... .......... .......... .......... ..........  0% 4.21M 1m42s
   500K .......... .......... .......... .......... ..........  0% 75.1M 93s
   550K .......... .......... .......... .......... ..........  0% 83.4M 86s
   600K .......... .......... .......... .......... ..........  0% 72.0M 80s
   650K .......... .......... .......... .......... ..........  0% 4.53M 82s
   700K .......... .......... .......... .......... ..........  0% 62.7M 77s
   750K .......... .......... .......... .......... ..........  0% 77.8M 73s
   800K .......... .......... .......... .......... ..........  0% 4.63M 75s
   850K .......... .......... .......... .......... ..........  0% 77.3M 71s
   900K .......... .......... .......... .......... ..........  0% 72.3M 68s
   950K .......... .......... .......... .......... ..........  0% 74.3M 65s
  1000K .......... .......... .......... .......... ..........  0% 81.4M 62s
  1050K .......... .......... .......... .......... ..........  0% 75.9M 59s
  1100K .......... .......... .......... .......... ..........  0% 74.7M 57s

Kustom42 · 02-18-2013, 06:31 PM

So your only issue is the pretty bar that it draws out showing your download progress? You will have to deal with that. Your redirection is working as intended.

Kustom42 · 02-18-2013, 06:33 PM

If you are also concerned that your download times are slightly different, that is the way the internet works. Your speed is never going to be 100% the same, however it should stay within a relative range.

colucix · 02-19-2013, 01:55 AM

There is a difference between being connected to a terminal and sending output to a file, as the man page of wget states:

Quote:

The "bar" indicator is used by default. It draws an ASCII progress bar graphics (a.k.a
"thermometer" display) indicating the status of retrieval. If the output is not a TTY, the "dot"
will be used by default.

but you can force wget to display the "bar" indicator by means of:

Code:

wget --progress=bar:force -o output.txt http://speedtest.wdc01.softlayer.com/downloads/test500.zip

Hope this helps.

jpollard · 02-19-2013, 11:33 AM

Quote:

Originally Posted by sanaz

Hi all, I'm trying to redirect output of a script to a file. But output in file differs from what has been shown on screen.
Is there any way to redirect exact screen output to a file?

As the hardest part of my problem, I'm using wget to download 999,999,999 files and I want to have the output in a file(then I'm processing these data), but each percentage of download which is completed is shown in file for several time, I want the exact output of screen to a file.

Try the "script" utility. It will record everything sent to the screen, including any escape sequences for handling color, screen positions...

It even records the <cr><lf> of the physical I/O for the end of lines.

smbhandary · 02-19-2013, 10:55 PM

based on my understanding of your problem statement : content of log file differs from console output.

Try tee

<command> | tee /tmp/log -- to create /tmp/log or null existing file /tmp/log and write to it.
<command> | tee -a /tmp.log -- to append output to file /tmp/log

jpollard · 02-20-2013, 05:59 AM

Tee works sometimes - It depends on whether the program tests for a real terminal or not and generates different output.

Tee is designed for use in a pipe (as shown), but not with some other applications (such as an editor).

Even "ls" works differently - if the output is attached to a terminal then it will use a multi-column output with color highlighting. If it is attached to a pipe, then it outputs a single column, with no escape sequences for color highlighting.

linosaurusroot · 02-20-2013, 06:21 AM

wget http://speedtest.wdc01.softlayer.com...ds/test500.zip > ./output.txt 2>&1