Perl File Handler - will autoflush clear the memory the file requires?

doug23 · 08-18-2009, 11:35 PM

Hi All,

Does $filehandle->autoflush(1) force Perl to clear the memory required by the $filehandle?

I understand that it flushed the buffer to the file after every write/print, but if I only need to append lines to a file, and never read the file, will this also reduce the memory requirements of my script?

I am using $filehandle along with Text::CSV_XS.

I will be reading in a >100GB file, and storing a large buffer of 20GB in filehandles waiting to write would significantly bog down (if not crash) the application.

Also, is there any difference in memory requirements between opening the filehandler's file in ">" mode versus append ">>" mode?

Are these the only two additions I would need to make to the code, or would I need to create a new IO object first?

Code:

use IO::Handle; 

	# create new output file for that year
	my $curOUTERRORfile = "ERRORSyear$CURyear.csv";

	my $curOUTERRORfilecsv = Text::CSV_XS-> new ({ binary => 1, eol => $/ });
	open my $curOUTERRORfilefh, ">", "$curOUTERRORfile" or die "$curOUTERRORfile: $!";
	$curOUTERRORfilefh->autoflush(1);

Thanks for your help!!!

Doug

chrism01 · 08-19-2009, 01:39 AM

autoflush only flushes the output buffer ie any/all 'data' previously 'written out' by a print cmd (or equiv).
It has nothing to do with any data you have in memory. Once you've issued a print cmd, that data is written to a mem buffer, and then to the disk if decided by the OS. If you have autoflush on, it writes it to disk immediately, instead of buffering. I'm pretty sure the buffer mem is still attached, just empty.
Think of it as write-through buffering.
http://perldoc.perl.org/IO/Handle.html

doug23 · 08-19-2009, 08:57 AM

Quote:

Originally Posted by chrism01

autoflush only flushes the output buffer ie any/all 'data' previously 'written out' by a print cmd (or equiv).
It has nothing to do with any data you have in memory. Once you've issued a print cmd, that data is written to a mem buffer, and then to the disk if decided by the OS. If you have autoflush on, it writes it to disk immediately, instead of buffering. I'm pretty sure the buffer mem is still attached, just empty.
Think of it as write-through buffering.
http://perldoc.perl.org/IO/Handle.html

Thanks. I have looked through the site before.

Right, so in terms of stuff I print (say 20-50gb) to that filehandle, will the code as it is written now no longer store the full 20-50gb in stuff I printed to the file in the buffer?

chrism01 · 08-20-2009, 12:32 AM

The buffer is a small fixed amt of mem, the OS doesn't try to buffer the whole lot in mem. Without autoflush, it fills the buffer and then writes to disk when the buffer fills up. With autoflush, it writes through the buffer to disk immediately.
Here's a really good page about it http://perl.plover.com/FAQs/Buffering.html