[Bioclusters] BioPerl and memory handling

[Malay]

Mike Cariaso wrote:
> This message is being cross posted from bioclusters to
> bioperl. I'd appreciate a clarification from anyone in
> bioperl who can speak more authoritatively than my
> semi-speculation.
> 
> 
> Perl does have a garbage collector. It is not wildly
> sophisticated. As you've suggested it uses simple
> reference counting. This means that circular
> references will cause memory to be held until program
> termination.

True.

> 
> However I think you are overstating the inefficiency
> in the system. While the perl GC *may* not release
> memory to the system, it does at least allow memory to
> be reused within the process. 
> 
That's exactly the point. If you start created objects one after another 
in a long running program, the process starts hogging the system memory 
and stops excecution altogether.

> If the system instead behaved as you describe, I think
> perl would hemorrhage memory and would be unsuitable
> for any long running processes. 

It's simply hogs the system memory. Wheather it's suitable for a 
particular job or not depends on the particular job at hand. 
Fundamentally, if you are creating millions of objects in RAM, Perl is 
the the system for you. But there are ways around the problem ofcourse, 
which can be discussed. More than one ways to do things...

> However I can say with considerable certainty that
> that BPLite is able to handle blast reports which
> cause SearchIO to thrash. I've attributed this to
> BPLite being a true stream processor, while SearchIO
> seems to slurp the whole file and object heirarchy
> into memory.

A BioPerl-er will be able to tell better.
> 
> I know that SearchIO is the prefered blast parser, but
> it seems that BPLite is not quite dead, for the
> reasons above. If this is infact the unique benefit of
> BPLite, perhaps the documentation should be clearer
> about this, as I suspect I'm not the only person to
> have had to reengineer a substantial piece of code to
> adjust between their different models. Had I known of
> this difference early on I would have chosen BPLite.
> 
> So, bioperlers (especially Jason Stajich) can you shed
> any light on this vestigial bioperl organ?
> 
> 

Neither I'll support BioPerl nor go against it. All I am saying, there 
are nuts that can be cracked by BioPerl while there are nuts that can't 
be cracked so easily by it.

Malay