[Bioclusters] how to run BLAST on a linux cluster

[Chris Dagdigian]

Jie,

I bet that more than a few people on this list have done exactly what 
you are describing below.

All of the DRM software suites I mentioned in the last email will handle 
the task of batching, scheduling and executing your blast and other 
jobs. If you are looking for the free ones you have the following options:

o OpenPBS: http://www.openpbs.org
o GridEngine: http://www.sun.com/gridware/
o Condor: http://www.cs.wisc.edu/condor/

OpenPBS is probably the most popular right now especially as it is often 
dropped in by default when you by a generic linux cluster or compute 
farm from somebody. Veridian systems (the people who make and market the 
commercial variant) provide support for OpenPBS via a fairly active 
mailing list.

I only recommend OpenPBS to people and groups who are comfortable 
getting down and dirty with the source code and scheduler minutae. It is 
not an appropriate choice for clusters where for instance it is expected 
  that corporate IT types will be responsible for administration tasks.

Platform used to spread FUD about OpenPBS by claiming that OpenPBS 
cannot handle more than 5,000 jobs without losing/dropping some of them 
into a "black hole". There is apparently some grain of truth to his if 
you read some of the comments in the beowulf mailing lists. One thing 
that I don't like about OpenPBS is that often times the answer that 
people give on the support mailing list is "upgrade to PBSPro to get rid 
of that problem". There seem to be enough unfixed bugs and stability 
issues with the OpenPBS product that I usually tell people that if they 
are interested in PBS they would be better off in the long run by 
purchasing the commercial version. Veridian sells the commercial version 
(with source) for a very reasonable price (last time I checked it was 
roughly $100 USD per CPU).

I can't comment on Condor all that much except to say that I know there 
are at least a few academic institutions who swear by it and easily cope 
with tens of thousands of jobs. I've never run into condor outside of 
academia though.

GridEngine works great and is getting better all the time. It's also 
free and has no trouble scaling up to hundreds of nodes and many 
thousands of jobs. The company I work for pretty much exclusively uses 
Platform LSF or GridEngine for the informatics systems we work on.


The software layer is an important part but it is not 100% of the 
solution. Make sure that you purchase your cluster from someone who 
knows how you are going to use it (and how Blast behaves). At the end of 
the day a well-designed cluster architecture is going to be more 
important than your eventual choice of load management layers.

-Chris








jie hu wrote:

> Chris,
> 
> Thank you very much for your info.  We are considering to buy this Linux 
> cluster with 30-40 processors.  It will run other applications in 
> addition to BLAST.  For BLAST part, it will be used for batch job. For 
> example, we want our tens of thousands of sequences to be blasted every 
> week.  So basically, I am trying to get some program that will allow me 
> to run this batch BLAST on the Linux cluster.  I heard there are such 
> programs freely available and I only need to write some shell scripts or 
> perl to get it running.
> 
> Jie