[Bioclusters] Ruby Queue

[Tim Cutts]

On 9 Feb 2006, at 10:27 pm, andy law (RI) wrote:

> All,
>
> One of my post-docs dropped this URL into my inbox.
>
> http://interactive.linuxjournal.com/article/7922
>
> It's quite old, but I don't remember seeing it discussed here  
> previously
> and I can't find mention of it in the archives.


That idea has been done, at quite large scale, by some organisations  
(Incyte, for one).  But they didn't use NFS; it doesn't scale well  
enough.  It can work quite well;  a broker process connected to a  
database knows about all the tasks, and one after another executes  
them on the list of currently available workers.

It works fine as long as the granularity of the workload is correct  
so you're not swamped in communication overhead and contention for  
accessing the task list, and also as long as the cluster is  
essentially doing tasks for only one political entity, so that first- 
come-first-served is a perfectly suitable scheduling policy.  If your  
cluster is shared by several groups with priorities which change over  
time, or with sudden deadlines, or any of a  million other reasons  
that FCFS is no longer a sensible scheduling strategy, you will need  
something more sophisticated.  You either go to the SGE's and LSF's  
of this world, or you divide your cluster up into small clusters each  
with their own broker, for particular tasks, which eventually turns  
out to be not a very efficient use of your resources.  It's worth  
noting that Incyte eventually abandoned their home-grown simple  
scheduler and bought LSF.

For a small lab cluster though, a simple solution like this is  
probably quite adequate.

Just my 0.02.

Tim

-- 
Dr Tim Cutts
Informatics Systems Group, Wellcome Trust Sanger Institute
GPG: 1024D/E3134233 FE3D 6C73 BBD6 726A A3F5  860B 3CDD 3F56 E313 4233