On 9 Feb 2006, at 10:27 pm, andy law (RI) wrote: > All, > > One of my post-docs dropped this URL into my inbox. > > http://interactive.linuxjournal.com/article/7922 > > It's quite old, but I don't remember seeing it discussed here > previously > and I can't find mention of it in the archives. That idea has been done, at quite large scale, by some organisations (Incyte, for one). But they didn't use NFS; it doesn't scale well enough. It can work quite well; a broker process connected to a database knows about all the tasks, and one after another executes them on the list of currently available workers. It works fine as long as the granularity of the workload is correct so you're not swamped in communication overhead and contention for accessing the task list, and also as long as the cluster is essentially doing tasks for only one political entity, so that first- come-first-served is a perfectly suitable scheduling policy. If your cluster is shared by several groups with priorities which change over time, or with sudden deadlines, or any of a million other reasons that FCFS is no longer a sensible scheduling strategy, you will need something more sophisticated. You either go to the SGE's and LSF's of this world, or you divide your cluster up into small clusters each with their own broker, for particular tasks, which eventually turns out to be not a very efficient use of your resources. It's worth noting that Incyte eventually abandoned their home-grown simple scheduler and bought LSF. For a small lab cluster though, a simple solution like this is probably quite adequate. Just my 0.02. Tim -- Dr Tim Cutts Informatics Systems Group, Wellcome Trust Sanger Institute GPG: 1024D/E3134233 FE3D 6C73 BBD6 726A A3F5 860B 3CDD 3F56 E313 4233