[Bioclusters] Best ways to tackle migration to dedicated cluster/Farm

Chris Dwan (CCGB) bioclusters@bioinformatics.org
Wed, 24 Mar 2004 15:45:43 -0600 (CST)


Ross,

You sound like something of an expert already!  That said, here are some
thoughts:

> I can no longer use this pool to expand capacity as our new
> CIO is dead against using this distributed model which seems a pity
> since 80 new DELL 2.8 GHz PCs  arrived on our site this month.

Do you have a feeling for where this anti-cycle-stealing attitude comes
from?  Like Chris Dagdigian said, it sounds like you've got benchmarking
well in hand, and there are a wide variety of examples of folks using
production workstations to augment their dedicated clusters.  If the
concerns are reliability, security, performance impact, or other technical
things, those can be worked with numbers and tests.

If, on the other hand, it's concern over trying something new, the Engine
system recently implemented at Novartis is a decent example of a
corporation gaining a great deal of horsepower this way.

However:

> Our CIO favours a fully dedicated system which would be great for us
> except his goals may not be identical to ours - he has cost drivers, we
> have performance drivers.

I will almost never get in the way of someone who wants to go to a
dedicated, cluster system if they need it to get their work done, and
they have the money to spend.  A well thought out, centralized resource
will almost always be easier (and cheaper) to administer than a cycle
stealing solution.

> Also, our existing farm uses "node pull"...

This is very cool.

I've heard stories about folks with big clusters using a "pull" based
system like Condor to backfill empty cycles on their scheduled cluster.

The bit about reading input from and sending output to a database server
rather than a fileserver is terrific.  I've found the same thing:  It's
far easier to get a connection to a database from some random node in
"grid land" than it is to reliably move files around, no matter what IO
layer (FTP, HTTP, NFS, ...) you choose.

-Chris Dwan