[Bioclusters] Business-ish stuff: monitoring cluster usage, and how to pay for it all

[Joseph Landman]

On Tue, 2002-09-03 at 22:22, Tim White wrote:

> 1.  What charging scheme do you use?  Options range from a one-off "lifetime
> membership" charge for a whole company or university to charging by the
> wallclock or CPU minute.

Most charging metrics sum up the usage of the entire process, so you
charge by the process CPU-memory integral.  However, chargeback kills
usage.  Especially when cycles are so inexpensive to purchase.

> 2.  How much interest do you have from the commercial sector for using up
> unused clock cycles?  Is this a useful approach for meeting costs?

Many commercial organizations want to keep their data in-house for legal
reasons.  

> 3.  How do you prioritise these users fairly?

Define "fair" and its context.  Is fair defined as everyone getting an
equal fraction of the machine?  Or what they paid for?  Or a randomly
selected ticket from a queue?

There are unix based methods to set nice levels for various users.  For
clusterwide versions of this, you can use the schedulers tools, and
create groups of users within the jobscheduler.

> 4.  Do you have a way of deciding how many nodes should be allocated to a
> particular batch task, based on the number and size of other batch requests
> that have occurred or are likely to occur?

Depends upon the code, the usage patterns, and the definition of
"fair".  It is best to set a policy and reexamine if users start
yelling.

> 5.  Are there particular usage patterns you have discovered (e.g. length and
> frequency of batch jobs, number of nodes requested or allocated etc.), which
> are important to take into account?
> 6.  (More technical)  Is there any software you would recommend for
> collected this information automatically?

There used to be job accounting packages independent of schedulers.  The
LSF product has this capability (among many others).  I do not know the
PowerCloud tool, but it may have this capability as well.

 
> At the moment we are planning to allow an initial 6-month period of free
> access to any user, to determine the level of interest in using such a
> system, the kinds of usage patterns and to build up an idea of how to manage
> the system as we go along, but it would be really beneficial to hear from
> others who have been there.
> 
> Please let me know if there are any further details you need to know.  I
> look forward to your comments!
> 
> Thanks in advance,
> 
> Tim White
> 
> 
> _______________________________________________
> Bioclusters maillist  -  Bioclusters@bioinformatics.org
> https://bioinformatics.org/mailman/listinfo/bioclusters
-- 
Joseph Landman, Ph.D
Scalable Informatics LLC
email: landman@scalableinformatics.com
  web: http://scalableinformatics.com
phone: +1 734 612 4615