[Bioclusters] Large numbers of files (again...?)

[Joe Landman]

On Wed, 2004-01-28 at 10:59, Dan Bolser wrote:
> ++ Joe Landman--
> > Hi Dan:
> >
> >   If it is a chore to have them switch, then the hashing function is a
> > much better choice.  I am not so sure about trusting real data to
> > Reiser, I have been (badly) burned by it in the past.
> 
> I know Reiser and xfs are still very new.

XFS is about 8 years old.  It is now in the Linux 2.4.24 kernel series.

> 
> Anyone actively using xfs?

Yes at a number of sites, and on all of my important machines.

> 
> I guess switch involves reformatting disks? Actually we have a large storage space
> yet to come 'on line'. It might be a good chance to try xfs.
> 
> 
> This reminds me, previously we have been using ext3 + raid + NFS, and for a while we
> were having some problems with this setup and we were wondering if the above
> combination was at fault.

ext3 has some issues with journal pressure.  Under heavy loads, the
journal becomes a bit of a liability as it is not an efficient execution
path (not well optimized on ext3 yet).  Combine ext3 and software RAID
and a heavy load, and you could hit the cases where ext3 does not do
well.  I have a few customers that see this regularly.

> 
> Can I ask what / how people set up file server space?
> 
> Actually it is a big problem for us - we have no disk use / backup policy yet.
> 
> Sorry if this a converstion you have already had - anyone provide a google interface
> to list archives?

This is a good conversation to have.  

> 
> Thanks very much for any feedback you can give, and thanks again for previous help,
> 
> Cheers,
> Dan.
> 
> >   Joe
> >
> > On Wed, 2004-01-28 at 09:34, Dan Bolser wrote:
> >> Thanks all for your help - not sure if I will try to persuade IT to make a shift
> >> to XFS or ReiserFS or just use the hashing function....
> >>
> >> I am very excited by this...
> >>
> >>
> >> > However, if you want to migrate to MySQL - check out this FS-lookalike  that
> >> lets you access MySQL data through a FS interface -
> >> > http://no.spam.ee/~tonu/modules.php?name=News&new_topic=2 . I'm not sure  it's
> >> ready for prime-time, though...
> >>
> >> and similar projects,
> >>
> >> But I think they need some time yet.
> >>
> >> Thanks again for help,
> >>
> >> Dan.
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Bioclusters maillist  -  Bioclusters@bioinformatics.org
> >> https://bioinformatics.org/mailman/listinfo/bioclusters
> > --
> > Joseph Landman, Ph.D
> > Scalable Informatics LLC,
> > email: landman@scalableinformatics.com
> > web  : http://scalableinformatics.com
> > phone: +1 734 612 4615
> >
> > _______________________________________________
> > Bioclusters maillist  -  Bioclusters@bioinformatics.org
> > https://bioinformatics.org/mailman/listinfo/bioclusters
-- 
Joseph Landman, Ph.D
Scalable Informatics LLC,
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
phone: +1 734 612 4615