Re: [Condor-users] tuning a file server

[Steffen Grunewald <steffen.grunewald@xxxxxxxxxx>]

On Tue, Aug 02, 2005 at 08:07:39AM +0200, Alain EMPAIN wrote:
> Hello David & Steffen,
> 
> indeed, 3ware is a very stable solution for RAID server; it runs smootly 
>  for 3 years in my lab, but a few weeks ago I met problems with condor 
> / NFS on 3ware RAID after tripling my nodes.

I didn't change the number of nodes. The type of tasks being run
is something I cannot predict - a simple change, e.g. of the output
or input path used for the script, would cause huge effects.

> So I suspected not the RAID but the NFS configuration : the solution was 
> simply to launch more nfs server daemons (default=4 on SuSE) and tuning 
> the client side.
> 
> -------------------------
> SERVER: /etc/sysconfig/nfs (SuSE)
> 
> # the kernel nfs-server supports multiple server threads
> #
> #AE:050727:USE_KERNEL_NFSD_NUMBER="4"
> USE_KERNEL_NFSD_NUMBER="12"

I'm already running 24 of them (/etc/default/nfs-kernel-server, Debian)
RPCNFSDCOUNT=24

(and to answer another question of this thread: it's xfs,
as it was before)

> --------------------------
> CLIENT: /etc/fstab
> 
> 10.185.2.25:/home/grid  /home/grid      nfs \
>         rw,hard,nointr,tcp,vers=3,rsize=32k,wsize=32k,bg \
>         0 0

My *sizes are 8192. Should not make a big difference...

> Now all my nodes are 'blasting' steadily.

I guess it's the access pattern of the many clients: if
100+ separate processes try to access small chunks of well-
separated (geometrically) files, it's time for a very clever
read(-ahead) algorithm.
Since I started to see this behaviour when I switched to a 
2.6 kernel I'm now looking for tuning tricks.... There are
several "strategies" to choose from; unfortunately the
documentation seems to be quite sparse.

Steffen

-- 
Steffen Grunewald * MPI fuer Gravitationsphysik (Albert-Einstein-Institut)
SciencePark Golm, Am Mühlenberg 1, D-14476 Potsdam * http://www.aei.mpg.de
* e-mail: steffen.grunewald(*)aei.mpg.de * +49-331-567-{fon:7233,fax:7298}
No Word/PPT mails - http://www.gnu.org/philosophy/no-word-attachments.html