Re: [Condor-users] stork

[Matthew Farrellee <matt@xxxxxxxxxx>]

On 02/22/2010 08:35 AM, Mag Gam wrote:
> Currently we get all of our files from 1 NFS server but it seems like
> that is the bottleneck. On all of our 500 simulation boxes we have
> 100GB of local disk space available for scratch space. I would like to
> place pieces of the data into the 100GB disks and access it like that
> instead of the NFS server. Is it possible to do this?
> 
> TIA

You should check out the Condor HDFS work,

http://www.cs.wisc.edu/condor/manual/v7.4/3_13Setting_Up.html#sec:Condor-HDFS

And the FILETRANSFER_PLUGINS,

http://www.cs.wisc.edu/condor/manual/v7.4/3_13Setting_Up.html#sec:URL-transfer

Together they should do what you want.

There was a bug report about the combination lately, so if you run into a problem don't hesitate to shout.

Best,


matt