Re: [HTCondor-users] Forwarding Kerberos-Credentials

[Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf
> Of FB
> Sent: Saturday, October 21, 2017 2:40 AM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: [External] Re: [HTCondor-users] Forwarding Kerberos-Credentials
> 
> 
> unfortunately my users are not that organized. It's hard to even convince
> them to use batch processing and not doing everything manually in a bunch
> of open Matlab windows. They're unable to tell exactly, which data they
> need to do a certain computation. This is why I need them to have direct
> access to their projects' data residing on NFS-fileservers.

[Michael Pelletier] 
A very familiar refrain around here, Frank.

Unfortunately, I wound up encouraging that approach when I wrote the HTCondor Parallel Computing Toolbox interface for the distributed compute workers - it doesn't do any file transfers. The dependent .m files are all copied into scratch-mounted /tmp automatically, but there too it needs a shared filesystem.

I'm definitely keen on adding user file transfers to the tool as well. Part of my problem was that I had to write it for legacy R2008b installations, so things weren't quite as well-developed in the toolbox as they are now.

You should definitely keep encouraging them to use file transfers and scratch, though - perhaps entice them with the fact that by using local-disk scratch instead of NFS, I gave one of the programs a 5-7% increase in job speed because it kept all its interim file IO work off of the network and on to the server's fancy RAID controller that was previously sitting around doing next to nothing.

And I also found that the exercise of figuring out exactly which data they need in order to do a certain computation is a useful one, and leads to better understanding of the task, easier recognition of optimization avenues, and (hopefully) better documentation of the compute task to prevent some mysterious dependency from being accidentally broken in the future.

	-Michael Pelletier.