[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] best way to use cached data
- Date: Sun, 13 Jan 2013 14:54:30 -0500
- From: John Wong <gokoproject@xxxxxxxxx>
- Subject: Re: [HTCondor-users] best way to use cached data
Just a minor update.
For caching data, we end up using iRODS.
It's basically the open-source, up-to-date version of SRB (storage resource brokers), very similar to SRM (storage resource management).
iRODs provides more integrated points for data lifetime, access and replication management.
Not the best, but I think it's a bit better than advertising because of the complexity of advertising. Imagine we are giving this to 100 users to run many jobs per day. Because we don't want users have direct access to the machine, or the abililty to write their own condor_submit <submit_file>, therefore, we are moving to iRODS for solution.
I will keep posting when we have our final solution setup. Besides using iRODS, we are also doing a bit of custom caching for other kinds of data.
On Sat, Jan 12, 2013 at 12:21 PM, Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>
As I understand startd cron, you configure your nodes to periodically run a script that publishes custom attributes, and the way a regular user would publish their custom attributes is by modifying the script. (Or am I missing something?)
On 1/12/2013 6:26 AM, Matthew Farrellee wrote:
I've had success with startd cron for advertising the contents of a
cache, and highly recommend it over configuration changes.
The script runs as condor (root) user, so security-wise this is worse than letting them 'sudo condor-reconfig' as now they can run anything as condor.
Either way, my point was that doesn't work when you're shipping (flocking, gliding) jobs off-site and have no control over execute nodes whatsoever.