[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] using hdfs with HTCondor awaring data locality

Easy... no.

I've pondered many options, but the most viable are:
1.) Create a classad function which determines distance(file_block, machine) and use that in a rank _expression_.  This is not pretty, because you would need to do that per block.
2.) Create a YARN based application. 



From: "ricky lee" <rickylee0815@xxxxxxxxx>
To: htcondor-users@xxxxxxxxxxx
Sent: Thursday, October 24, 2013 3:45:09 PM
Subject: [HTCondor-users] using hdfs with HTCondor awaring data locality

Hi all,

I saw a link of using HDFS with HTCondor. 

After deploying HDFS through condor, I want to submit a condor job with a requirement of HDFS data locality aware. For example, a job requirement should look like


Is there an easy way to express the HDFS data locality with Condor? thx.  

HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: