[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Steering Jobs to a File on HDFS

There is no one supported method right now. Condor provides multiple mechanisms to achieve your goal, and there are number of people working on different approaches.

When the set of data-files is small, some have found advertising information about blocks in slot ads is a workable approach. The data info is then available to job RANK and REQUIREMENTS expressions. Others have looked into extending the ClassAd language with a verb that calculates how much of a file is present on a given node, and then used that information primarily in RANK expressions (though REQUIERMENTS is also possible).

Maybe someone who is actively doing this now can speak up and share some of their setup and code.



On 09/16/2010 06:23 PM, Dvorak DDV wrote:

Hi Experts,

I see activities of using HDFS as the underlying FS for Condor. As far
as I understand condor is copying the file out of HDFS before a job
runs, is this correct?

Since HDFS can be exported to workloads (jobs) directly via e.g. Fuse. I
am just wondering is it possible that, in a condor cluster where compute
worker nodes are also HDFS storage nodes, condor can send the job
directly to the node where the file is located on HDFS?


On Wed, Jul 21, 2010 at 4:24 PM, Yushu Yao <yao.yushu@xxxxxxxxx
<mailto:yao.yushu@xxxxxxxxx>> wrote:

    Hi Developers,

    In the 2009 what't new in Condor talk (
    ), page 32, it is said:

    "Thinking about how Condor can steer jobs to the data
    Via a ClassAd function used in the RANK expression?"

    Just wondering is this done now? If yes is there documentations on
    how to do it?

    Thanks a lot!

    | Yushu Yao
    | Ph:1-510-486-4690
    | Lawrence Berkeley National Lab
    | Mailstop 50B-6222
    | 1 Cyclotron Road
    | Berkeley CA 94720-8147 - USA

Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: