Re: [HTCondor-users] Rank expression for match making using a python script

[L Kreczko <L.Kreczko@xxxxxxxxxxxxx>]

Hi Jose,

Thanks for the reply. Answers are inline.

On 11 August 2016 at 15:35, Jose Caballero <jcaballero.hep@xxxxxxxxx> wrote:

HI Luke,

I am not really an expert, but anyways...
Quick question: when (or where) do you want to analyze the input
files? Before submitting the job, or once the job have been submitted
but just before the matchmaking?

After submission, before matchmaking. Essentially I want to be able to influence on which node a job can run based on the listed input files (or a custom jobAd).

If you can analyze the input files before submission, I guess you can
always set a convenient list of classads in the submit file, as a
result of the analysis, and then let the rank _expression_ + matchmaking
to pick the right node.

I see. So I could add a list to the JobAd asking for preferably and of the nodes in the list.

If that is not feasible, and must be done after submission, you can
try manipulating the job classads just before the matchmaking is being
called using the Job Router hooks
(which I am currently learning myself how to use, so still in the
process of understanding them):

   http://research.cs.wisc.edu/htcondor/manual/v8.5/4_4Hooks.html#SECTION00542000000000000000

Yes, I think this is where I've seen the python script being used. Job Router sounds good, might try this one:

PILOT_HOOK_TRANSLATE_JOB = /usr/libexec/htcondor-pilot-job-router/pilot-translate.py
JOB_ROUTER_HOOK_KEYWORD = PILOT

from https://github.com/treydock/htcondor-pilot-job-router

What I am not sure if that you will be able to inspect the content of
the input file at that point. Maybe you can. It will be nice to know.

Essentially I want to check the path of the input file (not the content) and ask our HDFS where the files (or chunks of them) are located.

This will give me a list of nodes that the job should run on (therefore reducing network traffic). So making HTCondor behave more like YARN.

Cheers,

Luke

2016-08-10 17:37 GMT-04:00 L Kreczko <L.Kreczko@xxxxxxxxxxxxx>:
> Dear experts,
>
> I am trying to write a simple python script that takes the job input files,
> does a quick analysis and returns useful results to condor for match making.
> I am very sure that this was demonstrated somewhere (python scripts in
> condor expressions), but I cannot find it in my notes nor via google.
>
> Could you please send me some pointers?
>
> What I want to do:
> - before matchmaking analyse job input files with python
> - return a result that condor can use for making the match making decision
> (i.e. some nodes are better for input A then others)
>
> Cheers,
> Luke
>
>

> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

*********************************************************
 Dr Lukasz Kreczko     ÂÂ
 Research Associate

 Department of Physics

 Particle Physics Group

 University of Bristol
 HH Wills Physics Lab
 University of Bristol
 Tyndall Avenue
 Bristol
 BS8 1TL

 +44 (0)117 928 8724Â

 L.Kreczko@xxxxxxxxxxxxx

ÂÂ

 A top 5 UK university with leading employers (2015)

 A top 5 UK university for research (2014 REF)

 A world top 40 university (QS Ranking 2015)

*********************************************************