Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Understanding Condor Policies on Jobs

On 3/22/2013 3:18 AM, Andrey Kuznetsov wrote:


I've been reading the documentation, and slowly figuring out what is
what, but some things are unclear.
If you are going to be the HTCondor admin at your site, you may be interested in viewing one of our HTCondorWeek Administration tutorials from a past HTCondor Week workshop (or come to Madison this May for live tutorials!). Materials/slides from past HTCondor Weeks are online; back in 2008 Red Hat was kind enough to record some of them. At URL 
  http://research.cs.wisc.edu/htcondor/tutorials/videos/cw2008/
see the "Administrating Condor" tutorial, which I think covers most of the concepts/questions you are asking below.
I will take a quick pass at answering your questions inline below, but of course the tutorial does a much better job than my pithy comments... 


From the documentation, WANT_SUSPEND = A boolean expression that, when

True, tells Condor to evaluate the SUSPEND expression.
SUSPEND = A boolean expression that, when True, causes Condor to
suspend running a Condor job. The machine may still be claimed, but
the job makes no further progress, and Condor does not generate a load
on the machine.

From default config, UWCS_WANT_SUSPEND = ( $(SmallJob) ||

$(KeyboardNotBusy) || $(IsVanilla) ) && ( $(SUSPEND) )
So SUSPEND will be evaluated if the job is small, likely some kind of
error in the job, but I am having trouble understanding the rest.
The default UWCS policy expressions in the default config are not as simple as they could (should?) be. For better or worse, these expressions relate to the default policy that was in use at the UW-Madison Computer Sciences department a while back. Something you should know is there are a lot of standard universe (aka relinked with condor_compile so they can checkpoint, and with 'universe=standard' in the submit description file) jobs submitted at UW-Madison. Since standard universe jobs can checkpoint and restart right where they left off, the UWCS policy expressions are optimized in many places to take advantage of that. If you are not relinking with condor_compile, you probably submitting vanilla universe jobs. Off the top of my head, a simple setup for HTCondor to relinquish one processor core when someone is typing either on the console or via ssh would be: 

  # Jobs can start anytime on slots > 1, and also can
  # start on slot 1 if there has been no keyboard activity for 15 min
  START = SlotID > 1 || KeyboardIdle > 900
  # When we see keyboard activity on Slot1, send the job a SIGTERM
  # and if the job is still around 10 seconds later send a SIGKILL.
  WANT_SUSPEND = False
  WANT_VACATE = True
  PREEMPT = SlotID > 1 && KeyboardIdle < 60
  MachineMaxVacateTime = 10
  KILL = False
Note that all the slot (machine) classads will be numbered via an attribute SlotID (SlotID=1, SlotID=2, etc), and KeyboardIdle will be the number of seconds the keyboard (or ssh) has not had any keystrokes. 

Warning: I didn't test the above, I just wrote it in my email client :)

More inline below...


1) Why is SUSPEND evaluated if there is no user at the keyboard
"KeyboardNotBusy", shouldn't it be the opposite? If the keyboard is
busy then I want the SUSPEND to be evaluated on the basis that someone
is using the machine, thus I want the job to be suspended to free
resources/processor for the user.
Note that UWCS_WANT_SUSPEND says "... $(KeyboardNotBusy) || $(IsVanilla) ...".
So for vanilla jobs, it indeed works the way you thought it should. It is only if the job is not vanilla that we KeyboardNotBusy comes into play. The thinking here is if the job is standard universe, don't bother suspending the job, just checkpoint and migrate it to a different machine right away. 


2) Why is SUSPEND evaluated when the job is running in VANILLA
universe? We are submitting jobs under VANILLA universe and add our
own environmental variables inside the jobs. It doesn't make sense why
condor would attempt to suspend a VANILLA universe job.
The thinking is VANILLA jobs cannot necessarily be checkpointed, and thus if they are bumped off the machine they would have to restart from the beginning. So the idea of suspending the job for a few minutes before killing it off is in hopes that the keyboard user will go away soon. Kinda a bummer if you have a job that runs for 12 hours, and at hour 11 a guy just checks his email for 3 minutes then leaves... may be better to simply suspend the job for 3 minutes instead of forcing the job to start over and loose 11 hours of computing. (of course, suspending may irritate some users... while a suspended job uses no CPU, it will still consume RAM and/or virtual memory) 



3) Why is SUSPEND in WANT_SUSPEND since when WANT_SUSPEND=TRUE, then
SUSPEND is evaluated, seems kind of redundant?!



I guess it is not how I would have written it...


Regarding, UWCS_CONTINUE = ( $(CPUIdle) && ($(ActivityTimer) > 10) &&
(KeyboardIdle > $(ContinueIdleTime)) )
ActivityTimer = Amount of time in seconds in the current activity.
4) What kind of activity is the timer tracking? CONTINUE is supposed
to reactivate a suspended job, that means that when the machine is
free from users and nothing is running on it, then ActivityTimer is
somehow supposed to be non-zero, and thus > 10, so what is it
tracking? Is ActivityTimer tracking the time since last user
click/interaction was made, thus if the user steps away for more than
10 seconds, condor job will continue/resume?
Slots in HTCondor are always in a specific state and activity. You see this when you do condor_status. When HTCondor suspends a job (when SUSPEND becomes true), that slot will change from acivity "Busy" to activity "Suspended" and then HTCondor evaluates CONTINUE. So in the above, $(ActivityTimer) timer represents the number of seconds the slot has been in the "Suspended" activity. 


5) What's the purpose of WANT_SUSPEND and SUSPEND? Seems like they
accomplish the same thing, except you run the check twice. Does
WANT_SUSPEND has some other kind of use?
While a job is running, if WANT_SUSPEND is True, HTCondor startd will continuously evaluate the SUSPEND expression. If WANT_SUSPEND is FALSE, it will not even look at the SUSPEND expression and will just continuously evaluate the the PREEMPT expression. So essentially it is just a way enable folks to write less complicated expressions. 


6) Why are some variable in the config in the bash form, and others
not, or is it a typo?
Take a look at where SUSPEND is evaluated:
UWCS_WANT_SUSPEND = ( $(SmallJob) || $(KeyboardNotBusy) ||
$(IsVanilla) ) && ( $(SUSPEND) )
UWCS_PREEMPT = ( ((Activity == "Suspended") && ($(ActivityTimer) >
$(MaxSuspendTime))) || (SUSPEND && (WANT_SUSPEND == False)) )
The ones in bash form aka $() just simple macros expanded from elsewhere in the condor_config file. The ones without $() are likely referring to ClassAd attributes, which are either characteristics about the machine or characteristics of the job. I think the tutorials cover this pretty well... 


7) Are variables case sensitive? In condor_config_var, they are
printed as all capitals, but in the defaults UWCS they are used often
as lower-case with first capital letters of the word:
"$(ActivityTimer)" vs "ACTIVITYTIMER = (time() -
EnteredCurrentActivity)"
Macro and attribute names are both case-insensitive. For instance, $(Hour) and $(HOUR) are interchangeable. 


8) How do you differentiate between variables set/updated by condor
and variables that you define? Like SUSPEND is defined in the config
by user, but "KeyboardIdle" is not in the config.
If it has $() it is from the config file, if it does not have $() that means it is referring to an attribute about the machine (or job). 


9) What is =?= and =!= ?



See
http://research.cs.wisc.edu/htcondor/manual/v7.9/4_1HTCondor_s_ClassAd.html#SECTION00513400000000000000
Essentially, what happens if you write foo == 5, but foo is not defined? Should it be true? False? In HTCondor, it will not be True or False, but will evaluate to UNDEFINED. This so-called three-value logic is common in databases as well (think the Null value). Three-value logic lets folks write policies that explicitly deal with cases where information is missing (i.e. i want folks to submit jobs and tell me their department in the submit file, and want to do something special if someone forgot to specify their department). If you never want to deal with UNDEFINED and just want good-ol boolean two-value logic, use =?= instead of ==, and =!= instead of !=. 



I am using:
SLOTS_CONNECTED_TO_CONSOLE = 1
SLOTS_CONNECTED_TO_KEYBOARD = 1

10) How does condor know which SlotID to reserve for the user when the
desktop is being used? Where is this set?
No idea off the top of my head. Note in my simple example above, I didn't bother with SLOTS_CONNECTED_TO_KEYBOARD myself, and instead explicitly referenced SlotID in my Start/Preempt expressions. Seems more clear/explicit to me (but in more complex configurations it may make more sense to use SLOTS_CONNECTED_TO_KEYBOARD...). 


Here's what my SUSPEND looks line:
SUSPEND = ( ($(KeyboardBusy) || $(ConsoleBusy)) && ((SlotID <=
SLOTS_CONNECTED_TO_CONSOLE) || (SlotID <= SLOTS_CONNECTED_TO_CONSOLE))
&& $(ActivationTimer) > 90)
In other words, if console or keyboard is being used, and the SlotID
is 1, meaning processor #1 out of a total of 4 processors (cores) in
my computer, and the job is mature, has been running for some time,
then suspend the job.
PREEMPT = ( ((Activity == "Suspended") && ($(ActivityTimer) >
$(MaxSuspendTime))) || (SUSPEND) )
WANT_SUSPEND = ( $(SmallJob) || $(KeyboardBusy) || $(ConsoleBusy) )
CONTINUE = ( $(CPUIdle) && ($(ActivityTimer) > 10) && (KeyboardIdle >
$(ContinueIdleTime)) )

I welcome any suggestions to improve my attempts at forcing condor to
relinquish 1 processor when a user is utilizing the computer.

Thank you very much for taking a look.




Hope the above helps and welcome to HTCondor,
Todd