[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Remote Submit Scheds with grid submission to CondorCE

Hi Brian,

many thanks for the hint!
Talking to the CondorCE schedd directly [1] makes it much easier and is probably the best solution to avoid any local detours here.

AFAIS, one has to set the Owner to `undefined` for the CE to map the DN to a local user, or? I guess, it is not straight forward with plain `condor_q` to query jobs on the CE, as one would have to know the remote mapped user and there is no option to forward the X509 proxy for authz in condor_q.

Cheers and thanks,

ps: the general idea would be to use it not for actual user job submissions to the grid (probably there is hardly any user left using the grid directly) - but to use it for power users/VOs for debugging and for our functionality tests to become more generic.

> condor_submit -remote grid-htcondorce0.desy.de -pool grid-htcondorce0:9619 HTCondorCE.submit

> cat HTCondorCE.submit
# universe = grid
# grid_resource = condor grid-htcondorce0.desy.de grid-htcondorce0.desy.de:9619
universe = vanilla
use_x509userproxy = true
# how to deal with dn:uid mapping?
+Owner = undefined
# Files
executable = mypayload.sh
output = stdout
error = stderr
log = logs
# File transfer behavior
ShouldTransferFiles = YES
WhenToTransferOutput = ON_EXIT
# Resources CE
#+xcount = 1
#+maxMemory = 2000
#+maxWallTime = 10
### +remote_queue = "osg"  # Request the OSG queue #??

On 11/02/2021 18.29, Brian Lin wrote:
Hi all,

If there's no local Schedd/Gridmanager but your users have access to condor_submit, then you can run something like:

condor_submit -remote <CE FQDN> -pool <CE FQDN>:<CE PORT> <vanilla universe submit file>

This is effectively what condor_ce_trace [1] and condor_ce_run [2] do. However, there are a few caveats to this method of submission:

- HTCondor-CE is designed with pilot jobs in mind, not user jobs. This doesn't mean that users can't submit their jobs to a CE, it's just that the CE may be very aggressive in job cleanup in ways that are unfriendly to end users - X.509 proxies will be sent initially but renewals will not be forwarded on (assuming x509userproxy/use_x509userproxy are set appropriately) - Users will not to run commands to fetch their output, making this method particularly ineffective for long workflows

- Brian

[1] https://github.com/htcondor/htcondor-ce/blob/master/src/condor_ce_trace
[2] https://github.com/htcondor/htcondor-ce/blob/master/src/condor_ce_run

On 2/11/21 11:13 AM, Thomas Hartmann wrote:
Hi Maarten,

maybe I am overthinking it, but AFAIS the problem is the remote submit needing to forward the proxy

The user servers are as remote submitters not running a local schedd but only forward the jobs to the actual schedulers. I am not sure, if the X509s are forwarded, or?


On 11/02/2021 18.09, Maarten Litmaath wrote:
Hi Thomas,
could it be as simple as using a JDL like this:

cmd = ...
output = ...
error = ...
log = ...
+TransferOutput = ""
periodic_hold = ...
periodic_remove = ...
universe = grid
grid_resource = condor htc-ce.some.domain htc-ce.some.domain:9619
use_x509userproxy = true
environment = ...
queue 1

That is how ALICE let a local HTCondor service deal with all
the intricacies of handling jobs for remote HTCondor CEs.

Maybe I am missing something?

From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of Thomas Hartmann [thomas.hartmann@xxxxxxx]
Sent: 11 February 2021 17:46
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Remote Submit Scheds with grid submission to CondorCE

Hi all,

does somebody has maybe already experiences with combining remote
submission with gridward submission to CondorCEs?

Thing is, that we have a number of nodes for our users, that act as
remote submitters to the actual schedulers.

Now it would come handy, not only to submit to the local Condor but also
allow our users to do submissions into the Grid targeting CondorCEs.
Since the Schedd would have to do the authentication through the users'
proxy X509, it would need to be passed forward like kerberos etc.
And are there other caveats, that might need to be considered in such a

Maybe somebody has already a working similar setup?

ÂÂÂ Thomas

HTCondor-users mailing list
To unsubscribe, send a message tohtcondor-users-request@xxxxxxxxxxx  with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature