Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Missing `x509UserProxyVOName` ClassAd in Condor 10.0.3

Hi Thomas,

I don't see classAd AuthTokenIssuer in any of our jobs. Is it being injected by HTCondor-CE? Note we use ARC.Â
It seems we will have to write a JOB_TRANSFORM to keep backwards compatibility.

Cheers,
Jose

El vie, 5 may 2023 a las 9:48, Thomas Hartmann (<thomas.hartmann@xxxxxxx>) escribiÃ:
Hi Jose,

AFAIS the token issuer ad should be somewhat a good identifier for the
VO. I.e., you could maybe write a transform matching & selecting on
 ÂAuthTokenIssuer
and inject the corresponding VO into x509UserProxyVOName, if it does not
exists.

As us, we see a mixed back of GSI and token authz jobs [1], so that one
might need to be careful not accidentially miss a case. (being a bit
worried about occasional token only jobs and the still large number of
pure GSI jobs)

Cheers,
 ÂThomas

[1]
Â> condor_ce_q -af AuthTokenIssuer x509UserProxyVOName | sort | uniq -c
  Â548 https://atlas-auth.web.cern.ch/ atlas
   Â4 https://atlas-auth.web.cern.ch/ undefined
  Â474 https://cms-auth.web.cern.ch/ cms
   Â3 https://cms-auth.web.cern.ch/ undefined
  4497 undefined belle
   Â6 undefined desy
   51 undefined ilc
  Â280 undefined lhcb
  Â672 undefined ops



On 05/05/2023 10.18, Jose Caballero wrote:
> Hi Maarten,
>
> Thanks a lot for the explanation.
> Would it be possible to replicate the old functionality with a
> JOB_TRANSFORM ?
>
> Cheers,
> Jose
>
> El vie, 5 may 2023 a las 9:00, Maarten Litmaath
> (<Maarten.Litmaath@xxxxxxx <mailto:Maarten.Litmaath@xxxxxxx>>) escribiÃ:
>
>Â Â ÂHi JosÃ,
>Â Â Âin 10.x there is no code that looks into the VOMS extensions that an
>Â Â ÂX509 proxy may have
>Â Â Âand hence there are no variables defined anymore for the VO and the
>Â Â ÂFQANs.
>
>Â Â ÂWe will need to decide on sustainable ways for the accounting to
>Â Â Âkeep working...
>
>
>Â Â Â------------------------------------------------------------------------
>Â Â Â*From:* HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx
>Â Â Â<mailto:htcondor-users-bounces@xxxxxxxxxxx>> on behalf of Jose
>Â Â ÂCaballero <jcaballero.hep@xxxxxxxxx <mailto:jcaballero.hep@xxxxxxxxx>>
>Â Â Â*Sent:* Friday, May 5, 2023 9:33 AM
>Â Â Â*To:* HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx
>Â Â Â<mailto:htcondor-users@xxxxxxxxxxx>>
>Â Â Â*Cc:* condor-users@xxxxxxxxxxx <mailto:condor-users@xxxxxxxxxxx>
>Â Â Â<condor-users@xxxxxxxxxxx <mailto:condor-users@xxxxxxxxxxx>>
>Â Â Â*Subject:* Re: [HTCondor-users] Missing `x509UserProxyVOName`
>Â Â ÂClassAd in Condor 10.0.3
>Â Â ÂHi,
>
>Â Â Âif I understand correctly what I see, the classAd mentioned by Tom,
>Â Â Â"x509UserProxyVOName", is added to the jobs at the schedd level.
>Â Â ÂPicking one random job on a schedd 9.0.5, this is the submit file
>Â Â Âfrom the CE middleware (ARC) [1] and these are the classad of the
>Â Â Âsubmitted job [2].
>Â Â ÂSo clearly the classAds x509* have been added by our local Schedd.
>
>Â Â ÂHowever, on a schedd 10.0.3, some of those classAds are missing [3].
>
>Â Â ÂI have downloaded the code from GITHUB, and a simple grep gives me
>Â Â Âthe same results for the main branch and tag V9_0_5.
>Â Â ÂAlso, the classAd x509UserProxyVOName is still mentioned in the
>Â Â Âdocumentation.
>Â Â ÂSo I am quite lost as well. Why suddenly the jobs submitted from
>Â Â Âschedd 10.x are missing these classads?
>
>Â Â ÂAny comment/question is more than welcome.
>
>Â Â ÂCheers,
>Â Â ÂJose
>
>Â Â Â[1]
>Â Â Â[root@arc-ce04
>Â Â ÂPG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm]# cat
>Â Â Âcondorjob.jdl
>Â Â Â# HTCondor job description built by arex
>Â Â ÂExecutable = condorjob.sh
>Â Â ÂInput = /dev/null
>Â Â ÂLog =
>Â Â Â/var/spool/arc/grid00/PG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm/log
>Â Â ÂOutput =
>Â Â Â/var/spool/arc/grid00/PG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm.comment
>Â Â ÂError =
>Â Â Â/var/spool/arc/grid00/PG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm.comment
>Â Â Â+NordugridQueue = "EL7"
>Â Â ÂDescription = gridjob
>Â Â ÂUniverse = vanilla
>Â Â ÂNotification = Never
>Â Â ÂRequirements = (NumJobStarts == 0) && ( (OpSys == "LINUX" &&
>Â Â ÂOpSysMajorVer >= 7) )
>Â Â ÂPriority = 0
>Â Â Âx509userproxy =
>Â Â Â/var/spool/arc/grid00/PG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm/user.proxy
>Â Â Ârequest_cpus = 1
>Â Â Ârequest_memory=4000
>Â Â Â+JobMemoryLimit = 4096000
>Â Â Âshould_transfer_files = YES
>Â Â ÂWhen_to_transfer_output = ON_EXIT_OR_EVICT
>Â Â ÂTransfer_input_files =
>Â Â Â/var/spool/arc/grid00/PG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm
>Â Â ÂPeriodic_remove = (JobStatus == 1 && NumJobStarts > 0) ||
>Â Â Â((ResidentSetSize isnt undefined ? ResidentSetSize : 0) >
>Â Â ÂJobMemoryLimit)
>Â Â ÂQueue
>
>Â Â Â[2]
>Â Â Â[root@arc-ce04
>Â Â ÂPG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm]# condor_q -l
>Â Â Â2479042 | grep ^x509
>Â Â Âx509userproxy =
>Â Â Â"/var/spool/arc/grid00/PG3NDmbdVE3nE6QDjqmt6UqoABFKDmABFKDmhZLKDmABFKDmpuXrkm/user.proxy"
>Â Â Âx509UserProxyEmail = "lb.pilot@xxxxxxx <mailto:lb.pilot@xxxxxxx>"
>Â Â Âx509UserProxyExpiration = 1683605339
>Â Â Âx509UserProxyFirstFQAN = "/lhcb/Role=pilot/Capability=NULL"
>Â Â Âx509UserProxyFQAN = "/DC=ch/DC=cern/OU=Organic
>Â Â ÂUnits/OU=Users/CN=lbpilot/CN=693025/CN=Robot: LHCb
>Â Â Âpilot,/lhcb/Role=pilot/Capability=NULL,/lhcb/Role=NULL/Capability=NULL"
>Â Â Âx509userproxysubject = "/DC=ch/DC=cern/OU=Organic
>Â Â ÂUnits/OU=Users/CN=lbpilot/CN=693025/CN=Robot: LHCb pilot"
>Â Â Âx509UserProxyVOName = "lhcb"
>
>Â Â Â[3]
>Â Â Â[root@arc-ce-test01 ~]# condor_history -l 605625.0 | grep ^x509
>Â Â Âx509UserProxyEmail = "Andrea.Sciaba@xxxxxxx
>Â Â Â<mailto:Andrea.Sciaba@xxxxxxx>"
>Â Â Âx509UserProxyExpiration = 1682927827
>Â Â Âx509userproxy =
>Â Â Â"/var/spool/arc/grid05/ZPsKDmZFHD3n61QDjqWNiMpoABFKDmABFKDmAaFKDmAEFKDmDzgJen/user.proxy"
>Â Â Âx509userproxysubject = "/DC=ch/DC=cern/OU=Organic
>Â Â ÂUnits/OU=Users/CN=sciaba/CN=430796/CN=Andrea Sciaba"
>
>
>
>
>Â Â ÂEl mar, 2 may 2023 a las 10:33, Thomas Hartmann
>Â Â Â(<thomas.hartmann@xxxxxxx <mailto:thomas.hartmann@xxxxxxx>>) escribiÃ:
>
>Â Â Â Â ÂHi Thomas,
>
>Â Â Â Â Âfrom Condor 10 on GSI is not supported anymore but only token authz.
>Â Â Â Â ÂAlso IIRC has ATLAS recently switched Harvester submission to
>Â Â Â Â ÂCondor 10
>Â Â Â Â Âas well, so that their jobs do not get submitted anymore with
>Â Â Â Â ÂX509 ads.
>
>Â Â Â Â ÂProbably the only option on the midterm run would be to add
>Â Â Â Â Âcases for
>Â Â Â Â Âroutes, that evaluate the Auth* ads similar as for X509 ads.
>
>Â Â Â Â ÂCheers,
>Â Â Â Â Â Â ÂThomas
>
>Â Â Â Â ÂOn 02/05/2023 10.07, Thomas Birkett - STFC UKRI via
>Â Â Â Â ÂHTCondor-users wrote:
>Â Â Â Â Â > Hi Condor community,
>Â Â Â Â Â >
>Â Â Â Â Â > I hope you are all keeping well, hopefully a simple fix but Iâve
>Â Â Â Â Â > recently upgraded our test Condor pool from 9.0.15 to 10.0.3
>Â Â Â Â Â(LTS) and I
>Â Â Â Â Â > notice that jobs no longer show the ClassAd
>Â Â Â Â Ââx509UserProxyVONameâ. The
>Â Â Â Â Â > following x509 classads are present when running a `condor_q
>Â Â Â Â Â-l *jobid*`
>Â Â Â Â Â >
>Â Â Â Â Â > x509UserProxyEmail
>Â Â Â Â Â >
>Â Â Â Â Â > x509UserProxyExpiration
>Â Â Â Â Â >
>Â Â Â Â Â > x509userproxy
>Â Â Â Â Â >
>Â Â Â Â Â > x509userproxysubject
>Â Â Â Â Â >
>Â Â Â Â Â > however, ` x509UserProxyVOName` is missing.
>Â Â Â Â Â >
>Â Â Â Â Â > This is a problem for us as a large proportion of our Job
>Â Â Â Â ÂTransforms use
>Â Â Â Â Â > this missing ClassAd `x509UserProxyVOName`. Downgrading to
>Â Â Â Â ÂCondor
>Â Â Â Â Â > 9.0.15, the ClassAd is then applied to new incoming jobs. Any
>Â Â Â Â Âhelp in
>Â Â Â Â Â > debugging this issue would be gratefully received.
>Â Â Â Â Â >
>Â Â Â Â Â > Many thanks,
>Â Â Â Â Â >
>Â Â Â Â Â > *Thomas Birkett*
>Â Â Â Â Â >
>Â Â Â Â Â > Senior Systems Administrator
>Â Â Â Â Â >
>Â Â Â Â Â > Scientific Computing Department
>Â Â Â Â Â >
>Â Â Â Â Â > Science and Technology Facilities Council (STFC)
>Â Â Â Â Â >
>Â Â Â Â Â > Rutherford Appleton Laboratory, Chilton, Didcot
>Â Â Â Â Â > OX11 0QX
>Â Â Â Â Â >
>Â Â Â Â Â > signature_609518872
>Â Â Â Â Â >
>Â Â Â Â Â >
>Â Â Â Â Â > _______________________________________________
>Â Â Â Â Â > HTCondor-users mailing list
>Â Â Â Â Â > To unsubscribe, send a message to
>Â Â Â Â Âhtcondor-users-request@xxxxxxxxxxx
>Â Â Â Â Â<mailto:htcondor-users-request@xxxxxxxxxxx> with a
>Â Â Â Â Â > subject: Unsubscribe
>Â Â Â Â Â > You can also unsubscribe by visiting
>Â Â Â Â Â > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>Â Â Â Â Â<https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users>
>Â Â Â Â Â >
>Â Â Â Â Â > The archives can be found at:
>Â Â Â Â Â > https://lists.cs.wisc.edu/archive/htcondor-users/
>Â Â Â Â Â<https://lists.cs.wisc.edu/archive/htcondor-users/>
>Â Â Â Â Â_______________________________________________
>Â Â Â Â ÂHTCondor-users mailing list
>Â Â Â Â ÂTo unsubscribe, send a message to
>Â Â Â Â Âhtcondor-users-request@xxxxxxxxxxx
>Â Â Â Â Â<mailto:htcondor-users-request@xxxxxxxxxxx> with a
>Â Â Â Â Âsubject: Unsubscribe
>Â Â Â Â ÂYou can also unsubscribe by visiting
>Â Â Â Â Âhttps://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>Â Â Â Â Â<https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users>
>
>Â Â Â Â ÂThe archives can be found at:
>Â Â Â Â Âhttps://lists.cs.wisc.edu/archive/htcondor-users/
>Â Â Â Â Â<https://lists.cs.wisc.edu/archive/htcondor-users/>
>
>Â Â Â_______________________________________________
>Â Â ÂHTCondor-users mailing list
>Â Â ÂTo unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
>Â Â Â<mailto:htcondor-users-request@xxxxxxxxxxx> with a
>Â Â Âsubject: Unsubscribe
>Â Â ÂYou can also unsubscribe by visiting
>Â Â Âhttps://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>Â Â Â<https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users>
>
>Â Â ÂThe archives can be found at:
>Â Â Âhttps://lists.cs.wisc.edu/archive/htcondor-users/
>Â Â Â<https://lists.cs.wisc.edu/archive/htcondor-users/>
>
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/