Re: [HTCondor-users] HTCondor Python

[Antonio Dorta <adorta@xxxxxx>]

Hi!

Thanks a lot for your reply! I'd actually prefer to have one only  
clusterId, that makes much easier the code. I eventually solved the  
problem using $CHOICE(index,listname) function macro, building a list  
with all directories and using $(Process) as index... it works so far!  
;)

Thanks again.
Cheers,

Antonio Dorta



Quoting John M Knoeller <johnkn@xxxxxxxxxxx>:

> There is currently no way to do it and end up with a single cluster  
> like condor_submit does, but you can do it in a single transaction.   
> You just do the looping in python, something like this.
>
> # Python code:
> ...
> sub = htcondor.Submit()
> ...
> sub['executable']            = "myprog"
> sub['transfer_input_files']  = "$(mydir)/"
> # How to do "matching"???
>
> with schedd.transaction() as txn
>    for subdir, dir, files in os.walk('/path/to/somewhere'):
>        sub['mydir'] = dir
>        clusterId = sub.queue(txn, 1)
>
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On  
> Behalf Of Antonio Dorta
> Sent: Thursday, June 22, 2017 10:40 AM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: [HTCondor-users] HTCondor Python
>
> Hi!
>
> I've been using HTCondor Python for a while and I'm wondering if there
> is a way to emulate the "queue var matching files/dirs" command that
> can be used in submit files. I basically need to create one job for
> each directory in a given path (directories have arbitrary names) and
> transfer all files of those directories. Something like:
>
> # Submit file
> ...
> executable              = myprog
> transfer_input_files    = $(mydir)/
> queue mydir matching dirs /path/to/somewhere/*
>
> # Python code:
> ...
> sub = htcondor.Submit()
> ...
> sub['executable']            = "myprog"
> sub['transfer_input_files']  = "$(mydir)/"
> # How to do "matching"???
>
> with schedd.transaction() as txn
>    clusterId = sub.queue(txn, 1)
>
>
> If that is not possible with HTCondor Python, I was also thinking in a
> workaround and build a CSV list with the names of the directories and
> then use $(Process) and some functions (substr, etc.) to access the
> list like an array, but there might be an easier way to do it...
>
> Thanks!!
>
> Antonio D.
>
>
>
> --
> Antonio Dorta
> Servicios InformÃticos EspecÃficos (SIE)
> InvestigaciÃn y EnseÃanza
> Instituto de AstrofÃsica de Canarias (IAC)
> C/ VÃa LÃctea, s/n. 38205 - La Laguna, Santa Cruz de Tenerife
> Despacho: 1124. Tfno: 922 60 5278. email: adorta@xxxxxx
> Supercomputing at IAC:
> http://www.iac.es/sieinvens/SINFIN/Main/supercomputing.php
> ----------------------------------------------------------------
> ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de
> Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php
> WARNING: For more information on privacy and fulfilment of the Law
> concerning the Protection of Data, consult
> http://www.iac.es/disclaimer.php?lang=en
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/


--
Antonio Dorta
Servicios InformÃticos EspecÃficos (SIE)
InvestigaciÃn y EnseÃanza
Instituto de AstrofÃsica de Canarias (IAC)
C/ VÃa LÃctea, s/n. 38205 - La Laguna, Santa Cruz de Tenerife
Despacho: 1124. Tfno: 922 60 5278. email: adorta@xxxxxx
Supercomputing at IAC:  
http://www.iac.es/sieinvens/SINFIN/Main/supercomputing.php
----------------------------------------------------------------
ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de  
Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php
WARNING: For more information on privacy and fulfilment of the Law  
concerning the Protection of Data, consult  
http://www.iac.es/disclaimer.php?lang=en