[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] HTCondor Python



There is currently no way to do it and end up with a single cluster like condor_submit does, but you can do it in a single transaction.  You just do the looping in python, something like this.

# Python code:
...
sub = htcondor.Submit()
...
sub['executable']            = "myprog"
sub['transfer_input_files']  = "$(mydir)/"
# How to do "matching"???

with schedd.transaction() as txn
   for subdir, dir, files in os.walk('/path/to/somewhere'):
       sub['mydir'] = dir
       clusterId = sub.queue(txn, 1)

-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Antonio Dorta
Sent: Thursday, June 22, 2017 10:40 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] HTCondor Python

Hi!

I've been using HTCondor Python for a while and I'm wondering if there  
is a way to emulate the "queue var matching files/dirs" command that  
can be used in submit files. I basically need to create one job for  
each directory in a given path (directories have arbitrary names) and  
transfer all files of those directories. Something like:

# Submit file
...
executable              = myprog
transfer_input_files    = $(mydir)/
queue mydir matching dirs /path/to/somewhere/*

# Python code:
...
sub = htcondor.Submit()
...
sub['executable']            = "myprog"
sub['transfer_input_files']  = "$(mydir)/"
# How to do "matching"???

with schedd.transaction() as txn
   clusterId = sub.queue(txn, 1)


If that is not possible with HTCondor Python, I was also thinking in a  
workaround and build a CSV list with the names of the directories and  
then use $(Process) and some functions (substr, etc.) to access the  
list like an array, but there might be an easier way to do it...

Thanks!!

Antonio D.



-- 
Antonio Dorta
Servicios InformÃticos EspecÃficos (SIE)
InvestigaciÃn y EnseÃanza
Instituto de AstrofÃsica de Canarias (IAC)
C/ VÃa LÃctea, s/n. 38205 - La Laguna, Santa Cruz de Tenerife
Despacho: 1124. Tfno: 922 60 5278. email: adorta@xxxxxx
Supercomputing at IAC:  
http://www.iac.es/sieinvens/SINFIN/Main/supercomputing.php
----------------------------------------------------------------
ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de  
Proteccion de Datos, acceda a http://www.iac.es/disclaimer.php
WARNING: For more information on privacy and fulfilment of the Law  
concerning the Protection of Data, consult  
http://www.iac.es/disclaimer.php?lang=en

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/