[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Specifying log files from python
- Date: Thu, 28 Dec 2017 14:44:17 -0500
- From: Larry Martell <larry.martell@xxxxxxxxx>
- Subject: Re: [HTCondor-users] Specifying log files from python
Thanks very much for the reply. Let me detail my use case. I have a
docker container and from there I want to submit thousands of jobs.
The content of the data, which changes on an hourly basis, determines
what jobs and what their arguments are. The jobs will be running on
bare metal not in a container. This is my first experience with
condor. From googling and researching and reading the docs it seemed
that using the python API would be the way to go with this. Reading
about that everything I found was about creating ClassAds.
If I were to do this using submit files, how would that work? Is there
a python API for creating those? Or would I have to just open the
files and write them out with python? I guess I could do this, but
then the management of all those files will become a headache. Putting
that aside for a moment, I would have to get the submit files from
inside the docker container to somewhere condor can find them. Do the
have to be on the condor master?
On Thu, Dec 28, 2017 at 11:37 AM, John M Knoeller <johnkn@xxxxxxxxxxx> wrote:
> That is not the equivalent submit classad for that submit file, itâs not
> even close. It might generously be considered âconceptuallyâ equivalent.
> But it *cannot* be used as an actual submit classad.
> There are at least a dozen required attributes missing, and the UserLog
> expression (and many others) must be a simple string, not an expression
> using strcat().
> You need to do the string concatenation in python and then assign the
> results to UserLog.
> Or better yet, ignore ANY guide to submitting jobs that does not use the
> htcondor.Submit() class. While submitting jobs as bare classads is
> technically possible, it is ridiculously hard to do correctly.
> If you want to see the equivalent submit classad for a given submit file,
> the condor_submit tool will show the resulting classad when you use either
> the -dump or -dry-run arguments.
> $ condor_submit -dump - example.sub
> Requirements=( TARGET.Arch == "X86_64" ) && ( TARGET.OpSys == "LINUX" ) && (
> TARGET.Disk >= RequestDisk ) && ( TARGET.Memory >= RequestMemory ) && (
> TARGET.HasFileTransfer )
> CondorVersion="$CondorVersion: 8.6.9 Dec 18 2017 BuildID: 427508 $"
> Arguments="foo bar"
> RequestMemory=ifthenelse(MemoryUsage =!= undefined,MemoryUsage,( ImageSize +
> 1023 ) / 1024)
> CondorPlatform="$CondorPlatform: x86_64_RedHat6 $"
> Notice that Out="test.out.0", and NOT an expression using strcat.
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf
> Of Larry Martell
> Sent: Wednesday, December 27, 2017 3:32 PM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] Specifying log files from python
> I am just getting started with condor. I do not have any submit files nor
> any legacy system. I am finding it very difficult to get going. I have been
> messing with it for 2 weeks and have yet to be able to successfully submit a
> On Wed, Dec 27, 2017 at 1:26 PM Todd Tannenbaum <tannenba@xxxxxxxxxxx>
> If you are using HTCondor v8.6.6 or above, I recommend using the
> htcondor.Submit() class to submit jobs instead of the Schedd() class. With
> the Submit class, you do not have to convert your submit file into a classad
> - the Submit class will do that for you. So if you have a condor_submit file
> that works, you are likely good to go. Much easier than trying to use the
> more primitive submit method in the Schedd class in my opinion. Details on
> the Submit class are in the Manual, and there is an example submitting a job
> with the Submit class in this ticket:
> Hope this helps
> Sent from my iPhone
> On Dec 27, 2017, at 11:59 AM, Larry Martell <larry.martell@xxxxxxxxx> wrote:
> On this site:
> I see this:
> For example, consider the following submit file:
> executable = test.sharguments = foo bar
> log = test.log
> output = test.out.$(Process)
> error = test.err
> transfer_output_files = output
> should_transfer_files = yes
> queue 1
> The equivalent submit ClassAd is:
> Cmd = "test.sh";
> Arguments = "foo bar"
> UserLog = "test.log";
> Out = strcat("test.out",ProcId);
> Err = "test.err";
> TransferOutput = "output";
> ShouldTransferFiles = "YES";
> So in my python code I create this logging in my ClassAd:
> Err =
> Out =
> UserLog =
> But I see this in the SchedLog:
> 12/27/17 12:34:15 (pid:3755290) WriteUserLog::initialize:
> ProcId)") failed - errno 2 (No such file or directory)
> /opt/capcompute/util/ is the dir the python script that is submitting
> the job is running from.
> What am I doing wrong here? How do I properly specify the path and
> file name for the logs?