Re: [HTCondor-users] How to use Dagman (python bindings) in condor 8.6 version.

[Josh Karpel <karpel@xxxxxxxx>]

Both the Python bindings and normal condor submit files can submit multiple jobs with the same cluster id using queue statements. This is true whether or not you use DAGMan.

Generally, you should think of the cluster id as a "submission id". If you can submit your jobs in a single submit command (either from the command line tools or from Python), they will share a cluster id. If you can't, they won't. See your other question for some more advice on this.

Jobs themselves have no way to describe dependencies, so you'll need some tool on top of HTCondor to manage dependent execution. DAGMan is recommended, but many workflow tools can sit on top of HTCondor (such as MakeflowÂhttp://ccl.cse.nd.edu/software/makeflow/). However, I don't believe many/any of them can use the Python bindings submission API at present, if that's a hard requirement.

(Incidentally, I'm interested in this "single text file" idea. Would you mind sending a sample of what that would look like, maybe off-list if you prefer?)

Josh Karpel

karpel@xxxxxxxx

On Mon, Jan 27, 2020 at 10:59 PM Htcondor user <htcondor.user@xxxxxxxxx> wrote:

HiÂJosh Karpel,

Thanks for yourÂreply :)

I am building a library where user can submit multiple jobs by passing a text file with 'n' no of jobs.But currently i am getting different clusterids for each job.

But as per our requirement 'n' no of jobs specified in a single text file must be running in same cluster & also handle dependency jobs.

To acheive my requirement i have found Dagman. but unfortunately i am using condor 8.6.12 version and as of now i can't upgrade to latestÂversion.

Now how do i achieveÂ:Â

       Â1) Same clusterid for multiple jobs without dagman?

       Â2) How do i handle dependency between those jobs?

If I can't use submit API with cmd condor_submit_dag my.dag how can i acheive my requirement? any other alternative solution?

Thanks inÂadvance.

On Mon, Jan 27, 2020 at 7:59 PM Josh Karpel via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Hi,

In its current form, DAGMan requires real condor_submit submit files on disk to work, so there's no way to use the Python bindings to directly provide submit descriptions for DAGMan at runtime. This is definitely an area we're trying to improve, so I'd love to hear more about your specific use case, especially if the below information isn't as helpful as you'd like...

We're working on improving integration between the bindings and DAGMan. As mentioned in a previous thread, we now haveÂhttps://htcondor.readthedocs.io/en/v8_9_5/apis/python-bindings/api/htcondor.html#htcondor.Submit.from_dagÂin the 8.9 series, which lets you do the equivalent of condor_submit_dag from Python (but the submit files the DAG references still need to exist on disk).

We also have a prototypeÂ"DAG builder" API for Python atÂhttps://github.com/htcondor/htcondor-dagsÂ(examples atÂhttps://github.com/htcondor/htcondor-dags/tree/master/examples). It requires the 8.8 bindings, but you can install those manually from pip if you're using a linux system of some kind. It lets you pass submit descriptions to it in the form of Python bindings Submit objects.

Josh Karpel

karpel@xxxxxxxx

On Sat, Jan 25, 2020 at 11:10 AM Htcondor user <htcondor.user@xxxxxxxxx> wrote:

Hi all,

I want to use dagman(Python bindings) so that user can specify dependency between jobs.

Unfortunately, i am using 8.6 version and dagman is been exposed 8.9 version. due to some reasons i can't upgrade my version.

so do i have any other alternative?

To submit a job I already created a structure as below:

sub = htcondor.Submit({
  "executable": "/bin/sleep",
  "arguments": "5m",
  "request_cpus": "1",
})

If i call dagman cmd (condor_submit_dag my.dag) how should i pass this to my above mentioned submit method?

Can anyone please help in providing the alternative solution?

ThanksÂin advance :)

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/