[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[HTCondor-users] Issues Running BLAST through HTCondor



Thank you!

Running one BLAST instance per host works for me.

On Wed, Mar 9, 2022 at 5:16 AM <htcondor-users-request@xxxxxxxxxxx> wrote:
Send HTCondor-users mailing list submissions to
    htcondor-users@xxxxxxxxxxx

To subscribe or unsubscribe via the World Wide Web, visit
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
or, via email, send a message with subject or body 'help' to
    htcondor-users-request@xxxxxxxxxxx

You can reach the person managing the list at
    htcondor-users-owner@xxxxxxxxxxx

When replying, please edit your Subject line so it is more specific
than "Re: Contents of HTCondor-users digest..."


Today's Topics:

 Â1. Re: Issues Running BLAST through HTCondor
   (dmitri.maziuk@xxxxxxxxx)
 Â2. Re: Docker_network_type (duduhandelman@xxxxxxxxxxx)
 Â3. Re: Docker_network_type (kaderno@xxxxxxxxx)
 Â4. Re: Docker_network_type (duduhandelman@xxxxxxxxxxx)


----------------------------------------------------------------------

Message: 1
Date: Tue, 08 Mar 2022 21:10:26 -0600
From: dmitri.maziuk@xxxxxxxxx
To: htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Issues Running BLAST through HTCondor
Message-ID: <6d056c30-c494-3ed4-e111-cc6ef1b06baf@xxxxxxxxx>
Content-Type: text/plain; charset=UTF-8; format=flowed

On 2022-03-08 6:44 PM, sameh.abedin@xxxxxxxxx wrote:
> I have 3 hosts that can execute jobs. 1 host has 16 slots, and the other
> 2 have 8 slots. So, 32 slots total.
>

How much RAM?

The best way to run blast is to put the databases on a RAM disk and run
multi-threaded on all available cores, with all of your query sequences
in the single query file ("all against all"). I.e. not really the way
condor does it.

I would try setting up whole-machine slots and only running one blast
instance per machine. It shouldn't be too hard to give it the correct
"numThreads" argument but you could start with the lowest common
denominator: 8. Your search DB seems small enough that RAM shouldn't
matter, but you really need enough RAM to hold it -- or 2GB, whichever's
smaller -- *per slot* just for blast.

That said, we only ran protein blast and when they switched to the new
and improved "blast+" it started taking 10 times longer which
effectively killed our pipeline. So I'm not sure what's happening with
your setup, exactly. Not enough RAM would be my first suspect if you
were searching nr or nt, but with your database I don't know.

Dima


------------------------------

Message: 2
Date: Wed, 09 Mar 2022 09:17:08 +0000
From: duduhandelman@xxxxxxxxxxx
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Docker_network_type
Message-ID:
    <VI1PR10MB32472C8A6FCBE4CC25988FBAA70A9@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>

Content-Type: text/plain; charset="us-ascii"

Pablo.
Which version are you using?

Thanks
David,
________________________________
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of kaderno@xxxxxxxxx <kaderno@xxxxxxxxx>
Sent: 09 March 2022 01:26
To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Docker_network_type

Hello all,

I hope you are doing fine. I'm trying to submit a docker universe job that makes use of the executor's host network, unfortunately I'm getting this warning message and the network type specification gets ignored by condor:

`WARNING: the line 'docker_network_type = host' was unused by condor_submit. Is it a typo?`


Has this option been removed/changed? Do I need to configure anything else prior to using that option?


Regards,
Pablo.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www-auth.cs.wisc.edu/lists/htcondor-users/attachments/20220309/74b3f0ce/attachment.html>

------------------------------

Message: 3
Date: Wed, 09 Mar 2022 21:05:50 +1100
From: kaderno@xxxxxxxxx
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Docker_network_type
Message-ID:
    <CALnuyD4QKvaVdHC9yJtJvpe2xh-XUtxM_VdjojNTb9cx61we9g@xxxxxxxxxxxxxx>
Content-Type: text/plain; charset="utf-8"

This warning message was on:
$CondorVersion: 8.8.15 Jul 29 2021 BuildID: 552034 PackageID: 8.8.15-1 $

Now I've switched to:
$CondorVersion: 8.9.13 Mar 30 2021 BuildID: 535058 PackageID: 8.9.13-1 $

and the warning message has disappeared but the docker command remains the
same.

Regards,
Pablo.

O m?r., 9 de mar. de 2022 ?s 20:17, <duduhandelman@xxxxxxxxxxx> escribiu:

> Pablo.
> Which version are you using?
>
> Thanks
> David,
> ------------------------------
> *From:* HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of
> kaderno@xxxxxxxxx <kaderno@xxxxxxxxx>
> *Sent:* 09 March 2022 01:26
> *To:* htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
> *Subject:* [HTCondor-users] Docker_network_type
>
> Hello all,
>
> I hope you are doing fine. I'm trying to submit a docker universe job that
> makes use of the executor's host network, unfortunately I'm getting this
> warning message and the network type specification gets ignored by condor:
>
> `WARNING: the line 'docker_network_type = host' was unused by
> condor_submit. Is it a typo?`
>
>
> Has this option been removed/changed? Do I need to configure anything else
> prior to using that option?
>
>
> Regards,
> Pablo.
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with
> a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www-auth.cs.wisc.edu/lists/htcondor-users/attachments/20220309/ffe9480f/attachment.html>

------------------------------

Message: 4
Date: Wed, 09 Mar 2022 10:15:17 +0000
From: duduhandelman@xxxxxxxxxxx
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Docker_network_type
Message-ID:
    <VI1PR10MB3247E91DBC8DC28A43DE629FA70A9@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>

Content-Type: text/plain; charset="iso-8859-1"

I think it was added in 8.9.
So all your poll should be 8.9

David

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of kaderno@xxxxxxxxx <kaderno@xxxxxxxxx>
Sent: Wednesday, March 9, 2022 12:05:50 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Docker_network_type

This warning message was on:
$CondorVersion: 8.8.15 Jul 29 2021 BuildID: 552034 PackageID: 8.8.15-1 $

Now I've switched to:
$CondorVersion: 8.9.13 Mar 30 2021 BuildID: 535058 PackageID: 8.9.13-1 $

and the warning message has disappeared but the docker command remains the same.

Regards,
Pablo.

O m?r., 9 de mar. de 2022 ?s 20:17, <duduhandelman@xxxxxxxxxxx<mailto:duduhandelman@xxxxxxxxxxx>> escribiu:
Pablo.
Which version are you using?

Thanks
David,
________________________________
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx<mailto:htcondor-users-bounces@xxxxxxxxxxx>> on behalf of kaderno@xxxxxxxxx<mailto:kaderno@xxxxxxxxx> <kaderno@xxxxxxxxx<mailto:kaderno@xxxxxxxxx>>
Sent: 09 March 2022 01:26
To: htcondor-users@xxxxxxxxxxx<mailto:htcondor-users@xxxxxxxxxxx> <htcondor-users@xxxxxxxxxxx<mailto:htcondor-users@xxxxxxxxxxx>>
Subject: [HTCondor-users] Docker_network_type

Hello all,

I hope you are doing fine. I'm trying to submit a docker universe job that makes use of the executor's host network, unfortunately I'm getting this warning message and the network type specification gets ignored by condor:

`WARNING: the line 'docker_network_type = host' was unused by condor_submit. Is it a typo?`


Has this option been removed/changed? Do I need to configure anything else prior to using that option?


Regards,
Pablo.
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx<mailto:htcondor-users-request@xxxxxxxxxxx> with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www-auth.cs.wisc.edu/lists/htcondor-users/attachments/20220309/6ee5ca66/attachment.html>

------------------------------

Subject: Digest Footer

_______________________________________________
HTCondor-users mailing list
HTCondor-users@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

------------------------------

End of HTCondor-users Digest, Vol 100, Issue 13
***********************************************