Re: [HTCondor-users] Docker universe error

["Beyer, Christoph" <christoph.beyer@xxxxxxx>]

Hi Iain,

I did start with 100M and did some iterations but ended with 1G did some more trials but the problem seems to be a different one: 

000 (234.000.000) 07/17 13:38:49 Job submitted from host: <131.169.71.96:48006?addrs=131.169.71.96-48006>
...
001 (234.000.000) 07/17 13:38:50 Job executing on host: <131.169.71.99:34112?addrs=131.169.71.99-34112>
...
007 (234.000.000) 07/17 13:38:51 Shadow exception!
	Error from slot1@xxxxxxxxxxxxxxxx: Docker job exhaused 5120 Mb memory
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
...
012 (234.000.000) 07/17 13:38:51 Job was held.
	Error from slot1@xxxxxxxxxxxxxxxx: Docker job exhaused 5120 Mb memory
	Code 34 Subcode 0
...
000 (235.000.000) 07/17 13:39:09 Job submitted from host: <131.169.71.96:48006?addrs=131.169.71.96-48006>
...
001 (235.000.000) 07/17 13:39:10 Job executing on host: <131.169.71.99:34112?addrs=131.169.71.99-34112>
...
007 (235.000.000) 07/17 13:39:11 Shadow exception!
	Error from slot1@xxxxxxxxxxxxxxxx: Docker job exhaused 10112 Mb memory
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
...
012 (235.000.000) 07/17 13:39:11 Job was held.
	Error from slot1@xxxxxxxxxxxxxxxx: Docker job exhaused 10112 Mb memory
	Code 34 Subcode 0
...

I did find a hint on some slides, it says: 

Exceeded memory limit?
	Just like vanilla job with cgroups
Â297.0  gthain     Â5/19 11:15 Error from slot1@localhost: Docker job exhaused 128 Mb memory

But what is that supposed to say ? 

cheers
        ~christoph


-- 
/*   Christoph Beyer     |   Office: Building 2b / 23     *\
 *   DESY                |    Phone: 040-8998-2317        *
 *   - IT -              |      Fax: 040-8994-2317        *
\*   22603 Hamburg       |     http://www.desy.de         */

----- Original Message -----
From: "Iain Steers" <iain.steers@xxxxxxx>
To: "htcondor-users" <htcondor-users@xxxxxxxxxxx>
Sent: Friday, 17 July, 2015 13:20:36
Subject: Re: [HTCondor-users] Docker universe error

Hi Christoph,

I've yet to try out the Docker universe.

My initial thought though is that perhaps the docker image itself is
being accounted for under memory usage.

i.e. your docker image running is > than the request memory.

Have you tried setting the request memory to something a lot higher like
3000M?

Cheers, Iain

On Fri, Jul 17, 2015 at 12:41:24PM +0200, Beyer, Christoph wrote:
> Hi,
> 
> I just upgraded my testbed to the fresh 8.3.6 release and everything looks really promising :) 
> 
> I tried the docker universe and everything works OK so far except my jobs do not run: 
> 
> [chbeyer@bm-test]/mnt/bshare/chbeyer% cat docker.submit
> universe                = docker
> docker_image            = centos:7
> executable              = /bin/cat
> arguments               = /etc/hosts
> should_transfer_files   = YES
> when_to_transfer_output = ON_EXIT
> output                  = out.$(Process)
> error                   = err.$(Process)
> log                     = log.$(Process)
> request_memory          = 1000M
> queue 1
> 
> results in: 
> 
> 000 (233.000.000) 07/17 10:54:55 Job submitted from host: <131.169.71.96:48006?addrs=131.169.71.96-48006>
> ...
> 001 (233.000.000) 07/17 10:54:56 Job executing on host: <131.169.71.99:34112?addrs=131.169.71.99-34112>
> ...
> 007 (233.000.000) 07/17 10:54:57 Shadow exception!
> 	Error from slot1@xxxxxxxxxxxxxxxx: Docker job exhaused 1024 Mb memory
> 	0  -  Run Bytes Sent By Job
> 	0  -  Run Bytes Received By Job
> ...
> 012 (233.000.000) 07/17 10:54:57 Job was held.
> 	Error from slot1@xxxxxxxxxxxxxxxx: Docker job exhaused 1024 Mb memory
> 	Code 34 Subcode 0
> 
> 
> Any hints someone ? 
> 
> cheers
>         ~christoph
> 
> 
> -- 
> /*   Christoph Beyer     |   Office: Building 2b / 23     *\
>  *   DESY                |    Phone: 040-8998-2317        *
>  *   - IT -              |      Fax: 040-8994-2317        *
> \*   22603 Hamburg       |     http://www.desy.de         */
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/