Re: [Condor-users] DAGs and Error msg "Too open files"

[Ziliang Guo <ziliang@xxxxxxxxxxx>]

It appears that opening files through the C runtime API caps it to 512
files by default.  I'll look through the codebase to see if we
actually make the call to raise that limit.

On Fri, Dec 9, 2011 at 9:51 AM, Michael O'Donnell <odonnellm@xxxxxxxx> wrote:
> Kent, thanks. I have changed my DAG and submit files for the DAG by not
> specifying log files and I will be testing this today. I suspect this will
> help a lot but it seems like this would be a problem if I had a much
> larger pool. Our pool is not that large, but I try to always keep the
> scalability in mind as I learn and implement Condor for my employer.
>
> thank you for the help,
> mike
>
>
>
>
>
>
>
> From:
> "R. Kent Wenger" <wenger@xxxxxxxxxxx>
> To:
> Condor-Users Mail List <condor-users@xxxxxxxxxxx>
> Date:
> 12/09/2011 08:30 AM
> Subject:
> Re: [Condor-users] DAGs and Error msg "Too open files"
> Sent by:
> condor-users-bounces@xxxxxxxxxxx
>
>
>
> On Fri, 9 Dec 2011, Michael O'Donnell wrote:
>
>> I am running a DAG in our pool (all machines including the central
> manager
>> run on a Windows OS (XP, 7, R2) with about 200 available machines for
> this
>> specific job and my DAG reports an error which changes its status from
>> Running to Idle (that is to say the jobs are running fine but the
>> condor_dag.exe produces an error).
>>
>> The error log file the DAG produces this:
>> 12/09/11 06:22:48 Can't open "ExtSimVal_DAG.dag.dagman.out"
>> dprintf() had a fatal error in pid 9684
>> Can't open "ExtSimVal_DAG.dag.dagman.out"
>> errno: 24 (Too many open files)
>
> I'm not an expert on the Windows side of things, but one thing you *can*
> do is reduce the number of file descriptors DAGMan uses by having all of
> your node jobs use the same log file.  The easiest way to do this
> (assuming you're running a fairly recent DAGMan -- 7.5 or later should be
> good, I think) is to just not specify a log file in your submit files, and
>
> let DAGMan generate a default log for you.
>
> (One note: we are working on changes to DAGMan that will limit the number
> of file descriptors it uses at any given time, no matter how many log
> files are used by the DAG node jobs.  But those changes are not ready
> yet...)
>
> Kent Wenger
> Condor Team
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/



-- 
Condor Project Windows Developer