Re: [HTCondor-users] Job threads getting held.

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

Quick thought:

On the machine where you are submitting the jobs (i.e. where you run 
condor_submit), is the C: close to running out of disk space?  Or close 
to running out of space such that the C:\condor\spool directory may have 
space for some jobs but not enough space for output files for all 256 jobs?

Thanks
Todd


On 10/3/2014 1:59 PM, Alexandre Leão wrote:
> Hi Todd,
>
> I think the reason is failure to transfer and write files from the node to the
> central manager, to
> "C:\condor\spool\[ClusterID]\[JobID]\cluster[ClusterID].proc[JobID].subproc0.tmp\",
> but I don't know what could be causing this...
>
> Here's the submit file:
> --------------------------------------------------------------------------
> universe = vanilla
> executable =HALF_3D_barreira_abso.exe
> should_transfer_files = YES
> When_To_Transfer_Output = ON_EXIT_OR_EVICT
> transfer_input_files = data
> input = in.$(Process)
> output = out.$(Process)
> error = error.txt
> log = log.txt
> #Requirements = (OpSys == "WINNT50") || \
> #               (OpSys == "WINNT51")
> notify_user = never
> queue 256
> --------------------------------------------------------------------------
>
> And bellow is the output of condor_q -hold
>
> The Job does have an error log file, but it has zero bytes (it's empty), and the
> nodes where the held jobs where running are completly random. Sometimes it's 4,
> 9, 11, ...
>
> Thanks for your help!
> Alexandre
>
>
> Here's the output of condor_q -hold:
>
> -----------------------------------------------------------------------------------------------------
> C:\Users\Administrator>condor_q -hold
>
>
> -- Submitter: condor : <10.2.0.70:49200> : condor
>    ID      OWNER          HELD_SINCE  HOLD_REASON
>
>      6.209 julieta        10/3  19:29 Error from slot15@cluster05: STARTER at
> 10.2.0.55 failed to send file(s) to <10.2.0.70:54887>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\6\209\cluster6.proc209.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      6.236 julieta        10/3  19:29 Error from slot24@cluster05: STARTER at
> 10.2.0.55 failed to send file(s) to <10.2.0.70:54964>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\6\236\cluster6.proc236.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      6.248 julieta        10/3  19:29 Error from slot5@cluster02: STARTER at
> 10.2.0.52 failed to send file(s) to <10.2.0.70:54828>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\6\248\cluster6.proc248.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.11  julieta        10/3  19:28 Error from slot9@cluster03: STARTER at
> 10.2.0.53 failed to send file(s) to <10.2.0.70:61012>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\11\cluster8.proc11.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.35  julieta        10/3  19:29 Error from slot17@cluster10: STARTER at
> 10.2.0.60 failed to send file(s) to <10.2.0.70:58240>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\35\cluster8.proc35.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.50  julieta        10/3  19:28 Error from slot2@cluster03: STARTER at
> 10.2.0.53 failed to send file(s) to <10.2.0.70:61135>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\50\cluster8.proc50.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.67  julieta        10/3  19:28 Error from slot20@cluster03: STARTER at
> 10.2.0.53 failed to send file(s) to <10.2.0.70:61426>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\67\cluster8.proc67.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.71  julieta        10/3  19:29 Error from slot23@cluster03: STARTER at
> 10.2.0.53 failed to send file(s) to <10.2.0.70:61990>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\71\cluster8.proc71.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.77  julieta        10/3  19:29 Error from slot11@cluster08: STARTER at
> 10.2.0.58 failed to send file(s) to <10.2.0.70:58267>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\77\cluster8.proc77.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.79  julieta        10/3  19:29 Error from slot13@cluster08: STARTER at
> 10.2.0.58 failed to send file(s) to <10.2.0.70:58364>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\79\cluster8.proc79.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.83  julieta        10/3  19:29 Error from slot17@cluster08: STARTER at
> 10.2.0.58 failed to send file(s) to <10.2.0.70:58483>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\83\cluster8.proc83.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.95  julieta        10/3  19:19 Error from slot17@cluster02: STARTER at
> 10.2.0.52 failed to send file(s) to <10.2.0.70:62057>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\95\cluster8.proc95.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.110 julieta        10/3  19:28 Error from slot9@cluster10: STARTER at
> 10.2.0.60 failed to send file(s) to <10.2.0.70:62137>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\110\cluster8.proc110.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.114 julieta        10/3  19:28 Error from slot10@cluster10: STARTER at
> 10.2.0.60 failed to send file(s) to <10.2.0.70:62142>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\114\cluster8.proc114.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.116 julieta        10/3  19:28 Error from slot12@cluster10: STARTER at
> 10.2.0.60 failed to send file(s) to <10.2.0.70:62145>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\116\cluster8.proc116.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.121 julieta        10/3  19:29 Error from slot7@cluster11: STARTER at
> 10.2.0.61 failed to send file(s) to <10.2.0.70:58438>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\121\cluster8.proc121.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.123 julieta        10/3  19:29 Error from slot9@cluster11: STARTER at
> 10.2.0.61 failed to send file(s) to <10.2.0.70:58468>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\123\cluster8.proc123.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.143 julieta        10/3  19:28 Error from slot20@cluster10: STARTER at
> 10.2.0.60 failed to send file(s) to <10.2.0.70:62418>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\143\cluster8.proc143.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.146 julieta        10/3  19:29 Error from slot8@cluster06: STARTER at
> 10.2.0.56 failed to send file(s) to <10.2.0.70:58610>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\146\cluster8.proc146.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.162 julieta        10/3  19:28 Error from slot2@cluster10: STARTER at
> 10.2.0.60 failed to send file(s) to <10.2.0.70:62499>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\162\cluster8.proc162.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.163 julieta        10/3  19:29 Error from slot7@cluster10: STARTER at
> 10.2.0.60 failed to send file(s) to <10.2.0.70:62504>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\163\cluster8.proc163.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.168 julieta        10/3  19:28 Error from slot3@cluster07: STARTER at
> 10.2.0.57 failed to send file(s) to <10.2.0.70:62526>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\168\cluster8.proc168.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.178 julieta        10/3  19:29 Error from slot6@cluster07: STARTER at
> 10.2.0.57 failed to send file(s) to <10.2.0.70:62562>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\178\cluster8.proc178.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.186 julieta        10/3  19:29 Error from slot24@cluster09: STARTER at
> 10.2.0.59 failed to send file(s) to <10.2.0.70:59167>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\186\cluster8.proc186.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.194 julieta        10/3  19:29 Error from slot8@cluster01: STARTER at
> 10.2.0.51 failed to send file(s) to <10.2.0.70:58938>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\194\cluster8.proc194.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.199 julieta        10/3  19:29 Error from slot10@cluster07: STARTER at
> 10.2.0.57 failed to send file(s) to <10.2.0.70:62579>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\199\cluster8.proc199.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.202 julieta        10/3  19:29 Error from slot5@cluster07: STARTER at
> 10.2.0.57 failed to send file(s) to <10.2.0.70:62607>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\202\cluster8.proc202.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.215 julieta        10/3  19:29 Error from slot15@cluster07: STARTER at
> 10.2.0.57 failed to send file(s) to <10.2.0.70:62699>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\215\cluster8.proc215.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.218 julieta        10/3  19:29 Error from slot18@cluster07: STARTER at
> 10.2.0.57 failed to send file(s) to <10.2.0.70:62744>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\218\cluster8.proc218.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.222 julieta        10/3  19:29 Error from slot22@cluster07: STARTER at
> 10.2.0.57 failed to send file(s) to <10.2.0.70:62807>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\222\cluster8.proc222.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.235 julieta        10/3  19:29 Error from slot7@cluster04: STARTER at
> 10.2.0.54 failed to send file(s) to <10.2.0.70:62749>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\235\cluster8.proc235.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.237 julieta        10/3  19:29 Error from slot9@cluster04: STARTER at
> 10.2.0.54 failed to send file(s) to <10.2.0.70:62780>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\237\cluster8.proc237.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.242 julieta        10/3  19:29 Error from slot14@cluster04: STARTER at
> 10.2.0.54 failed to send file(s) to <10.2.0.70:62873>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\242\cluster8.proc242.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.243 julieta        10/3  19:29 Error from slot15@cluster04: STARTER at
> 10.2.0.54 failed to send file(s) to <10.2.0.70:62880>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\243\cluster8.proc243.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.249 julieta        10/3  19:29 Error from slot21@cluster04: STARTER at
> 10.2.0.54 failed to send file(s) to <10.2.0.70:62937>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\249\cluster8.proc249.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.252 julieta        10/3  19:29 Error from slot2@cluster11: STARTER at
> 10.2.0.61 failed to send file(s) to <10.2.0.70:62860>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\252\cluster8.proc252.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>      8.255 julieta        10/3  19:29 Error from slot12@cluster11: STARTER at
> 10.2.0.61 failed to send file(s) to <10.2.0.70:62822>; SHADOW at 10.2.0.70
> failed to write to file
> C:\condor\spool\8\255\cluster8.proc255.subproc0.tmp\_condor_stdout: (errno 2) No
> such file or directory
>
> 37 jobs; 0 completed, 0 removed, 0 idle, 0 running, 37 held, 0 suspended
> -----------------------------------------------------------------------------------------------------
>
>
>
> > Todd Tannenbaum <mailto:tannenba@xxxxxxxxxxx>
> > sexta-feira, 3 de Outubro de 2014 20:07
> >
> >
> > What is the hold reason(s)?  To see this, do "condor_q -hold". Also, could you
> > post your submit file?  That would help understand what is supposed to be
> > happening.
> >
> > regards,
> > Todd
> >
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
> >
> >
> >
> >
> >
> > Alexandre Leão <mailto:leao@xxxxxxxxxxxxx>
> > sexta-feira, 3 de Outubro de 2014 19:27
> > Hi all,
> >
> > I’m having some troubles getting HTCondor to work.
> >
> > When a user submits a job, some of the threads get held. This is random... For
> > example, in a job with 254 processes, 54 get held.
> >
> > Has this happened to anyone? Could someone help me?
> >
> > The central manager and the nodes are all Windows Server 2008 R3 (x64).
> >
> > Bellow are the logs.
> >
> > Thanks!!
> >
> > Alexandre
> >
> >
> > Here is the related output on SchedLog.log:
> >
> > […]
> > 10/03/14 11:49:30 (pid:3420) Starting add_shadow_birthdate(6.237)
> > 10/03/14 11:49:30 (pid:3420) Started shadow for job 6.237 on slot22@cluster09
> > <10.2.0.59:49163> for julieta, (shadow pid = 8692)
> > […]
> > 10/03/14 11:50:25 (pid:3420) Sent vacate command to <10.2.0.59:49163> for job
> > 6.237
> > […]
> > 10/03/14 11:50:28 (pid:3420) Shadow pid 8692 for job 6.237 exited with status 112
> > 10/03/14 11:50:28 (pid:3420) Putting job 6.237 on hold
> > […]
> > 10/03/14 17:04:37 (pid:3420) OwnerCheck(Administrator) failed in SetAttribute
> > for job 6.237
> > […]
> >
> > And here is the related output on ShadowLog.log:
> >
> > 10/03/14 11:49:30 Initializing a VANILLA shadow for job 6.237
> > […]
> > 10/03/14 11:49:32 (6.237) (8692): Request to run on slot22@cluster09
> > <10.2.0.59:49163> was ACCEPTED
> > […]
> > 10/03/14 11:50:25 (6.237) (8692): perm: OpenProcessToken failed: 5
> > […]
> > 10/03/14 11:50:25 (6.237) (8692):
> > perm::set_acls(C:\condor\spool\6\237\cluster6.proc237.subproc0): Unable to set
> > file ACL(err=0).
> > […]
> > 10/03/14 11:50:25 (6.237) (8692): perm: OpenProcessToken failed: 5
> > […]
> > 10/03/14 11:50:25 (6.237) (8692): perm:
> > SetNamedSecurityInfo(C:\condor\spool\6\237\cluster6.proc237.subproc0) failed
> > (err=5)
> > […]
> > 10/03/14 11:50:25 (6.237) (8692): (6.237) Failed to chown
> > C:\condor\spool\6\237\cluster6.proc237.subproc0 from to 8256944\8256608.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\fort.16, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 16 bytes of file
> > transmission
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\fort.16:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\GEOME, errno = 2: No such
> > file or directory.
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\GEOME:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\geome1, errno = 2: No such
> > file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 265438 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\geome1:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\h100238, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\h100238:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\i100238, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\i100238:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\lado12, errno = 2: No such
> > file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 1061752 bytes of file
> > transmission
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\lado12:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\ovalp1, errno = 2: No such
> > file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 265438 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\ovalp1:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\ovalp2, errno = 2: No such
> > file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 265438 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\ovalp2:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\ovalpm, errno = 2: No such
> > file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 265438 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\ovalpm:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\p100238, errno = 2: No
> > such file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\p100238:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\pontos, errno = 2: No such
> > file or directory.
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\pontos:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\RECEIVER, errno = 2: No
> > such file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 532350 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\RECEIVER:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\recfreq_amp, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\recfreq_amp:
> > (errno 2) No such file or directory
> >
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\recfreq_ri, errno = 2: No
> > such file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\recfreq_ri:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\rec_horizontal, errno = 2:
> > No such file or directory.
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): get_file(): consumed 430828 bytes of file
> > transmission
> > […]
> > 10/03/14 11:50:27 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\rec_horizontal: (errno 2)
> > No such file or directory
> > […]
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\rec_perpend, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 194176 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\rec_perpend:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\rec_vert1, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 162874 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\rec_vert1:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\res1_absorcao, errno = 2:
> > No such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\res1_absorcao: (errno 2)
> > No such file or directory
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\res2_absorcao, errno = 2:
> > No such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\res2_absorcao: (errno 2)
> > No such file or directory
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\RESULT, errno = 2: No such
> > file or directory.
> > […]
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 841921 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\RESULT:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\s100238, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\s100238:
> > (errno 2) No such file or directory
> > […]
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\v100238, errno = 2: No
> > such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 0 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\v100238:
> > (errno 2) No such file or directory
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\_condor_stderr, errno = 2:
> > No such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 783 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\_condor_stderr: (errno 2)
> > No such file or directory
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): Failed to open file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\_condor_stdout, errno = 2:
> > No such file or directory.
> > 10/03/14 11:50:28 (6.237) (8692): get_file(): consumed 4544 bytes of file
> > transmission
> > 10/03/14 11:50:28 (6.237) (8692): DoDownload: consuming rest of transfer and
> > failing after encountering the following error: SHADOW at 10.2.0.70 failed to
> > write to file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\_condor_stdout: (errno 2)
> > No such file or directory
> > […]
> > 10/03/14 11:50:28 (6.237) (8692): Mock terminating job 6.237:
> > exited_by_signal=FALSE, exit_code=8 OR exit_signal=0, core_dumped=FALSE,
> > exit_reason="Exited normally"
> > […]
> > 10/03/14 11:50:28 (6.237) (8692): Job 6.237 going into Hold state (code 12,2):
> > Error from slot22@cluster09: STARTER at 10.2.0.59 failed to send file(s) to
> > <10.2.0.70:50914>; SHADOW at 10.2.0.70 failed to write to file
> > C:\condor\spool\6\237\cluster6.proc237.subproc0.tmp\_condor_stdout: (errno 2)
> > No such file or directory
> > […]
> > 10/03/14 11:50:28 (6.237) (8692): **** condor_shadow (condor_SHADOW) pid 8692
> > EXITING WITH STATUS 112
> > […]
>
>
>
>
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/


--
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685