Re: [HTCondor-users] Splitting stdout and stderr for a job running on multiple nodes

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Marco,

you can also rename files with
  transfer_output_remaps
when transfering them back (a good example is documented by SIE [1])

probably you can redirect stdout&stderr to (same named) named pipes per
job (to avoid file cluttering) and rename during transfer to include
each ID?

Cheers,
  Thomas

[1]
http://www.iac.es/sieinvens/siepedia/pmwiki.php?n=HOWTOs.CondorSubmitFile

On 2018-04-24 22:38, Marco Mambelli wrote:
> Thanks Jaime,
> Any suggestion on how to use the wrapper script to transfer the different stdout/err?
> 
> I thought about using tee and transferring back the file generated but I donât know if it will work.
> If multiple nodes transfer the same file probably theyâll overwrite each other.
> What is happening if they transfer the same directory and files in the directory are different?
> Can I use a wildcard in the transfer_output_files specification and transfer all matching files?
> 
> Thank you,
> Marco
> 
>> On Apr 19, 2018, at 3:36 PM, Jaime Frey <jfrey@xxxxxxxxxxx> wrote:
>>
>>> On Apr 18, 2018, at 4:54 PM, Marco Mambelli <marcom@xxxxxxxx> wrote:
>>>
>>> Greetings,
>>> Iâm submitting a condor job to a local batch manager (batch job, using blahp) and it is a multi-node job.
>>> I.e. one submission and the same executable is started on more than one node.
>>> Currently stdout and stderr from all the nodes are interleaved in the same file.
>>>
>>> Is it there a way to separate them?
>>> Separate files or having them in the same file one after the other would both be OK but interleaved messages make difficult to follow what is going on on a node.
>>>
>>> Thank you,
>>> Marco
>>
>>
>> Currently, thereâs no way to have HTCondor/blahp separate the stdout/err streams of a multi-node job.
>> Youâd have to use a wrapper script to get separated output.
>>
>> - Jaime
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature