Re: [HTCondor-users] transfer_in/output_files only if they exist

[John M Knoeller <johnkn@xxxxxxxxxxx>]

the trailing / does let HTCondor know it's a directory, but it ALSO says that the contents of the directory should be transferred to the root of the destination, so that doesn't really work here because we need the checkpoint file to stay in a directory.

-tj

-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
Sent: Thursday, February 7, 2019 12:16 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist

Good to know! When I encountered that, I saw a user education nightmare.

I recall trying some trailing-slash tricks to see whether that made condor think about the arguments as if they are a directory. Is anything like that supported?

Tom

ïOn 2/7/19, 12:10 PM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:

    I think that only happens when you enable filechecks, which is now off by default.
    
    condor_submit has no way of knowing that 
    
         transfer_output_files = checkpoint
      
    refers to a directory if the directory does not yet exist.  So with filechecks enabled, condor_submit would create a zero length file in order
    to see if you have permission to transfer the output. This would then prevent creating a directory later. 
    
    But filechecks is now off by default, in part because it caused problems like this.
    
    -tj
    
    
    -----Original Message-----
    From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Tom Downes
    Sent: Thursday, February 7, 2019 11:56 AM
    To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
    Subject: Re: [HTCondor-users] transfer_in/output_files only if they exist
    
    TJ:
    
    In my experience, when the directory doesn't exist at submit time, condor_submit creates a file that clobbers the reverse transfer of the directory upon job completion. So there's some badput involved.
    
    I haven't tested that experience in a while, though, so can't claim that it affects today's stable/dev releases.
    
    Tom
    
    On 2/7/19, 11:49 AM, "HTCondor-users on behalf of John M Knoeller" <htcondor-users-bounces@xxxxxxxxxxx on behalf of johnkn@xxxxxxxxxxx> wrote:
    
        You can do this by using a directory transfer.  If you specify a directory in the transfer in/out list.  then all of the files in the directory will be transferred - but it is not an error for the directory to be empty.  so if checkpoint is a directory on the submit side, then 
        
        transfer_input_files = checkpoint
        transfer_output_files = checkpoint
        
        Will transfer everything in the checkpoint directory on the submit side, to a checkpoint directory on the execute node, and then
        transfer everything from that directory back to the submit node. 
        
        Note - It is an error if the checkpoint directory does not exist on the submit side before you submit the job.
        
        -tj
        
        -----Original Message-----
        From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Duncan Brown
        Sent: Thursday, February 7, 2019 8:01 AM
        To: htcondor-users@xxxxxxxxxxx
        Subject: [HTCondor-users] transfer_in/output_files only if they exist
        
        Hi Todd,
        
        Is there a way to tell condor that it's OK if a specific file listed in transfer_input_files does not exist (and the same question with an output file)? The use case is using condor file i/o to manage a checkpoint file. The first time the job is run, the checkpoint file does not exist so the job gets stuck in hold state. I want to be able to tell condor that it's OK that this file is not there.
        
        Cheers,
        Duncan.
        
        -- 
        
        Duncan Brown                              Room 263-1, Physics Department
        Charles Brightman Professor of Physics     Syracuse University, NY 13244
        http://dabrown.expressions.syr.edu                   Phone: 315 443 5993
        
        
        _______________________________________________
        HTCondor-users mailing list
        To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
        subject: Unsubscribe
        You can also unsubscribe by visiting
        https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
        
        The archives can be found at:
        https://lists.cs.wisc.edu/archive/htcondor-users/
        
        _______________________________________________
        HTCondor-users mailing list
        To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
        subject: Unsubscribe
        You can also unsubscribe by visiting
        https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
        
        The archives can be found at:
        https://lists.cs.wisc.edu/archive/htcondor-users/
        
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/