[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Transfer output files also on Job removal?



Dear Mark,

Am 20.10.20 um 23:43 schrieb Mark Coatsworth:
> Hi Oliver,
> 
> We have a little known knob that I think does exactly what you want.
> Try adding the following to your submit file:
> 
> +SpoolOnEvict = false
> 
> I just did a test here and that seems to send the specified output
> files back into my submit directory. Please give it a try and let us
> know?

wow! Yes, that does the trick, indeed â this knob is a hidden gem for this use case :-). 

While I tried to find the code doing the spooling, I apparently missed the correct place[0],
so many thanks for pointing this out! 
Many users here in Bonn will be thankful once we take our JupyterHub in production :-). 

Cheers and all the best,
	Oliver

[0] https://github.com/htcondor/htcondor/blob/a2b31117a4a7deb12fac208cd0b2b6012555eb88/src/condor_starter.V6.1/jic_shadow.cpp#L437-L440

> 
> Mark
> 
> On Tue, Oct 20, 2020 at 4:17 PM Michael Pelletier via HTCondor-users
> <htcondor-users@xxxxxxxxxxx> wrote:
>>
>> This came up in the 2019 HTCondor Week, and I discussed it a bit with the team, but I don't recall the specifics. Maybe one of them will chime in.
>>
>> Michael V Pelletier
>> Principal Engineer
>>
>> Raytheon Technologies
>> Information Technology
>> Digital Transormation & Innovation
>>
>>
>>
>> -----Original Message-----
>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Oliver Freyermuth
>> Sent: Tuesday, October 20, 2020 7:27 AM
>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>> Cc: Peter Wienemann <peter.wienemann@xxxxxxxxxxx>
>> Subject: [External] [HTCondor-users] Transfer output files also on Job removal?
>>
>> Dear HTCondor experts,
>>
>> I've been looking for a knob, but could not find one for this use case:
>> We'd love to transfer the output files of some jobs even in case of removal, i.e. condor_rm, or periodic_remove e.g. when a time limit is hit.
>>
>> The example use case here are interactive Jupyter notebooks: We don't want to share a file system with all the resources the notebooks may run on, and would love to use HTCondor file transfer to get things to and fro.
>> However, interactive jobs are usually either cancelled by a time limit (i.e. periodic_remove), or (in case of JupyterHub's batchspawner) by condor_rm, so file transfer would be especially helpful for interactive use cases without shared file system.
>>
>> So it burns down to this:
>>   Is there a way to also trigger file transfer on job removal (I have control over both schedd and submit file)?
>>
>>
>> I have for now tried to set:
>>   when_to_transfer_output = ON_EXIT_OR_EVICT
>>   TransferOutputFiles = jupyter
>>   ShouldTransferFiles = True
>> and if I parse the logs correctly, indeed this triggers a file transfer to SPOOL on the schedd node, but then the files are purged afterwards, since the job was removed. Having them transferred back to the original place (or at least accessible) would be ideal in our case.
>>
>> Cheers and thanks in advance,
>>         Oliver
>>
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> 
> 

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature